A leading US education academic has warned that it would be “extremely foolish” and even damaging to student learning if NAPLAN writing tests were marked by computers next year, as education ministers across Australia back a move to online marking.
Les Perelman, an internationally renowned expert in writing assessment from the Massachusetts Institute of Technology, said a report on automated marking of NAPLAN was “so methodologically flawed and so massively incomplete” that it could not be used to justify any use of automated essay scoring of NAPLAN.
Dr Perelman was commissioned by the NSW Teachers Federation to review a report by the Australian Curriculum, Assessment and Reporting Authority (ACARA) into automated NAPLAN marking of persuasive writing tests.
ACARA’s report, released in late 2015, said a “significant body of literature” confirmed that automated essay scoring met or surpassed the quality of human markers. But Dr Perelman’s review said a major failing of the ACARA report was that it “completely ignores” any research that was critical of automated essay scoring.
“Until these critical studies are completed and carefully evaluated, it would be extremely foolish and possibly damaging to student learning to institute machine grading of the NAPLAN essay, including dual grading by a machine and a human marker,” Dr Perelman wrote.
The release of Dr Perelman’s review comes as ministers at September’s meeting of the federal Education Council gave in-principle support for NAPLAN writing tests to be marked by a computer and a human in 2018.
Stanley Rabinowitz, general manager of assessment and reporting at ACARA, said all NAPLAN writing tasks completed online next year would be marked by a person as well as an automated scoring system.
“This is to provide reassurance that automated marking achieves scores comparable to human markers, but faster,” Dr Rabinowitz said.
Dr Rabinowitz said ACARA had done further research since the 2015 report, including work based on Dr Perelman’s research, which would be released next month.
He defended ACARA’s report and said Dr Perelman and the Teachers Federation “are known critics of automated marking systems and the report findings should be viewed with this in mind.”
The acting president of the NSW Teachers Federation, Gary Zadkovich, said parents, teachers and principals had not been consulted about the “radical plans” to move to online marking.
“The federal agency in charge of NAPLAN is rushing through with plans to have robots mark next year’s NAPLAN tests despite their justifications being discredited by world-leading research,” Mr Zadkovich said.
Mr Zadkovich urged education ministers to reject ACARA’s plan to “bring robots into the marking of extended pieces of children’s work”.
Mr Zadkovich said Dr Perelman’s report warned that computers could only detect “low grade attributes of writing” and cannot detect “the most important elements of a text”.
Automated marking can discriminate against some social groups and is even flawed when it comes to grammar checking, he said.
Robyn Cox, the president of the Primary English Teaching Association Australia, said she did not oppose the role of artificial intelligence in education but warned that its ongoing involvement in areas such as writing could have negative consequences.
“My concern is that this will serve the needs of the computer and not the needs of humanity,” Dr Cox said.
“It won’t take us long before big corporations or text book publishers start developing software or text books that prepare kids for the writing task that a computer wants to see.”
The ACARA said the number of schools taking part in the double marking had not yet been confirmed but the Education Council was told that the move will deliver a “significant increase in costs”.
The ministers also agreed to extend the timeline for all schools to transition to NAPLAN online to 2020. All year 3 students will do NAPLAN writing tasks with pencil and paper regardless of when their school moves online.