School tests widely flawed
Recent troubles in Hawaii are
in line with a pattern of
problems nationwide
The flaws just discovered on tests taken this spring by Hawaii schoolchildren follow foul-ups in the national testing industry, which is working to keep up with federal mandates for more testing than ever before.
A study published last year documented numerous errors by testing companies in recent years, including mistakes that led students to be denied diplomas. The paper, "Errors in Standardized Tests: A Systematic Problem," was issued by the National Board on Educational Testing and Public Policy, based at Boston College.
"Over the last few years, all the major test companies have had problems," Monty Neill, executive director of FairTest, an advocacy group based in Cambridge, Mass., said yesterday. "There's more testing, under higher stakes, with quicker turnaround and more complicated tests."
The San Antonio-based company that produced Hawaii's tests, now known as Harcourt Assessment Inc., has been involved in several cases. Among the recent snafus by Harcourt and others on the national testing front:
» In Nevada, 736 sophomores and juniors failed math tests in 2002 that they should have passed because Harcourt Education Management had miscalculated the cut score by one point. The company was fined $425,000. The following year, it used the wrong scoring system to grade third- and fifth-graders, and agreed to provide $435,000 in additional services to make up for it.While giving the Hawaii State Assessment this spring, teachers and test coordinators discovered several errors, including missing pages and mistakes in instructions. Sample questions that were given to some students had incorrect answers, which could have thrown off the children's confidence and concentration and taken away time from the rest of the test as they puzzled over them.» A Minnesota lawyer who demanded to see the math test his daughter had failed in 2000 found several scoring errors by National Computing Systems. It turned out that math scores for 45,700 Minnesota students in grades 8-12 were wrong, and 7,930 students who failed had actually passed because the wrong scoring form was used.
» In Georgia the Board of Education deemed its students' 2002 test results unusable because they were so late and riddled with errors. It withheld payment to Harcourt Brace.
» Earlier this year, CTB/McGraw Hill agreed to rescore the open-ended questions on its tests in Connecticut because the results did not jibe with those of previous years.
All students in grades 3, 5, 8 and 10 took the Hawaii State Assessment in math, reading and writing this year, and some students in grades 4, 6 and 7 took new tests that are being evaluated for next year. More than 30 different test forms were used.
Harcourt is working with Hawaii officials to figure out how widespread the errors were, what impact they had and how to remedy them. Every scorable question on the tests has been field-tested, so previous results can be used as a benchmark to gauge whether students appeared to stumble.
"We're going to make sure that Harcourt does this right," said Bob McClelland, acting director of planning and evaluation for the state Department of Education. "That means ensuring that none of these errors will negatively impact any student or any school."
State Superintendent Pat Hamamoto said Thursday that she does not expect any students will have to be retested, although that possibility has not been ruled out.
Unlike states where a single test can spell the difference between graduating or not, scores on the Hawaii State Assessment do not count toward a child's grades or promotion to the next grade. However, they can dramatically affect a school's future. The scores determine which schools fail to meet federal benchmarks and could face sanctions such as replacement of staff.
The No Child Left Behind Act, which became law in 2002, requires testing of all students in reading and math in grades 3 through 8 starting next year.
"One of the concerns of educators, given the magnitude and scope of testing across the nation under No Child Left Behind, was whether these testing companies have the capacity to take care of the needs," Hamamoto said.
Neill, of FairTest, said the mistakes that have emerged show they do not and that other errors no doubt go undetected.
"The states are requesting customized tests, and there's a lot of work to be done and it's technically complicated work," he said. "There's a limited supply of qualified people. The testing companies are really in a jam, and they are likely to stay in a jam."
Testing company officials counter that they have hired more staff to keep up with growing demand, and while they strive for perfection, mistakes can happen when millions of students are being tested and time is limited.
"Every testing company has experienced errors," said Mark Slitt, spokesman for Harcourt Assessment Inc., the San Antonio-based company that produced Hawaii's tests. "We are made up of human beings, and human beings sometimes make errors."
"We have a very vigorous quality-assurance department, and clearly something here didn't pass muster," he said. "We don't think there's a capacity issue. As we've gotten more state contracts, we have added new people."
Neill said the pressure could be relieved if the federal government backed off on its mandate that all students in grades 3 through 8 be tested, and instead allowed states to choose selected grades. He also said the government should not mandate high-stakes decisions based on the results of a test.
"If you remove the high stakes, then there may be errors," he said, "but they won't have these kinds of damaging consequences."