Time crunch led
to test errorsThe maker of the faulty school
tests says it is changing procedures
Tests given recently to public school students in Hawaii contained a series of errors because the Texas company that created them was rushing to meet deadlines, officials said yesterday.
"Mostly due to a time crunch, certain quality-assurance steps were skipped," said John Tanner, vice president of testing services for Harcourt Assessment Inc., which is based in San Antonio. "Employees went ahead and took a couple of shortcuts."
He and other company representatives came to Honolulu to explain what the company is doing to fix the problem and ensure it doesn't happen again. Among planned changes, the company will now add another layer of checks to its quality-control system, requiring a sign-off that each step has been completed, Tanner said.
Before any test is approved, the company will also conduct dry runs that "replicate every aspect of the entire test experience," rather than simply have people take the test and look at answers, he said.
Testing specialists are still determining the impact of the errors, but they say no children will need to be retested. The 45 mistakes found by teachers, state test specialists and Harcourt were sprinkled through more than 30 test forms, as well as associated paperwork such as instructions to test administrators. Problems included punctuation, misleading instructions, missing items and wrong answers in sample questions given to guide students.
Only one error was in a scorable question. New questions that are field-tested each year don't count toward a student's score. Mistakes in sample questions and instructions, however, could have thrown off students.
"A lot of these things might have affected students' performance," Tanner said in an interview. "We don't expect we'll find that, but if we do, we have methodologies that will allow us to address that. We can work student by student to ensure we have results with integrity."
All students in grades 3, 5, 8 and 10 took the Hawaii State Assessment in math, reading and writing; some students in grades 4, 6 and 7 took field tests being evaluated for use next year.
Tracy Gardner, senior psychometrician for Harcourt, said company specialists and an independent team will analyze each student's results. Because students take the test over several sessions, experts can see whether performance appeared to suffer as a result of an error, she said. Every scorable question on the tests has been field-tested, so previous results can be used as a gauge, too.
Rather than retesting, students could be credited for items or questions could be thrown out. Each test contained roughly 70 points each, so there should be enough information to determine the performance of each child. Under federal mandates, children are ranked as "well below proficiency," "approaches proficiency," "meets proficiency" or "exceeds proficiency."
"Even in the worst-case scenario, we believe we can still put kids into the four buckets correctly," said Selvin Chin-Chance, head of the Test Development Section of the state Department of Education. "The errors were scattered throughout the grades, throughout the test forms, minimizing the impact on each."