Errors may void
student scoresFlaws turn up in tests given
statewide under a federal law
designed to assess isle schools
Errors in the statewide tests just given to tens of thousands of students in Hawaii's public schools could invalidate some results and require some retesting, officials acknowledged yesterday.
"We made the errors, we feel terrible about the errors and, based on our conversations with the state of Hawaii, we will come to some agreement about what is fair and equitable compensation," said Mark Slitt, spokesman for Harcourt Assessment Inc., the San Antonio, Texas-based company that developed the tests as part of a five-year, $20 million contract with the state.
"We're working very closely with the state to understand the extent of the problem and to determine what impact it may have had on the children and on the schools," he said. "We've had a relationship with the state of Hawaii for about 30 years, and this is the first time we've had errors of this magnitude."
A lot is riding on the results of the Hawaii State Assessment, a series of tests that determine which schools meet federal standards and which face sanctions under the No Child Left Behind Act. Teachers and students have been gearing up all year for the tests, which were given in March and April, with scores expected to be released in August.
All students in grades 3, 5, 8 and 10 took the Hawaii State Assessment in math, reading and writing this year, and some students in grades 4, 6, and 7 took new tests that are being evaluated for next year.
But as the tests were being given, teachers discovered errors in directions and in sample questions and answers given to students. For example, on one version of the eighth-grade math test, a sample item required students to calculate how much a person earns working different hours on different days.
"According to the directions, we tell them that the answer is $117.50," said Rob Flemm, testing coordinator at Kealakehe Intermediate School. "But the correct answer to the problem is $150."
"The students work the sample problems, come up with the correct answer, and then the directions tell them that they were wrong," said Flemm, who alerted authorities to several mistakes on the test. "Students then go on to the scorable section of the test believing that their reasoning or process for solving the problem was wrong. My training has taught me that an error such as this invalidates the entire test."
In another case, Flemm said, students were told to quit working on page 55 when there were actually three more pages of problems for that section. Because teachers don't have answer keys to the tests, they could uncover problems only in the instructions, he said.
"What is my assurance that Harcourt didn't make the same mistakes on the test itself?" he asked.
Selvin Chin-Chance, head of the Test Development Section of the Hawaii Department of Education, said the department is working with test coordinators, teachers and Harcourt to compile a list of errors to determine what needs to be done. The problem is complicated by the fact that there are multiple versions of each test, because the department field-tests new items each year, he said.
Harcourt is conducting a thorough review of all the tests, including the answer keys on the 30-plus versions of the test, Chin-Chance said yesterday.
"We're in a fact-finding process at the moment," he said. "We're hoping that within a week we can have a complete listing of all the issues that have been raised that are potential errors."
The testing company and school officials will then analyze each problem to see how it might have affected students' responses. Every scorable question on the tests has been field-tested, so previous results can be used as a benchmark to gauge whether students appeared to stumble, Chin-Chance said. Problematic items could be removed.
"There are fairly routine professional ways of confirming that, yes, there is a problem, and exactly what procedures you use to wipe out the impact," Chin-Chance said. "We want to ensure that the students are not penalized. That's the last thing we want to happen."
Chin-Chance said an error in a sample problem would "not necessarily" invalidate the test, but could require the students who took that version of the test to retake that portion of it, a cost that would be covered by Harcourt.
"That would not be easy because school ends in a month," Chin-Chance said, adding that it might have to take place after summer break, when students will have moved on to different grades.
Flemm said he is frustrated because he found errors on the test last year, including a misplaced "stop" sign that signaled students to quit working. In that case, he was told students would not be penalized for missing questions that followed, and a Harcourt official assured him it would not happen again.
"They said that they would fix it," he said. "Instead, it got worse this year. People are going to evaluate schools on the basis of a test that has not been proven to be valid. I have a real problem with that."
Harcourt has statewide contracts with more than 20 states, Slitt said. Its Hawaii contract, which started with last year's test, covers test development, administration, scoring, manuals, workshops and reports. It will include tests in four subjects and eight grade levels by 2007.
"Our goal is to deliver a testing program that is error-free and defect-free, that people can have confidence in, and fulfill our mission to help Hawaii educate its children," Slitt said. "I can assure you that interest in resolving this matter goes to the very top of the company. The president is keenly involved."