State: No evidence computer glitch had 'discernible impact' on ISTEP scores

A review of alleged scoring errors on Indiana's high-stakes ISTEP tests found that a computer glitch had no "discernible impact" on scores but determined that scoring errors on individual student exams could not be ruled out.

An expert panel conducted the review on behalf of the state using data provided by CTB/McGraw Hill, the state's ISTEP testing vendor. But the panel said its overall conclusion "does not mean that each student was scored accurately," and they noted that a better assessment of the issue was impossible since the problem occurred more than six months earlier.

Indiana Superintendent of Public Instruction Glenda Ritz said the review provides evidence that the glitch did not negatively affect scores.

"While the experts can’t go back and actually review every single student test score, there is no evidence or indication that this particular allegation had any impact on the scores at all," she said.

The state launched the review of the scoring results after a Dec. 13 Indianapolis Star investigation raised questions about a computer glitch that may have inadvertently changed scores on test questions. Seven scoring supervisors who worked for CTB/McGraw Hill told The Star that the glitch was widespread enough to warrant rescoring, but that management decided against it. Some supervisors estimated that tens of thousands of test questions were scored incorrectly.

Three experts hired by the State Board of Education focused on data regarding scores on the ISTEP's writing responses, where they suspected the computer malfunction's impact would be more pronounced. The data from CTB showed no pattern suggesting the glitch caused widespread errors.

But the state's experts did not interview any scoring supervisors and did not review data from open-ended math questions, which supervisors told The Star would have been more susceptible to the glitch.

One of the state's experts acknowledged in an interview with The Star that without a "hugely expensive" regrading of the test, it is impossible to know whether student scores were impacted.

"Without rescoring individual students, you couldn't tell if it had affected individual students," said Ed Roeber, one of the state's consultants.

Even if CTB did rescore the tests, it might be difficult to detect errors because different evaluators could produce different results, he said.

Wes Bruce, another state consultant, said some errors are inevitable.

"This is called 'human scoring' for a reason," he said. "It's always possible that a scorer can enter an incorrect score for a student on any item, not just by this issue."

But that's little consolation to teachers whose pay is determined in part by ISTEP results, said Teresa Meredith, president of the Indiana State Teachers Association.

"If anything (the report) just validates how teachers feel about the data," she said. "It may be accurate, it may not be accurate. We have no confidence in ISTEP or the data it produces."

Concerns about the scoring malfunction are just the latest problems to plague the ISTEP. Lower test scores due to a new, more rigorous exam last school year, delays in scoring results and past testing disruptions have already caused outrage among parents, teachers and school administrators.

Although Ritz and Gov. Mike Pence said they are encouraged by the results of the review, they still want lawmakers to prevent this year's lower ISTEP scores from hurting teachers and schools amid the transition to a harder test.

And although Ritz is standing by the validity of the scores, she said her department could seek damages from CTB for not notifying the state about the computer glitch when it was initially discovered in April.

State consultants who spoke with the The Star said CTB should have notified the department earlier, which would have allowed a real-time analysis of the computer glitch.

Instead, the state must rely on data that can only provide limited information about the impact of the glitch, making it hard to determine whether Ritz achieved the goal she outlined in a letter to school administrators earlier this month.

“I remain committed," she wrote, "to ensuring that every student receives proper credit for their performance on the ISTEP+."

Call Star reporter Chelsea Schneider at (317) 444-6077. Follow her on Twitter: @IndyStarChelsea. Call Star reporter Tony Cook at (317) 444-6081. Follow him on Twitter: @indystartony.