Report 2011-117 Data Reliability Assessments

To identify students' performance on their first attempt at the exit examination during the second half of grade 10.

Undetermined Reliability—We performed data-set verification procedures and electronic testing of key data elements and did not identify any issues.

The department contracted with Educational Testing Service (ETS) to provide for the development, administration, scoring, analysis, and reporting of the exit examination. ETS subcontracted for the design, printing, and scanning of hard-copy student answer documents.

We were unable to perform accuracy and completeness testing as required by government auditing standards because ETS' subcontractor previously destroyed the hard-copy student answer documents in accordance with the terms of its contract.

Alternatively, we interviewed officials at the California Department of Education (department) and reviewed relevant documentation to determine whether it conducted any work to verify that student essays were accurately scored. We learned that the department contracted with Human Resources Research Organization (HumRRO) to conduct an independent evaluation of the exit examination test results through September 2008. As part of its evaluation, HumRRO analyzed the consistency with which student essays were scored for the 2007-08 school year and found that the results were generally comparable to the 2006-07 school year and somewhat improved in comparison to previous years.

We also interviewed officials at the department, ETS, and ETS' subcontractor and reviewed relevant documentation to determine whether they conducted any work to verify the accuracy of the electronic file of student multiple-choice answers against the hard-copy answer documents. Although ETS asserted to us that it performed quality control measures related to the accuracy of the data, it was not able to produce the detailed results of this analysis. We found that the level of detail ETS provides to the department when reporting the results of its quality control measures could be improved. This was communicated to the department in a separate management letter.

To calculate dropout and graduation rates among various demographic categories and by California High School Exit Examination (exit examination) result for the 2009-10 cohort.

Undetermined Reliability—We performed data-set verification procedures and electronic testing of key data elements and did not identify any issues.

We did not perform accuracy and completeness testing because the source documentation for this system is not centrally stored at the California Department of Education, thereby making such testing impractical.

Alternatively, we examined 20 student records from each of the six school districts we visited to determine how schools assigned exit codes and whether they conformed to the department's instructions.