Report 2012-107 Data Reliability Assessments

To calculate the total amount of regular and overtime pay, the average number of employees by pay type for each of the departments listed (California Department of Developmental Services, California Department of Corrections and Rehabilitation, California Department of State Hospitals, and the California Department of Social Services), and overtime pay as a percent of total earnings--the sum of regular and overtime pay-- for 2012.

To calculate the total amount of regular and overtime pay, and overtime pay as a percent of total earnings, by employee class title for each of the departments listed for 2012.

To calculate overtime paid as a percent of total earnings for the California Department of Developmental Services' (Developmental Services) Office of Protective Services' (OPS) employees as compared to all other department employees by location and department-wide for each year between 2008 and 2012.

For a selection of 62 Developmental Services' employees who were paid the same amount in overtime pay as regular pay for the period of 2008 through 2012, calculate the total amount of regular pay and overtime pay, and the number of employees by class title.

For a selection of eight Developmental Services' employees who were paid at least 150 percent of their regular pay in overtime for the period of 2008 through 2012, identify the employee's class title and work location, calculate the average weekly hours worked by calendar year, and total the employee's regular pay and overtime pay.

Sufficiently Reliable—We performed data-set verification procedures and electronic testing of key data elements and found no issues.

We relied on the completeness testing performed as part of the State's annual financial audit for payroll transactions between January 2008 and June 2012. Since the State's financial audit for fiscal year 2012-13 is still in progress, we cannot rely on this report to verify completeness between July 2012 and December 2012. However, because we found the payroll data to be complete between January 2008 and June 2012, we have reasonable assurance that the payroll data for the period of July 2012 through December 2012 are also complete. Consequently, we found the data to be complete.

We relied on the accuracy testing performed as part of the State's annual financial audit for payroll transactions between January 2008 and June 2011. For the period July 2011 through December 2012, we performed accuracy testing on a selection of 10 regular or overtime payroll transactions by tracing key data elements to supporting documentation and found no errors.

We performed additional accuracy testing on two key data elements—overtime hours paid and work location—for the selection of 62 Developmental Services employees by tracing key data elements to supporting documentation and found no errors.

To calculate the total accrued compensating time off leave hours for a selection of eight California Department of Developmental Services (Developmental Services) employees who were paid at least 150 percent of their regular pay in overtime for the period of 2008 through 2012.

Sufficiently Reliable—We performed data-set verification procedures and electronic testing of key data elements and found no issues.

We performed accuracy testing on a random sample of 31 total unique payroll transactions—selected from 62 Developmental Services employees who were paid at least the same amount in overtime as regular pay for the period of 2008 through 2012. Since our initial random sample of 29 only contained three transactions where the employee accrued compensating time off, we augmented our testing by selecting the next two transactions in our sample where the employee accrued compensating time off hours to gain additional assurance of the accuracy of these hours. We found no errors.

To verify completeness, we used the initial random sample selected for accuracy testing—described in detail above—in which we selected 29 unique payroll transactions. We traced payroll timekeeping documents to the leave accounting data to determine if accrued compensating time off was properly recorded in the leave accounting data. We found no errors.

To calculate the total amount of regular and overtime pay, the average number of employees by pay type for each of the departments listed (California Department of Developmental Services, California Department of Corrections and Rehabilitation, California Department of State Hospitals, and the California Department of Social Services), and overtime pay as a percent of total earnings--the sum of regular and overtime pay-- for 2012.

To calculate the total amount of regular and overtime pay, and overtime pay as a percent of total earnings, by employee class title for each of the departments listed for 2012.

To calculate overtime paid as a percent of total earnings for the California Department of Developmental Services' (Developmental Services) Office of Protective Services' (OPS) employees as compared to all other department employees by location and department-wide for each year between 2008 and 2012.

For a selection of 62 Developmental Services' employees who were paid the same amount in overtime pay as regular pay for the period of 2008 through 2012, calculate the total amount of regular pay and overtime pay, and the number of employees by class title.

For a selection of eight Developmental Services' employees who were paid at least 150 percent of their regular pay in overtime for the period of 2008 through 2012, identify the employee's class title and work location, calculate the average weekly hours worked by calendar year, and total the employee's regular pay and overtime pay.

Sufficiently Reliable—We performed data-set verification procedures and electronic testing of key data elements and found no issues.

We relied on the completeness testing performed as part of the State's annual financial audit for payroll transactions between January 2008 and June 2012. Since the State's financial audit for fiscal year 2012-13 is still in progress, we cannot rely on this report to verify completeness between July 2012 and December 2012. However, because we found the payroll data to be complete between January 2008 and June 2012, we have reasonable assurance that the payroll data for the period of July 2012 through December 2012 are also complete. Consequently, we found the data to be complete.

We relied on the accuracy testing performed as part of the State's annual financial audit for payroll transactions between January 2008 and June 2011. For the period July 2011 through December 2012, we performed accuracy testing on a selection of 10 regular or overtime payroll transactions by tracing key data elements to supporting documentation and found no errors.

We performed additional accuracy testing on two key data elements—overtime hours paid and work location—for the selection of 62 Developmental Services employees by tracing key data elements to supporting documentation and found no errors.

For Canyon Springs and Lanterman, determine the number of documented alleged resident abuse cases and of those cases the number investigated and substantiated.

For all five developmental centers, determine the disposition of alleged resident abuse cases, such as the number sent to the district attorney and the number that resulted in criminal complaints filed by the district attorney within the past five years.

Undetermined Reliability—We did not test the reliability of these data because it would not be cost-effective to trace this summary-level data back to the individual transactions that support the totals.

Determine the number of documented alleged resident abuse cases.

Determine the disposition of those cases, such as the number investigated and the number substantiated.

Sufficiently Reliable—We performed data-set verification procedures and electronic testing of key data elements and found no issues.

To test the accuracy of the developmental center's data, we randomly selected a sample of 29 incidents and verified that key data elements matched source documentation. We found no errors.

To test the completeness of the developmental center's data, we haphazardly selected 29 incidents and traced them from hard copy files back to the Incident Reporting Information System. We found the data to be complete.

Determine the number of documented alleged resident abuse cases.

Determine the disposition of those cases, such as the number investigated and the number substantiated

Sufficiently Reliable—We performed data-set verification procedures and electronic testing of key data elements and found no significant issues.

To test the accuracy of the developmental center's data, we randomly selected a sample of 29 incidents and verified that key data elements matched source documentation. We found one error within the special investigation number data field. Therefore, we randomly selected another 17 incidents to verify the special investigation number data field matched source documentation. We found no additional errors.

To test the completeness of the developmental center's data, we haphazardly selected 29 incidents and traced them from hard copy files back to the electronic database. We found the data to be complete.

Determine the number of documented alleged resident abuse cases.

Determine the disposition of those cases, such as the number investigated and the number substantiated.

Not Sufficiently Reliable—We performed data-set verification procedures and electronic testing of key data elements and found no significant issues.

Although accuracy testing is not required because we found the data to be incomplete, we decided the results of the testing would be important and informative. To test the accuracy of the developmental center's data, we randomly selected a sample of 29 incidents and verified that key data elements matched source documentation. We found no errors.

To test the completeness of the developmental center's data, we haphazardly selected 29 incidents and traced them from hard copy files back to the electronic database. We found one incident report that was not documented in the Risk Management Database. As a result, the data is incomplete.

Determine the number of documented alleged resident abuse cases.

Determine the disposition of those cases, such as the number investigated.

Undetermined Reliability—We performed data-set verification procedures and electronic testing of key data elements and found no issues.

We did not perform accuracy or completeness testing because hard copy source documentation was not available for this review. Alternatively, following the U.S. Government Accountability Office's guidelines, we could have reviewed the adequacy of selected system controls that include general and application controls. However, we did not conduct these reviews because the data we are analyzing was for one location for a two month period. We determined that the level of effort required to perform these reviews was not cost effective.

Determine the number of documented alleged resident abuse cases.

Sufficiently Reliable—We performed data-set verification procedures and electronic testing of key data elements and found no significant issues.

To test the accuracy of the developmental center's data, we randomly selected a sample of 29 incidents and verified that key data elements matched source documentation. We found no errors.

To test the completeness of the developmental center's data, we haphazardly selected 29 incidents and traced them from hard copy files back to the New Incident Reporting Management Application. We found the data to be complete.

Determine the number of documented alleged resident abuse cases that were investigated.

Not Sufficiently Reliable—We performed data-set verification procedures and electronic testing of this key data element and found no significant issues.

We did not test the accuracy of the number of resident abuse cases investigated because developmental center staff informed us that its employees do not consistently use the special investigation number—the data field that indicates whether a case was investigated—in the New Incident Reporting Management Application (NIRMA).

To test the completeness of the developmental center's data, we haphazardly selected 29 incidents and traced them from hardcopy files back to NIRMA. We found the data to be complete.

Determine the disposition of alleged resident abuse cases, such as the number substantiated.

Not Sufficiently Reliable—We performed data-set verification procedures and electronic testing of key data elements and found no significant issues.

To test the accuracy of the developmental center's data, we randomly selected a sample of 29 incidents and verified whether the key data elements matched source documentation. We found two errors in the Office of Protective Services case number data field, seven errors in the case type data field, and six errors in the disposition data field.

To test the completeness of the developmental center's data, we haphazardly selected 29 incidents and traced them from hardcopy files back to the Microsoft Excel files. We found the data to be complete.

To determine the classification and the amount of time it took the California Department of Public Health (Public Health) to initiate investigations into complaints and incidents.

Not Sufficiently Reliable—As we reported in a previous audit report, Department of Health Services: Its Licensing and Certification Division Is Struggling to Meet State and Federal Oversight Requirements of Skilled Nursing Facilities (2006-106, April 2007), the federal database Public Health uses to track its investigations, ASPEN, has weak controls that preclude Public Health (previously part of the Department of Health Services) from preventing erroneous data entry or detecting data errors for key data fields such as dates when complaints are received and investigated.

There were 10,746 complaints and incidents in our audit period and nine of them were assigned to more than one priority level. We did not include these nine in our testing. However, through our testing we discovered data limitations. Specifically, we found 863 of the 10,737 complaints and incidents we tested were either missing a date or had an investigation initiation date that occurred before the intake date. Although Public Health provided documents indicating that its investigation was initiated after the complaint or incident was received for four of these occurrences, we could not easily gather the documents needed to find the missing dates, evaluate illogical data, or rule out data entry errors for the remaining 859 records.

Public Health indicated that it has requested the federal government to make improvements to ASPEN.

To determine whether state licensing and federal certification surveys and certification revisits occurred when required.

Undetermined Reliability—Because the Department of Public Health indicated that it has requested the federal government to make improvements to ASPEN, we did not perform any further testing of the data we obtained from the federal database.

To calculate the total amount of regular and overtime pay, the average number of employees by pay type for each of the departments listed (California Department of Developmental Services, California Department of Corrections and Rehabilitation, California Department of State Hospitals, and the California Department of Social Services), and overtime pay as a percent of total earnings--the sum of regular and overtime pay-- for 2012.

To calculate the total amount of regular and overtime pay, and overtime pay as a percent of total earnings, by employee class title for each of the departments listed for 2012.

To calculate overtime paid as a percent of total earnings for the California Department of Developmental Services' (Developmental Services) Office of Protective Services' (OPS) employees as compared to all other department employees by location and department-wide for each year between 2008 and 2012.

For a selection of 62 Developmental Services' employees who were paid the same amount in overtime pay as regular pay for the period of 2008 through 2012, calculate the total amount of regular pay and overtime pay, and the number of employees by class title.

For a selection of eight Developmental Services' employees who were paid at least 150 percent of their regular pay in overtime for the period of 2008 through 2012, identify the employee's class title and work location, calculate the average weekly hours worked by calendar year, and total the employee's regular pay and overtime pay.

Sufficiently Reliable—We performed data-set verification procedures and electronic testing of key data elements and found no issues.

We relied on the completeness testing performed as part of the State's annual financial audit for payroll transactions between January 2008 and June 2012. Since the State's financial audit for fiscal year 2012-13 is still in progress, we cannot rely on this report to verify completeness between July 2012 and December 2012. However, because we found the payroll data to be complete between January 2008 and June 2012, we have reasonable assurance that the payroll data for the period of July 2012 through December 2012 are also complete. Consequently, we found the data to be complete.

We relied on the accuracy testing performed as part of the State's annual financial audit for payroll transactions between January 2008 and June 2011. For the period July 2011 through December 2012, we performed accuracy testing on a selection of 10 regular or overtime payroll transactions by tracing key data elements to supporting documentation and found no errors.

We performed additional accuracy testing on two key data elements—overtime hours paid and work location—for the selection of 62 Developmental Services employees by tracing key data elements to supporting documentation and found no errors.

To calculate the total amount of regular and overtime pay, the average number of employees by pay type for each of the departments listed (California Department of Developmental Services, California Department of Corrections and Rehabilitation, California Department of State Hospitals, and the California Department of Social Services), and overtime pay as a percent of total earnings--the sum of regular and overtime pay-- for 2012.

To calculate the total amount of regular and overtime pay, and overtime pay as a percent of total earnings, by employee class title for each of the departments listed for 2012.

To calculate overtime paid as a percent of total earnings for the California Department of Developmental Services' (Developmental Services) Office of Protective Services' (OPS) employees as compared to all other department employees by location and department-wide for each year between 2008 and 2012.

For a selection of 62 Developmental Services' employees who were paid the same amount in overtime pay as regular pay for the period of 2008 through 2012, calculate the total amount of regular pay and overtime pay, and the number of employees by class title.

For a selection of eight Developmental Services' employees who were paid at least 150 percent of their regular pay in overtime for the period of 2008 through 2012, identify the employee's class title and work location, calculate the average weekly hours worked by calendar year, and total the employee's regular pay and overtime pay.

Sufficiently Reliable—We performed data-set verification procedures and electronic testing of key data elements and found no issues.

We relied on the completeness testing performed as part of the State's annual financial audit for payroll transactions between January 2008 and June 2012. Since the State's financial audit for fiscal year 2012-13 is still in progress, we cannot rely on this report to verify completeness between July 2012 and December 2012. However, because we found the payroll data to be complete between January 2008 and June 2012, we have reasonable assurance that the payroll data for the period of July 2012 through December 2012 are also complete. Consequently, we found the data to be complete.

We relied on the accuracy testing performed as part of the State's annual financial audit for payroll transactions between January 2008 and June 2011. For the period July 2011 through December 2012, we performed accuracy testing on a selection of 10 regular or overtime payroll transactions by tracing key data elements to supporting documentation and found no errors.

We performed additional accuracy testing on two key data elements—overtime hours paid and work location—for the selection of 62 Developmental Services employees by tracing key data elements to supporting documentation and found no errors.