Report 2011-103 Data Reliability Assessments

To determine pay rates for personnel and hours billed to paying entities.

Sufficiently Reliable—We performed data-set verification procedures, electronic testing of key data elements, accuracy, and completeness testing. We identified no issues when performing data-set verification procedures or electronic testing of key data elements. To test the accuracy of the data, we randomly selected a sample of 29 records from the invoicing system and traced key data elements to source documents. We found an error in each of the six key data fields. In response to these errors, we increased our accuracy sample from 29 to 46 records for these six key data fields, and found no additional errors. Finally, we tested the completeness of the invoicing system by comparing a haphazard sample of 29 records collected from Emergency Services' office against Emergency Services' invoicing system and found no exceptions.

To compare the dollar amount of aid provided and received between 2006 and 2010 and to review various components in the amount of aid, such as the average hourly rates.

Sufficiently Reliable—We performed data-set verification procedures, electronic testing of key data elements, accuracy, and completeness testing. We identified no issues when performing data-set verification procedures or electronic testing of key data elements. To test the accuracy of the data, we randomly selected a sample of 29 records from the invoicing system and traced key data elements to source documents. We found an error in each of the three key data fields. In response to these errors, we increased our accuracy sample from 29 to 46 records for these three key data fields and found no additional errors. Finally, we tested the completeness of the invoicing system by comparing a haphazard sample of 29 records collected from Emergency Services' office against Emergency Services' invoicing system and found no exceptions.

To identify the number of firefighting resources agencies have available.

Not Sufficiently Reliable—We performed data-set verification procedures, electronic testing of key data elements, accuracy, and completeness testing. We identified no issues when performing data-set verification procedures or electronic testing of key data elements. To test the accuracy of the data, we randomly selected a sample of 29 records from the invoicing system and traced key data elements to source documents. We found that seven key data fields had two or more errors in the first three records we tested. Due to the number of errors identified, we did not continue testing the remaining sample items. Finally, we tested the completeness of the invoicing system by electronically comparing the emergency aid records data to inventory records to ensure that the data contained inventory records for all agencies responding to emergencies. We found the data to be materially complete.

To select a sample of law enforcement missions.

Undetermined Reliability—A November 2008 feasibility study report prepared by the Governor's Office of Emergency Services identified challenges reported by RIMS users, such as the lack of a consistent method for updating RIMS reports and the loss of large amounts of data due to system "timeouts" when users were completing forms. Further, a 2009 report prepared by Emergency Services' consultant found that data and information reported within RIMS were frequently inconsistent, not credible, and unusable for analyses because Emergency Services lacked standardized processes and detailed desk instructions and because it did not have a comprehensive approach to train all staff in the use of RIMS. The consultant also found that Emergency Services lacked adequate internal controls over RIMS, such as the ability to overwrite data fields when staff do not use the "update" feature. Finally, we interviewed an Emergency Services official who stated that the information in RIMS may not always be accurate. As a result, we did not conduct data-set verification procedures or completeness testing.

To select fire emergencies.

Undetermined Reliability—We did not perform data-set verification procedures or completeness testing because we were unable to obtain an extract from the ROSS database.