Report 2014-120 Recommendation 4 Responses

Report 2014-120: California Public Utilities Commission: It Needs to Improve the Quality of Its Consumer Complaint Data and the Controls Over Its Information Systems (Release Date: April 2015)

Recommendation #4 To: Public Utilities Commission

To ensure that policy makers, enforcement officials, and the general public have access to accurate consumer complaint data in CIMS, the branch should develop and implement tools by September 30, 2015, to measure the quality management team program's effectiveness.

Annual Follow-Up Agency Response From October 2021

The Quality Management Team (QMT) supervisor produces a QMT report every two weeks that identifies the most common errors when processing a written complaint case. The QMT reports are also used to facilitate discussions at team meetings to ensure consistency. The Quality Management Team (QMT) is comprised of Consumer Affairs Representatives that ensure accuracy by performing a review of the coding (e.g., categorizing and attributes) for all written cases. Additionally, a Quality Assurance Unit (QAU) was established that currently focuses on internal unit training and overseeing the call monitoring quality program. The QAU also uses the QMT reports to facilitate internal trainings as needed.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

CPUC provided data addressing complaints for a two-week period that were reviewed by QMT supervisors, but it did not demonstrate how it was measuring the degree to which these reviews improved the quality of data collection over time. CPUC indicated that it has a quality assurance project underway that will produce automated reporting of the QMT's efforts and effectiveness. It expects to implement the project in the second quarter of 2022.

Annual Follow-Up Agency Response From November 2020

Due to the loss of IT Resources assigned to higher priority CPUC projects, the timeline for completion has changed. Currently, development progress is at 90% complete. In order to complete the implementation of this recommendation, CPUC needs to finish the CIMS-QA Reports, Testing (unit testing, test scripts, and User Acceptance Testing (UAT)), Training, and User Guide.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From October 2019

Updated 10/14/19 - Partially Implemented.

Due to the loss of IT Resources being assigned to higher priority CPUC projects, the timeline for completion has slipped. New IT Resources are being assigned to the project. A learning curve will be there for these new IT Resources. Development progress is at 57% currently. Remaining is finishing up the programming for the CIMS-QA code, Reports, Testing, UAT.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From October 2018

Although the project requirements and design phases were completed in 2017, the lack of IT resources in 2018 resulted in significant delays in automating and improving CIMS quality management processes. Specifically, the CPUC's IT Applications Programming staff only made limited progress in application development between January 2018 and July 2018 as staff resources were reassigned to higher priority CPUC projects. In addition, IT Project Management staff was replaced with new staff in June 2018. As a result of the delays, the contract for vendor resources also expired and the Branch is in the process of securing new vendor resources to complete technical manuals once project development and testing are complete. In August 2018, IT began shifting some IT Applications Program resouces back to the Branch's project and correspondingly estimates that the project will be complete and the automation be in place in July 2019. As of September 30, 2018, the project is on track to meet July 2019 completion.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From November 2017

In 2017, the Branch has secured resources to automate and improve parts of the quality management processes within the CIMS database. Resources include the CPUC's IT Applications Programming and Project Management units as well as a vendor specializing in business analysis. The project was chartered on 11/09/2016 by the CPUC as the "Consumer Information Management System - Audit Response Mitigation for Quality Assurance". The project was approved by the California Department of Technology in a Stage 1 Business Analysis on 01/23/2017. (Public Utilities Commission (8660): 8660-082 CIMS Audit Response Mitigation for Quality Assurance) The project requirements and design phases were approved on 03/17/2017 and 07/05/2017, respectively. The applications development was initiated on 08/14/2017. It is estimated that the project will complete and the automation be in place in mid-2018.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From October 2016

The Branch has updated its plan to measure and improve the effectiveness of its quality management team (see attached "CAB Quality Management Team Enhancement Plan"). As described in the plan, the Branch has completed the quality management process analysis described as Phase I. In Phase II, the Branch has begun to analyze baseline measures of its processes. That analysis has been completed for 2013-14 and 2014-15; it is anticipated that the analysis for 2015-16 will be completed in late 2016. Moreover, the Branch is actively pursuing a project to automate and improve parts of the quality management processes (Phases III and IV). As noted in the prior audit status responses, these process improvements will require resources from outside of the Branch, including support from the CPUC's IT unit and the database vendor, as well as additional staffing resources for CAB to ensure optimal quality management. To that end, the Branch's project request to build a database module to automate quality management processes was updated and approved in August 2016. The Branch began work in September 2016 securing funding to use a vendor to create a business analysis for the project. As of September 28, 2016, the Branch received budget approval to move forward with the project. It is anticipated that the project will begin in November 2016, with an estimated duration of six to nine months.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

The Branch has created a draft plan to measure and improve the effectiveness of its quality management team. As part of the draft plan, the Branch has begun to analyze baseline measures of its process. The Branch continues to research the feasibility of automating parts of the quality management processes and continues to research ability of its database to create multiple alerts to enable case progress to be better measured. Preliminary findings are the process improvements will require resources from outside of the Branch, including support from CPUC IT and the database vendor.

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

Branch has analyzed current QMT processes and is researching the feasibility of automating parts of the processes. Preliminary findings are that process improvements will require resources from outside of the branch, including support from CPUC IT and the CIMS database vendor. Current estimates are that IT resources will not be available until early to mid-2016.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

Branch is working with IT to expand its measurement capabilities in CIMS to assist in quality management team efforts.

California State Auditor's Assessment of 60-Day Status: Pending

All Recommendations in 2014-120

Agency responses received are posted verbatim.