Report 2017-117 Recommendation 9 Responses

Report 2017-117: Mental Health Services Act: The State Could Better Ensure the Effective Use of Mental Health Services Act Funding (Release Date: February 2018)

Recommendation #9 To: Mental Health Services Oversight and Accountability Commission

To ensure proper oversight and evaluation of outcomes for the Prevention and Innovation projects, the Oversight Commission should finalize its internal processes for reviewing and analyzing the program status reports no later than July 2018. Further, in order to fulfill its statutory responsibility to provide oversight and accountability for MHSA programs, the Oversight Commission should ensure that it launches all three data tools to track local mental health agencies' funding, services, and outcomes as it intends.

Annual Follow-Up Agency Response From October 2019

Commission staff completed and implemented internal processes for reviewing and analyzing the county Annual PEI MHSA reports in early 2019. Additionally, the Commission has launched all three data tools as part of its Transparency Suite which is on the Commission website. The first tool focused on fiscal transparency and was launched in August 2017 and substantially revised in March 2018. The second tool allows the public to search for, view, and compare MHSA-funded programs on key characteristics, such as target population, budgets, and program goals was launched in April 2019. The third tool allows the public to explore program outcomes data, with an initial emphasis on Full Service Partnership programs, was also launched in April 2019.

Due to data privacy concerns as well as staffing and fiscal constraints, the Commission has not yet been able to incorporate appropriate service statistics from the FY 2016-17 annual reports into the Transparency Suite database. Staff also are analyzing the first triennial PEI evaluation reports, which covered the FY 2016-17 and 2017-18 service years and were due in June 2019. The Commission expects to incorporate program service data and evaluation results from both the annual reports and the triennial reports during the first half of calendar year 2020.

The Commission anticipates some fiscal constrains to maintaining and updating these data tools. The Commission was able to initially launch the tools with administrative savings and it is unclear if we can sustain this effort with savings.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

1-Year Agency Response

The Commission has established a two-phase process for the internal review and analysis of the Prevention and Early Intervention (PEI) program and Innovation project annual reports. Phase one, a preliminary review of compliance with the regulatory requirements has been completed. The second phase, a program-level data review, which extracts information about each program including demographics of those served, is approximately two-thirds completed. This information will be analyzed to evaluate programmatic trends, including whether the programs are serving the intended populations and whether they are addressing the seven negative outcomes of untreated mental illness specified in the Mental Health Services Act. Information from these reports is being used to populate parts of the Commission's on-line transparency data tools.

The Commission has made great progress towards launching all three data tools, for fiscal transparency, program transparency, and outcomes transparency, respectively. The first tool was launched in August 2017 and substantially revised and extended in March 2018. The second tool is currently in final testing, with publication expected in April 2019. This tool allows the public to search for, view, and compare MHSA-funded programs on key characteristics, such as target population, budgets, and program goals. The database, based on information provided by the counties, as of February 22, 2019 includes information on 887 PEI programs. The third tool will allow the public to explore program outcomes data, with an initial emphasis on Full Service Partnership programs. This tool is currently in final testing, with publication expected in April 2019.

The Commission anticipates some fiscal constrains to maintaining and updating these data tools. The Commission was able to initially launch the tools with administrative savings and it is unclear if we can sustain this effort with savings.

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

The Commission has established a two-phase process for the internal review and analysis of the Prevention and Early Intervention program and Innovation project annual reports. Phase one, a preliminary review of compliance with the regulatory requirements has been completed. The second phase, a program-level data review, which extracts information about each program including demographics of those served, is in progress. This information will be analyzed to evaluate programmatic trends, including whether the programs are serving the intended populations, as well as whether they are addressing the seven negative outcomes of untreated mental illness specified in the MHSA. Information from these reports is being used to populate aspects of the Commission's on-line transparency data tools.

The Commission has made great progress towards completing the second part of Recommendation #9, that the Commission launch all three data tools, for fiscal transparency, program transparency, and outcomes transparency, respectively. The Commission's first tool (Fiscal Transparency) was launched in August 2017 and substantially revised and extended in March 2018. The second tool (MHSA Program Search, Display and Compare) is currently in beta testing, with publication expected in November 2018. This tool will allow the public to search for, view, and compare MHSA-funded programs on key characteristics, such as target population, key intervention strategies, budgets, and program goals. The database, based on information provided by the counties in their 2017-2020 MHSA Three-Year Program and Expenditure Plans, annual MHSA Revenue and Expenditure Reports, and annual Prevention and Early Intervention program reports, as of August 22, 2018 includes information on 446 Prevention and Early Intervention programs from across the state.

The Commission has launched its data tools with administrative savings and it is unclear if we can sustain this effort or timeframe with savings.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

The Commission has established a two-phase process for the internal review and analysis of the Prevention and Early Intervention program and Innovation project annual reports. Phase one includes a preliminary review, which identifies which counties have submitted their reports. Commission staff will follow up with the counties that have not submitted their reports until all the required reports have been submitted. This first phase includes a preliminary review of compliance with the regulatory requirements. The second phase is a program-level data review, which extracts information about each program including demographics of those served. This information will be analyzed to evaluate programmatic trends, including whether the programs are serving the intended populations, as well as whether they are addressing the seven negative outcomes of untreated mental illness specified in the Mental Health Services Act.

Information from these reports will be used to populate the Commission's on-line transparency data tools. In August 2017 the Commission launched the first of these tools, the Fiscal Transparency Tool that displays counties' annual MHSA revenues, expenditures, and year-end balances of unspent funds. The second tool will provide information on the programs and services and the third tool will provide information on the outcomes of those programs.

The Commission has made great progress towards completing the second part of Recommendation #9, that the Commission launch all three data tools. The Commission's second tool is on track to be completed by September 2018. This tool will allow the public to search for, view, and compare key program characteristics, such as target population, key intervention strategies, budgets, and program goals.

The Commission's third data tool is under development and the Commission anticipates piloting key outcome metrics specified in the MHSA late in the 2018-19 fiscal year.

California State Auditor's Assessment of 60-Day Status: Pending

All Recommendations in 2017-117

Agency responses received are posted verbatim.