2019-041 - Status of Recommendations - Table 2

10. To increase its pool of expert witnesses, by June 2017, BRN should take the steps necessary to increase the hourly wage it pays expert witnesses.

17. To the extent that Consumer Affairs chooses to implement BreEZe at the phase 3 regulatory entities, it should first complete a formal cost-benefit analysis to ensure that BreEZe is a cost-effective solution to meet these regulatory entities' business needs. To make certain this analysis is complete, it should include an assessment of the potential changes these regulatory entities may require to be made of the BreEZe system and the associated costs. Consumer Affairs should complete the cost-benefit analysis before investing any more resources into the implementation of BreEZe at the phase 3 regulatory entities, and it should update this analysis periodically as significant assumptions change.

35. To ensure that future training for BreEZe system rollouts is timely and effective, Consumer Affairs should provide training on the BreEZe system as close to the rollout date as possible to ensure that staff retain the information for using the system as it is implemented.

36. To ensure that future training for BreEZe system rollouts is timely and effective, Consumer Affairs should work with the regulatory entities to develop training that is specific to each entity's business processes.

5. To ensure that it appropriately monitors CalHome as required by statute, regulation, and program guidelines, HCD should immediately collect all required reports and follow up with recipients to obtain missing reports. Staff should withhold fund disbursements from recipients that have not submitted required reports. If the submitted reports reveal a problematic trend, such as a recipient not disbursing funds, HCD should take appropriate corrective action with the recipient.

8. To ensure that the IIG program award funds benefit the target population, HCD should develop and use a tool by December 1, 2018, to track which awards are monitored by local jurisdictions or by other HCD programs and which are not monitored at all. HCD should then immediately obtain monitoring reports from the local jurisdictions and other HCD programs to verify monitoring and review the results of such monitoring. HCD should follow up on any noted deficiencies. Further, HCD should, by January 1, 2019, develop a plan to perform on-site visits for those recipients that do not receive adequate monitoring from another source, and it should perform the planned on-site monitoring.

11. To ensure that it maximizes the benefit of the funds it has invested in CAPES's development and to support its ongoing efforts to improve CAPES's usability, HCD should, by January 1, 2019, determine to what extent the programs could be using CAPES at its current level of development to capture information.

14. To ensure that it maximizes the benefit of the funds it has invested in CAPES's development and to support its ongoing efforts to improve CAPES's usability, HCD should, by January 1, 2019, develop a documented process to ensure that all data in CAPES are accurate and complete. This process should include all phases of contract management, including monitoring. HCD should implement a routine periodic review of this process and update the process as necessary.

16. To ensure that it is able to meet its administrative monitoring obligations and that it uses housing bond funds in compliance with state law, regulations, and program guidelines, HCD should develop a long-term plan by January 1, 2019, for how it will avoid exceeding the administrative cost limits of those programs in the most immediate danger of overage and for how it will address instances when it has exceeded administrative cost limits. The plan should identify the programs at risk of exceeding the limit; the actions HCD will take for each program to gain efficiencies; its plan for moving staff between programs; a request for more money or legislative changes such as modifying the statutory limit on administrative spending, if necessary; and an evaluation of the consequences of not fulfilling its monitoring obligations.

18. To ensure that it complies with state law, prudently uses administrative funding, and promotes transparency, HCD should calculate and retain only funds equal to its actual administrative costs in instances when it does not disburse awarded funds to a recipient and subsequently grants the funds to another recipient.

19. To ensure that it does not exceed administrative cost restrictions and that it maximizes the funds intended to address target populations' housing needs, HCD should estimate when it will run out of administrative funds for any specific program, document its projection methodology, and provide underlying data and support for its estimates. The projections should include, but not be limited to, actual staff time spent on the program, the number of awards being monitored, and the length of monitoring. Staff should provide these projections and methodologies to management for review and approval by December 1, 2018, and then at least biannually thereafter.

35. Following the completion of the analysis (described in Recommendation 34), the Medical Board should take the appropriate follow-up actions that it deems necessary, including the investigation of physicians identified in its analysis.

14. Ensure that scheduling staff use the appropriate post codes in the scheduling software for all nursing staff so their time is accurately reported.

15. Ensure that all other nursing staff assignments to ad hoc posts are appropriate, and require schedulers to enter a note in the scheduling software indicating an employee's duties while in a general ad hoc post.

5. To ensure that it can better monitor how its medical staff and contractors adhere to the informed consent requirements of Title 22, sections 70707.1 through 70707.7, the Receiver's Office should develop a plan by August 2014 to implement a process by December 2014 that would include working with Corrections to establish a process whereby inmates can have witnesses of their choice when consenting to sterilization, as required by Title 22, or working to revise such requirements so that there is an appropriate balance between the need for secure custody and the inmate's ability to have a witness of her choice.

2. Corrections should immediately require mental health staff to score 100 percent on risk evaluation audits in order to pass. If a staff member does not pass, Corrections should require the prison to follow its current policies by reviewing additional risk evaluations to determine whether the staff member needs to undergo additional mentoring.

8. To monitor prisons' compliance with its requirement that inmates in crisis beds receive daily progress notes, Corrections should implement monitoring of these notes electronically into its audit process by the time the electronic health record system is in use systemwide in October 2017. Corrections should require prisons that are out of compliance to develop and implement quality improvement plans, and it should follow up on the prisons' implementation of those plans.

11. To address the unique circumstances that may increase its female inmates' rates of suicide and suicide attempts, Corrections should continue to explore additional programs that could address the suicide risk factors for female inmates.

12. To ensure that all prison staff receive required training related to suicide prevention and response, Corrections should immediately implement a process for identifying prisons where staff are not attending required trainings and for working with the prisons to solve the issues preventing attendance.

13. To ensure that trainers and risk evaluation mentors at all prisons are able to train staff effectively, Corrections should immediately begin requiring prisons to report the percentage of their trainers and mentors who have received training on how to conduct training and mentoring. It should work with prisons to ensure that all trainers and mentors receive adequate training.

14. To maximize the value of its trainings related to suicide prevention and response, Corrections should ensure that starting in January 2018, its trainings include all content that the special master and its own policies require.

15. To ensure that it has enough staff to provide mental health services to all inmates who require care, Corrections should review and revise its mental health staffing model by August 2018.

16. To ensure that prisons comply with its policies related to suicide prevention and response, Corrections should continue to develop its audit process and implement it at all prisons by February 2018. The process should include, but not be limited to, audits of the quality of prisons' risk evaluations and treatment plans.

17. To ensure that prisons can easily access Corrections' current policies related to mental health, Corrections should ensure that its program guide is current and complete as it works to incorporate the program guide into regulations. Corrections should immediately begin working with federal court monitors to draft regulations.

18. To ensure that suicide prevention teams meet quorum requirements, Corrections should, starting January 2018, work with prisons that consistently fail to achieve a quorum to resolve issues that may be preventing the teams from having all required members present at meetings.

3. To make certain that victims are aware of threats to their health, CIM should follow state law and ensure that its medical personnel immediately inform victims of gassing attacks of any evidence suggesting that the inmates involved have a communicable disease. It should further document that it has provided this information to victims.

21. To ensure the health and safety of its officers when interacting with inmates, CIM should provide annual training that is specific to preventing and responding to gassing attacks.

22. To ensure that it is able to identify high-risk situations and deter repeat offenders, CIM should specifically track all gassing attacks and use the tracking data as a tool to prevent future gassing attacks.

19. Corrections and Correctional Health Care should run a query of exempt positions related to the chief psychologist classification, such as clinical psychologists and senior psychologists, to determine whether any other exempt employees were improperly credited or paid for on-call or call-back assignments prior to December 2014, and seek recovery through reducing those employees' accumulated leave balances.

15. CDCR should revise the Institutional Worker Supervision Pay (IWSP) procedure to require that personnel staff review and ensure that an employee's direct supervisor signs the qualifying employee's timesheets and IWSP documents each month.

17. CDCR should enforce its current procedure to retain Institutional Worker Supervision Pay (IWSP) documentation.

19. CDCR should train all employees, supervisors, and personnel staff who receive, approve, or issue the extra pay to ensure that they are familiar with the requirements of the IWSP procedure and Pay Differential 67.

20. Fully implement and continue to follow recommendations from prior investigative reports involving similar inappropriate inmate supervision pay at other CDCR facilities, including the recommendation to train all employees who receive, approve, or issue the inmate supervision pay.

4. By January 1, 2020, CalRecycle should update the baseline and goals for mattress recycling to reflect the most current available information it has related to the number of mattresses disposed of statewide. In addition, it should ensure that its recycling goals are statewide in scope by including information about recycling and renovation from entities that do not contract with the Mattress Council.

5. In order to bring violators of the recycling act into compliance and to ensure that its enforcement activities are timely, CalRecycle should do the following:

- Assess penalties for noncompliance with the recycling act.

- Publicize any penalties it assesses against violators of the recycling act as a deterrent to potential violators.

- Monitor inspection cases to ensure that it does not complete them before the retailers in question have remedied any instances of noncompliance.

- Execute a plan to verify compliance for all inspections in which it did not obtain evidence of compliance.

- Develop and implement a timeline for the penalty phase of the enforcement process.

- Regularly review the timeliness of its enforcement process and prioritize any overdue enforcement actions based on its enforcement timelines.

2. To ensure it can demonstrate that its fraud prevention efforts are maximizing financial recoveries for the beverage program, CalRecycle should both modify and annually update its fraud management plan to include the following:
- By December 31, 2014, formally establish a systematic process for analyzing, monitoring, and responding to the risk of fraudulent recycling of out-of-state beverage containers.
- Develop fraud estimates—by type of fraudulent activity—that quantify the potential financial losses to the beverage program and the methodology CalRecycle used to develop these estimates.
- Identify the amount of actual fraud in the prior year by type of fraudulent activity, such as the financial losses resulting from the redemption of out-of-state beverage containers or the falsification of reports used to substantiate program payments.
- Identify the amount actually recovered for the beverage program in the form of cash for restitution and penalties resulting from fraud.

3. To allow for public input and to prevent any legal challenges claiming that its policies and procedures regarding prepayment holds constitute unenforceable underground regulations, CalRecycle should adopt these policies and procedures as regulations in accordance with the Administrative Procedure Act.

1. Take appropriate corrective action against the district engineer and the supervisors for their participation in or failure to address the conflict of interest.

6. To ensure that the regional boards obtain adequate and consistent information on the storm water management costs local jurisdictions incur, the State Water Board should develop statewide guidance by August 2018 for local jurisdictions on methods for tracking the cost of storm water management. If the State Water Board believes it does not have the expertise to develop such guidance, it should hire or contract with an expert in municipal finance who can assist in developing that guidance.

7. If the State Water Board believes regulations are necessary to ensure that the regional boards and local jurisdictions follow its guidance regarding adequate and consistent information pertaining to their costs for storm water management, the State Water Board should adopt such regulations.

11. The State Water Board should revise its trash policy to focus it on local jurisdictions that have water bodies that are harmed by trash, as identified by the polluted waters list. In addition, the State Water Board should review the polluted waters list at least biannually to identify any additional water bodies recently determined to be harmed by trash and impose its trash policy on the applicable jurisdictions.

14. The commission should follow the requirement in state law to inspect and audit the accounting records of utilities it regulates within required time frames. If the commission chooses to continue to meet this requirement through the general rate case process, it should ensure that all utilities file a general rate case on a regular schedule so as to comply with the state law's audit requirement. However, the commission should follow alternate methods to comply with the audit requirement when a utility will not be filing for its general rate case in time to be audited within three or five years, depending on the timing of the required audit for that utility.

2. To ensure that it resolves complaints against passenger carriers in a timely manner, the commission should establish a method for prioritizing complaints and it should implement a policy specifying the maximum amount of time between the receipt of a complaint and the completion of the subsequent investigation. Further, the commission should require branch management to monitor and report regularly on its performance in meeting that policy.

6. To ensure that the branch conducts thorough investigations of passenger carriers, the commission should implement a formal training program to ensure that all investigators have adequate knowledge and skills related to regulating passenger carriers.

9. To ensure that passenger carriers submit accurate fee payments, the commission should require its fiscal staff to implement a process to verify passenger carrier fee payments and associated revenue.

10. To ensure that it complies with state law and uses passenger carrier fees appropriately, the commission should implement a process to ensure that passenger carrier fee revenues more closely match related enforcement costs.

12. To detect and deter carriers from operating illegally at airports, the branch should use as intended the five positions added for passenger carrier enforcement at airports. If the branch chooses not to designate five positions solely for this purpose, then it must be prepared to demonstrate regularly that an equivalent number of full-time positions are working on this activity.

3. To ensure that policy makers, enforcement officials, and the general public have access to accurate consumer complaint data in CIMS, the branch should continue to implement its quality management team program component focused on reviewing the categorization of complaints and correcting identified errors.

4. To ensure that policy makers, enforcement officials, and the general public have access to accurate consumer complaint data in CIMS, the branch should develop and implement tools by September 30, 2015, to measure the quality management team program's effectiveness.

11. The commission should ensure that it complies with all policy requirements in SAM Chapter 5300 no later than April 2016.

13. As part of developing, implementing, and maintaining an entitywide information security program, the commission should develop a risk management and privacy plan and conduct an assessment of risks facing its information assets.

16. The commission should revise its existing recovery plan to include a list of applications supporting critical business functions, their maximum acceptable outage time frames, and detailed recovery strategies for each application.

17. The commission should revise its existing recovery plan to include detailed procedures for rebuilding its technology infrastructure at an alternate processing site.

18. The commission should conduct regular tests and exercises to assess the sufficiency of the revised recovery plan and refine the plan when necessary.

2. To ensure that the choice of a vendor is sufficiently justified and that the vendor represents the best value, the CPUC should explain in its final decision how the vendor was the most qualified in all cases when the CPUC does not competitively select the vendor it directs utilities to contract with.

4. To avoid the appearance of inappropriate relationships, the CPUC should adopt a policy to prohibit commissioners from accepting gifts from regulated utilities and energy companies and free travel from organizations with significant ties to regulated utilities and other parties with financial interests in CPUC proceedings.

6. To ensure that its contracting practices align with state requirements and best practices, the CPUC should update, distribute, and follow its contracting procedures manual. The manual should identify specific responsibilities for both contracts office staff and project managers, and it should provide specific guidance about the processes the CPUC will employ to do the following:

-Fully justify civil service exemptions.

-Conduct market research for exempt contracts.

-Fully support the need for additional funding.

-Ensure that it does not change the scope of work too significantly from the original.

-Monitor contractor performance against criteria included in its contracts.

-Avoid sole-source contracts when it is able to solicit competitive bids for services.

7. To ensure that its contracting practices align with state requirements and best practices, the CPUC should provide immediate refresher training to its contract analysts and contracts office manager, and establish a regular schedule of annual training for them to attend.

8. To ensure that its contracting practices align with state requirements and best practices, the CPUC should designate a limited number of project managers for each division at the CPUC, and provide those individuals with training on the CPUC's processes related to contracting, including how to monitor progress of a contractor's work.

10. The CPUC should update its general policy on responding to California Public Records Act requests so that the policy aligns with state law.

13. The CPUC should update its regulations to require parties joining a proceeding by filing a protest or response to an application or petition, or by filing comments in response to a rulemaking proceeding to fully disclose their interests in the proceeding.

3. To show how air pollution emissions reductions related to the solar initiative benefit the State, the commission should include in future reports the measurable benefits of those reductions.

7. To further its mission to obtain the lowest possible rates for reliable and safe utility service for ratepayers through its reviews of balancing accounts, Ratepayer Advocates should use the commission's list of balancing accounts to guide its selection of the number, size, and type of balancing accounts to review so that its review coverage is more proportional across all utilities.

14. To ensure that IT projects have the oversight needed to better position them for success, CalTech should develop thresholds relating to IT project cost increases and schedule delays to inform and better justify its decision to allow an IT project to continue. If a department's IT project reaches or exceeds these thresholds, CalTech should require the department to conduct a cost-benefit analysis for the project and include this analysis in an SPR. CalTech should consider the results of this analysis in its decision to approve or deny the SPR and, if warranted, take action to suspend or terminate the project so that it does not allow projects with significant problems to continue without correction.

2. To ensure long-term efficient and effective delivery of projects, the division, in its planned implementation of its new project management system in July 2017, should ensure that the project management system can centrally track and extract all data regarding project status, including time delays, cost overages, and the reasons for each.

3. To ensure long-term efficient and effective delivery of projects, the division, in its planned implementation of its new project management system in July 2017, should track the reasons that projects are pending to identify its true backlog of projects. In doing so, it should develop a process to follow up on those projects that are pending to ensure that they are not on hold unnecessarily and are appropriately moving forward.

4. To ensure long-term efficient and effective delivery of projects, the division, in its planned implementation of its new project management system in July 2017, should, at least annually, use the centrally tracked data to identify common themes in the causes for project delays and cost overages and develop solutions to address these issues. Further, it should report the results of its review to General Services' executive management.

5. Until the division implements its planned project management system, it should, by September 2016, develop a process to, at a minimum, identify project status and reasons for project delays as well as cost overages. Using these data, the division should modify its project management processes to ensure the efficient and effective delivery of projects.

6. The division should develop and implement a process for preparing reasonable time frames and cost estimates for its projects within the building management branch. To better inform the development of this process, the division should evaluate the branch's structure, which should include a staffing analysis, to determine whether it is effectively organized and whether it should add cost estimator positions.

7. To ensure that client agencies are paying equitable rates, by December 2016 General Services should develop and implement a strategy for allocating its administrative costs equally among all the projects it completes for client agencies, including those portions outsourced to private firms.

8. To ensure that the project management branch charges its client agencies a competitive hourly rate, by December 2016 and every two years thereafter, the division should conduct a rate analysis that fully accounts for differences between the project management branch's rate and private firms' rates. If it finds that the rates are not competitive, the division should identify and implement strategies to ensure that the project management branch's rates are as competitive as they can be with those of its private firm counterparts. Further, the division should explore and implement any other reasonable methods to ensure that it is delivering projects as cost effectively as possible.

10. To improve its communication with client agencies, the division should develop a process for providing periodic detailed bills and invoices to client agencies clearly describing the work for which it is charging.

11. To effectively evaluate the performance of its branches in delivering projects, the division should develop meaningful goals and objectives and a method of measuring its success in achieving them as part of its strategic plan that is focused on ensuring that projects are delivered on time and within budgeted cost estimates.

12. To ensure that its project management staff are adequately trained and have the information necessary to deliver projects as efficiently and effectively as possible, the division should conduct a comprehensive survey every other year of all of its client agencies to inform necessary improvements to its processes and training program and, in the interest of transparency, make the survey results public.

3. To improve its oversight of the State's noncompetitive contracts, General Services should, within 90 days, create plans for regularly performing statewide analyses to identify potential abuse or overuse of noncompetitive contracts. These analyses should include, but not be limited to, calculating the proportional value and number of the State's competitive and noncompetitive contracts and amendments, examining trends in agencies' use of noncompetitive contracts and amendments, and identifying unusual patterns among vendors receiving state contracts through noncompetitive means.

2. To increase its efforts to prevent and control diabetes, Public Health should develop a process for identifying and applying for federal funding opportunities, including routinely and proactively searching for grants. In addition, Public Health should seek funding for a grants specialist position to identify and apply for federal and other grants.

3. To protect the health, safety, and well-being of residents in long-term health care facilities, Public Health should improve its oversight of complaint processing. Specifically, by May 1, 2015, Public Health should establish a specific time frame for completing facility-related complaint investigations and ERI investigations and inform staff of the expectation that they will meet the time frame. Public Health should also require district offices to provide adequate, documented justification whenever they fail to meet this time frame.

4. To protect the health, safety, and well-being of residents in long-term health care facilities, Public Health should improve its oversight of complaint processing. Specifically, by May 1, 2015, Public Health should develop formal written policies and procedures for PCB to process complaints about certified individuals in a timely manner. These policies and procedures should include specific time frames for prioritizing and assigning complaints to investigators, for initiating investigations, and for completing the investigations. Public Health should also inform staff of the expectation that they will meet these time frames. It should require PCB to provide adequate, documented justification whenever PCB fails to meet the time frames.

8. To protect the residents in long-term health care facilities from potential harm, Public Health should ensure that its district offices have adequate staffing levels for its licensing and certification responsibilities, including staffing levels that allow prompt investigations of complaints. Specifically, Public Health should continue working with CalHR to complete the reclassification of district offices' investigator supervisor and manager positions and then quickly fill the vacant positions at district offices.

8. To ensure it can provide effective oversight of labs as state law requires, Laboratory Services should address staffing issues by preparing and resubmitting to Public Health a recruitment and retention proposal, developing a succession plan, and taking necessary steps to implement its planned reorganization.

9. To ensure it can provide effective oversight of labs as state law requires, Laboratory Services should ensure that its information technology data systems have necessary safeguards, contain accurate and complete data, and support its program needs.

10. To ensure it can provide effective oversight of labs as state law requires, Laboratory Services should update and develop its regulations as necessary to ensure consistency with existing state law.

8. As the Legislature considers changes to state law, Public Health should take the steps necessary to ensure that its oversight results in nursing facilities improving their quality of care by amending its application licensing reviews by developing a defined process that specifies how an analyst will determine whether an applicant has demonstrated its ability to comply with state and federal requirements. This process also needs to ensure that analysts conduct complete and standardized reviews of each nursing facility application within each class of facility. Specifically, these processes should clearly outline what factors analysts will consider when determining whether an applicant is in compliance, how analysts will weigh those factors for each class of facility, and what objective thresholds will prompt analysts to elevate applications for review and approval by higher-level management. Additionally, Public Health should document the additional factors higher-level management will consider if applications are elevated for their review to ensure that Public Health conducts standardized reviews of nursing facility applications of the same class. Finally, Public Health should develop processes ensuring that it documents its decisions adequately.

9. As the Legislature considers changes to state law, Public Health should take the steps necessary to ensure that its oversight results in nursing facilities improving their quality of care by ensuring that it issues citations in a timely manner, especially for immediate jeopardy deficiencies.

11. To improve the availability and transparency of information, Public Health should upload all inspection findings to Cal Health Find and review ownership data by May 2019.

5. To ensure that it more effectively shares, receives, and uses administrative action information, Social Services should develop and maintain a centralized database containing its own administrative actions and those received from other state departments, in order to share this information among these departments as required by state law. Social Services should seek funding if it believes additional resources are necessary.

10. To ensure that Social Services evaluates the risk individuals may pose to vulnerable populations in its licensed care facilities as quickly as possible, by July 2017 Social Services should establish time frames for staff to evaluate individuals who are present in their facilities and who have received administrative actions from other departments. In addition, it should monitor and follow up with the appropriate staff regarding the status of their assessments of these individuals and their final decisions.

13. To comply with state law and better protect vulnerable populations in California's licensed care facilities, Social Services should immediately change its policy to require that its exemption analysts evaluate all infraction convictions, other than minor traffic violations, before granting exemptions to individuals. If Social Services believes it is not feasible to evaluate all of these convictions, it should report to the Legislature by June 2017 how it ensures that vulnerable populations are not at risk and should request that the Legislature change the law to eliminate infraction convictions as a crime category that Social Services must evaluate in order to grant an exemption.

16. Until the Legislature requires that Social Services receive both California and federal criminal history information before issuing a clearance or processing an exemption, to better protect vulnerable populations, Social Services should immediately revise its policy to require its regional offices to obtain all self-disclosure forms for individuals who submit fingerprints to Justice as part of an application to be present in a licensed facility. The regional offices should then forward to the CBCB all self-disclosure forms that identify a conviction.

17. Until the Legislature requires that Social Services receive both California and federal criminal history information before issuing a clearance or processing an exemption, to better protect vulnerable populations, Social Services should immediately change its practice of allowing individuals who have not submitted a self-disclosure form to Social Services to have access to licensed facilities, thus reflecting the requirements of state law. In addition, the CBCB should develop a process to ensure that individuals cannot receive a clearance or an exemption without the CBCB first receiving both California and federal criminal history information if a regional office does not have a self-disclosure form for the individual.

26. To ensure that regional offices pursue legal actions in a timely manner, by July 2017 Social Services' headquarters should identify a resource—such as a unit—to monitor and follow up with the regional offices regarding the status of their legal actions related to substantiated address matches of registered sex offenders at licensed facilities.

30. To improve the oversight of psychotropic medications prescribed to foster children, Social Services should collaborate with the counties and other relevant stakeholders—including Health Care Services, as necessary—to develop and implement a reasonable oversight structure that ensures the coordination of the State's and counties' various oversight mechanisms as well as the accuracy and completeness of the information in Social Services' data system. This structure should include at least the following items:
- Identification of the specific oversight responsibilities to be performed by the various state and local government agencies.
- An agreement on how county staff such as social workers, probation officers, and public health nurses will use printed Health and Education Passports to obtain foster children's necessary mental health information—including psychotropic medications and psychosocial services—for inclusion in Social Services' data system.
- A plan to ensure that counties have sufficient staff available to enter foster children's mental health information into Social Services' data system and the resources to pay for those staff.
- An agreement on the specific information related to psychotropic medication—including but not limited to the medication name, maximum daily dosage, and court authorization date—and psychosocial services and medication follow-up appointment information that county staff must enter into Social Services' data system for inclusion in foster children's Health and Education Passports.
- Specific directions from Social Services regarding the correct medication start dates and court authorization dates counties should include in its data system and foster children's Health and Education Passports.
- An agreement on the training or guidance Social Services should provide to county staff members working with Social Services' data system to ensure that they know how to completely and accurately update foster children's Health and Education Passports.
- An agreement on how the counties will use information on the new authorization forms that the Judicial Council approved to better oversee the prescription of psychotropic medications to foster children.
- An agreement regarding how counties will implement, use, or disseminate the educational and informational materials the Quality Improvement Project has produced, including the "California Guidelines for the Use of Psychotropic Medication with Children and Youth in Foster Care", "Questions to Ask About Medications", and the "Foster Youth Mental Health Bill of Rights".
- An agreement on the specific measures and the best available sources of data the State and counties will use to oversee foster children prescribed psychotropic medications, including psychosocial services and medication follow-up appointments.
- An agreement on how the State and counties will oversee psychotropic medications prescribed to foster children by fee-for-service providers who are not affiliated with county Medi-Cal mental health plans.
- An agreement on the extent of information related to psychotropic medications prescribed to foster children that counties will include in the self-assessments, system improvement plans, and annual progress reports they develop as part of Social Services' California Child and Family Services Reviews.
- An agreement on the extent of the information related to psychotropic medications prescribed to foster children that counties will include in their responses to Health Care Services' reviews, including its county Medi-Cal mental health plan compliance reviews and external quality reviews.

40. To promote continued improvement in the CWS system, Social Services should encourage each county CWS agency to designate personnel to update regularly their policies and procedures, to include a detailed description of the need for ongoing supervisory reviews of key aspects of their respective service processes and incorporate that description into their policies and procedures, and to designate personnel to perform regular quality assurance reviews.

41. To promote continued improvement in the CWS system, Social Services should ask each county CWS agency to report to Social Services on the status of these efforts within 60 days, six months, and one year from the publication of this audit report.

4. To ensure that counties' use of foster family agency placements is justified, Social Services should take action to implement the recommendation we previously made in our 2011 audit. Specifically, Social Services should require counties to give licensed foster homes a higher priority than foster family agencies for children that do not have identified treatment needs.

5. To ensure that counties' use of foster family agency placements is justified, Social Services should take action to implement the recommendation we previously made in our 2011 audit. Specifically, Social Services should require counties to prepare a detailed justification for any child placed with a foster family agency.

4. To ensure that all counties consistently gauge the cost-effectiveness of their early fraud detection activities and ongoing investigation efforts for the CalWORKs and CalFresh programs, Social Services should develop a formula to regularly perform a cost-effectiveness analysis using information that the counties currently submit. Specifically, this formula should measure the savings that a county achieves for each dollar spent on antifraud efforts.

5. To make certain that counties receive the greatest benefit from the resources they spend on antifraud efforts related to CalWORKs and CalFresh cases, Social Services should, using the results from the recommended cost-effectiveness analysis, determine why some counties' efforts to combat welfare fraud are more cost-effective than others.

6. To make certain that counties receive the greatest benefit from the resources they spend on antifraud efforts related to CalWORKs and CalFresh cases, Social Services should seek to replicate the most cost-effective practices among all counties. Social Services should work with its legal counsel to determine whether to withhold information about these practices from public disclosure.

7. Social Services should track counties' prosecution thresholds for welfare fraud cases and determine whether they affect counties' decisions to investigate potential fraud, with a focus on determining best practices and cost-effective thresholds. If Social Services' analysis determines that varying prosecution thresholds do affect counties' decisions, it should then work with counties to implement the consistent use of these cost-effective prosecution thresholds.

9. To make certain that counties receive the greatest benefit from the resources they spend on antifraud efforts related to CalWORKs and CalFresh cases, Social Services should address and promptly act on the four remaining recommendations that its steering committee provided in 2008.

12. To make counties' review of match lists more efficient, Social Services should revive its efforts to work with the state and federal agencies that prepare the match lists to address the counties' concerns about match list formats, content, and criteria.

2. To reduce the likelihood of inadvertent errors on replacement timesheets, Social Services should create functionality within CMIPS II to allow replacement timesheets to be printed with data that had been submitted correctly on the original timesheet. Social Services should develop a plan by August 2017 that outlines actions, such as assessing the cost and seeking funding from the Legislature if necessary, that will be taken to create the functionality.

10. To enable it to track whether EDD is meeting its contractual time frame for printing and mailing timesheets, Social Services should either modify its current agreement or require in the renewal of its agreement a method for tracking the time required to print and mail timesheets. Social Services should also perform monthly reviews of the activities performed by EDD and SCO to ensure compliance with the time frames for each agreement. Additionally, Social Services should implement a process to regularly test EDD and SCO processes to ensure that they are within the required time frames.

13. Until state law is changed to facilitate providers' efforts to report their time and to reduce the potential for providers to be inadvertently suspended from the IHSS program, Social Services should inform providers of the weekly maximum number of service hours for each variation in the length of the month, rather than using a standard conversion that results in providers claiming more hours than their recipients are authorized.

14. If the Legislature amends state law as we recommend, Social Services should modify the timesheet format to incorporate the weekly authorization for services and the new two-workweek pay period. Social Services should also reconfigure its timesheet to require that all information be entered on one side of the document, including the signatures of the provider and recipient.

2. To ensure timelier fee assessments, Developmental Services should hold regional centers accountable for providing the monthly placement reports and copies of information letters required by state regulations. To encourage compliance, Developmental Services should specify in its regional center contracts that noncompliant regional centers will pay financial penalties equal to the amount of revenue lost because of their inaction.

4. To ensure that regional centers are aware of the benefits, including cost savings to the State that can be realized by using FMS vendors, DDS should formally communicate to regional centers regarding the model.

5. To ensure that in-home respite vendors are providing quality services and that vendors are adhering to state requirements, DDS should issue regulations requiring regional centers to conduct periodic and ongoing reviews of vendors' programs, employees, and consumer records.

6. To ensure that in-home respite vendors comply with vendor requirements on an ongoing basis, DDS should require the regional centers to develop a process to conduct biennial reviews of the vendor files the regional centers maintain and document the outcome of the review in the files. DDS should require the regional centers to take appropriate action to ensure that vendors comply, up to and including terminating the vendorization, if necessary.

7. To ensure that it is providing oversight in accordance with state law and federal requirements, DDS should ensure that it performs audits of each regional center every two years as required. In conducting these audits, DDS should consistently include a review of in-home respite services.

6. If Health Care Services finds significant errors in a health plan's provider directory, it should work with that health plan to identify reasons for the inaccuracies and require the health plan to develop processes to eliminate the inaccuracies.

1. To ensure that it provides claiming units with reasonable opportunities to address concerns with its decisions or actions, Health Care Services should, within three months, begin preparing regulations to establish and implement a formal appeals process that allows claiming units to directly appeal Health Care Services' decisions.

2. To ensure that it provides claiming units with reasonable opportunities to address concerns with its decisions or actions, Health Care Services should, within three months, inform all stakeholders, including claiming units, of the existence of this appeals process.

3. Until the Legislature implements our recommendation in Chapter 2, Health Care Services should immediately resolve weaknesses in its oversight of local educational consortia and local governmental agencies to ensure that these entities sufficiently meet their responsibilities under the administrative activities program and meet the terms of their contracts with Health Care Services. Health Care Services should update its site review and desk review procedures to include the following steps:
* A risk-based approach to selecting entities for review.
* Verification that local educational consortia and local governmental agencies are adequately meeting the oversight and administrative responsibilities described in their contracts with Health Care Services.
* Verification that contracts between local educational consortia or local governmental agencies and their claiming units do not include provisions that could result in disallowed costs, such as allowing Health Care Services' participation fee to be included in the claim calculations.
* Examination of local educational consortia and local governmental agencies' records to ensure that:
-- Costs they claim for federal reimbursement are necessary and reasonable.
-- The entities are not inappropriately earning a profit based on the fees they collect from claiming units.
-- The coding performed by local educational consortia that charge claiming units a percentage of their federal reimbursement is reasonably accurate.

4. Until the Legislature implements our recommendation in Chapter 2, Health Care Services should immediately resolve weaknesses in its oversight of local educational consortia and local governmental agencies to ensure that these entities sufficiently meet their responsibilities under the administrative activities program and meet the terms of their contracts with Health Care Services. Health Care Services should complete the oversight reviews for at least three high-risk local educational consortia or local governmental agencies by December 31, 2015, and post the results to its website.

5. Until the Legislature implements our recommendation in Chapter 2, Health Care Services should immediately resolve weaknesses in its oversight of local educational consortia and local governmental agencies to ensure that these entities sufficiently meet their responsibilities under the administrative activities program and meet the terms of their contracts with Health Care Services. Health Care Services should complete the oversight reviews for any remaining high-risk local educational consortia or local governmental agencies by June 30, 2016, and post the results to its website.

8. To minimize the risk that claiming units could include unallowable costs when calculating their reimbursement claims, Health Care Services should remind all local educational consortia and local governmental agencies that contracts with their claiming units should prohibit claiming units from seeking federal reimbursement of Health Care Services' participation fee.

10. To streamline the organizational structure of its administrative activities program and to improve the program's cost-effectiveness, Heath Care Services should implement a single statewide quarterly random moment time survey and develop and implement a plan to take over responsibility for conducting quarterly time surveys and performing related activities as soon as reasonably possible.

11. To streamline the organizational structure of its administrative activities program and to improve the program's cost-effectiveness, Heath Care Services should implement a single statewide quarterly random moment time survey and develop and issue a request for proposals to identify a responsible vendor to assist in implementing a statewide quarterly random moment time survey.

12. To streamline the organizational structure of its administrative activities program and to improve the program's cost-effectiveness, Heath Care Services should implement a single statewide quarterly random moment time survey and draft revisions to regulations as appropriate and to applicable documents, including the manual, oversight strategies and plans, and policy and procedure letters.

13. To the extent that local educational consortia and local governmental agencies are no longer involved in the administrative activities program, Health Care Services should develop and issue a standard contract for claiming units to sign to participate in the program.

17. To better maximize federal reimbursements for the administrative activities program, Health Care Services should, within six months, develop and implement a method to oversee and track the outreach efforts that local educational consortia and local governmental agencies use for ensuring that nonparticipating claiming units understand the benefits and consider participating in the administrative activities program.

21. To provide the public with the ability to participate fully in developing the rules governing the administrative activities program, Health Care Services should, in accordance with California's Administrative Procedure Act (APA), immediately develop and adopt the regulations cited in the four subdivisions of Section 14132.47 of the California Welfare and Institutions Code.

5. To help increase the number of providers participating in the program's fee-for-service delivery system, Health Care Services should improve its identification and implementation of changes that minimize or simplify administrative processes for providers. These changes should include revising its processes pertaining to dental procedures that require radiographs or photographs.

20. To make certain that it meets the requirements of the new state law and that its performance measures are accurate, Health Care Services should establish the provider-to-beneficiary ratio statewide and by county as performance measures designed to evaluate access and availability of dental services and include this measure in its October 2015 report to the Legislature.

23. To ensure that Health Care Services and its fiscal intermediaries reimburse providers only for services rendered to eligible beneficiaries, Health Care Services should obtain Social Security's Death Master File and update monthly its beneficiary eligibility system with death information.

2. To recover inappropriately spent funds, prevent future erroneous payments, and ensure eligible individuals' access to care, Health Care Services should resolve the discrepancies we identified and recover erroneous payments where allowable by June 30, 2019.

3. To prevent future erroneous payments, Health Care Services should implement procedures by December 31, 2018, to ensure the timely resolution of system discrepancies. These procedures should include Health Care Services regularly following up on recurring, unresolved system discrepancies with the responsible county.

4. To prevent future erroneous payments, Health Care Services should establish procedures by December 31, 2018, that define when it will use its authority as defined in state law to sanction unresponsive counties that do not remedy known discrepancies.

5. To assist counties in addressing discrepancies, Health Care Services should find a cost-effective method to provide its exception reports in an electronic format readable by common database and spreadsheet software products that would allow users to sort and filter the data readily by December 31, 2018.

6. To assist counties in addressing discrepancies, Health Care Services should reevaluate and update its guidance to the counties related to prioritizing MEDS alerts by December 31, 2018.

1. To effectively monitor MHSA spending and provide guidance to the local mental health agencies, Health Care Services should publish its proposed regulations in the California Regulatory Notice Register by June 2018 and subsequently develop an MHSA fiscal reversion process to ensure that the State can reallocate any MHSA funds that local mental health agencies do not spend within the statutory reversion time frames to other local mental health agencies that are better positioned to use the funds to meet the MHSA's intent.

3. To effectively monitor MHSA spending and provide guidance to the local mental health agencies, Health Care Services should publish its proposed regulations in the California Regulatory Notice Register by June 2018 and subsequently establish and enforce an MHSA reserve level that will allow local mental health agencies to maintain sufficient funds to continue providing crucial mental health services in times of economic hardship, but that will not result in them holding reserves that are excessive. Health Care Services should also establish controls over local mental health agencies' deposits and withdrawals to their reserves.

5. To ensure that it provides effective oversight of local mental health agencies' reporting of MHSA funds, Health Care Services should publish its proposed regulations in the California Regulatory Notice Register by June 2018. Health Care Services should then subsequently implement a process that will enable it to withhold MHSA funds from local mental health agencies that fail to submit their annual reports on time.

6. To ensure that local mental health agencies appropriately spend MHSA funds, Health Care Services should publish its proposed regulations in the California Regulatory Notice Register by September 2018. It should then develop and implement an MHSA fiscal audit process, independent of the Medi-Cal reviews, to review revenues and expenditures for the most recent fiscal year.

12. Health Care Services should use current data to revise and update the peer groups it uses to set Medi-Cal rates. In doing so, it should take into consideration the consolidation of the nursing facility industry.

2. To ensure that management and staff involved in the grant process are sufficiently informed about the process and their responsibilities, Rehabilitation should require these employees to attend a kickoff meeting before the development of each RFA in which participants discuss the key stages of the grant review process, each individual's roles and responsibilities, and requirements surrounding conflicts of interest and confidentiality. Further, it should record these discussions in meeting minutes to ensure that expectations of employees are clearly defined and documented.

3. To comply with state laws and regulations and help ensure that staff involved in making governmental decisions during the grant process are impartial, Rehabilitation should ensure that they receive ethics training, which includes conflict-of-interest training, at least every two years.

4. To help ensure that staff involved in the grant process adequately protect confidential information, Rehabilitation should develop confidentiality procedures for each grant. Further, it should ensure that staff involved in the grant process sign the conflict-of-interest and confidentiality forms before the development of the RFA for each grant.

5. To ensure that it has received sufficient input and feedback from the disability community to inform the development of RFAs, Rehabilitation should solicit and document stakeholder input and feedback before and during the development of each RFA.

6. To increase transparency and ensure that applicants have the information necessary to understand the grant process, Rehabilitation should include in its RFAs clear scoring criteria and descriptions of the evaluation, award, and appeals processes, including the process it will use to address applications that receive tied scores.

7. To ensure that Rehabilitation maintains all relevant grant documentation and responds fully to requests for public records, it should immediately adhere to its records retention policy and save all grant-related documents, including email correspondence and attachments, to a centralized location.

9. To ensure consistency and fairness in the evaluation process, Rehabilitation should make sure that it accepts only complete applications submitted before the deadline, unless otherwise specified in the RFA. If the RFA specifies a hard deadline and applicants submit incomplete applications, Rehabilitation should not accept any portions of the applications submitted after the deadline and should assess the penalty for incomplete applications specified in the RFA.

10. To help ensure that evaluators adequately protect confidential information and that the evaluation process is fair, Rehabilitation should develop standardized evaluator training for confidentiality procedures and conflicts of interest, including a discussion of bias or the appearance of bias. Rehabilitation should also ensure that the candidates receive this training and sign conflict-of-interest and confidentiality forms before it selects evaluators. Further, it should prohibit program staff who participate in the development of an RFA from acting as evaluators for the applications Rehabilitation receives in response to that RFA.

11. To increase the transparency of its selection process and to ensure that it receives the most qualified evaluators possible, Rehabilitation should issue a public solicitation for evaluators for each grant that includes a description of essential and desirable qualifications.

12. To ensure that evaluators have the information necessary to sufficiently and fairly assess and score applications, Rehabilitation should develop training by December 2018 that can be tailored to each grant and includes at minimum the following topics:

-The purpose and relevant regulatory requirements for the grant.

- Instructions on how to score applications, including an applicant's financial information, and direction that they must provide comments to support their scores.

Rehabilitation should provide this training to evaluators before allowing them to score applications.

13. To ensure that it provides sufficient oversight of the grant process, Rehabilitation should ensure that the technical review teams its assigns to grants provide the director and chief deputy with a memorandum summarizing the evaluation process and the evaluators' recommended grant awardees. Rehabilitation should also designate an individual responsible for reviewing and approving the memorandum and recommended awardees before it publishes its notice of intent to award.

14. If it finds errors in an evaluation that merit restarting the grant process, rescoring of applications, or convening a new evaluation panel, Rehabilitation should resolve any issues before it begins the rescoring process. It should also notify applicants to ensure that they are aware of any changes to the process due to the errors. Further, it should consider promulgating regulations and amending its grant manual to permit staff to request evaluators to rescore applications or convene a new evaluation panel when it finds issues with an evaluation.

15. To ensure that it consistently and thoroughly evaluates appeals, Rehabilitation should establish in state regulations and its grant manual that staff at the appropriate level of authority are to acknowledge all appeal requests, notify intended awardees that could be affected by the appeals, and inform the appellant of the qualifications of the review committee members. Staff at the appropriate level of authority must also notify all affected parties of the review committee's final decision within the time frame Rehabilitation establishes in regulations.

16. To ensure that Rehabilitation has appropriate oversight of its grant process and can sufficiently demonstrate that it followed the process, it should designate staff, separate from those involved in the respective grant process, to conduct a review of each grant process for procedural errors, evaluator prejudice, and whether evaluators supported their scores with evidence from the relevant applications before it awards grants.

17. To comply with federal and state requirements, and to ensure consistency and fairness in its grant process, Rehabilitation should revise and formalize the policies and procedures in its grant manual to incorporate the rules adopted by regulation and to address the recommendations in this report. The grant manual should specify that any deviations from the required grant process must be for good cause and be documented.

18. To ensure that it consistently and thoroughly evaluates appeals, Rehabilitation should establish in state regulations and its grant manual a process for the review committees to request additional information from appellants or program staff. To allow time for an adequate review of any additional information, Rehabilitation should consider extending the time for review committees to issue their decision on appeals from 30 days to 45 days.

19. To ensure that it consistently and thoroughly evaluates appeals, Rehabilitation should establish in state regulations and its grant manual that to be able to rescore applications when necessary, the review committee members should be subject-matter experts or, if they are not subject-matter experts, the review committee should have the authority to recommend a new evaluation panel instead of rescoring applications itself when it identifies a reason to invalidate previous evaluations.

10. To ensure that the MHSA-funded triage grants are effective, the Oversight Commission should require that local mental health agencies uniformly report data on their uses of triage grants. It should also establish statewide metrics to evaluate the impact of triage grants by July 2018.

10. To ensure that it provides the public with nursing facility information that is accurate and comprehensible, Health Planning should update its regulations to do the following:

- Append additional schedules to the template for the annual cost report to enable nursing facilities to fully disclose related-party transactions.

- Provide a single location in the annual cost report template for nursing facilities to enter related-party transaction amounts next to the amounts they are claiming for Medi-Cal reimbursement.

- Create an additional schedule in the cost report template that depicts how a company is investing in quality-of-care improvements.

70. Channel Islands should immediately begin following its policies to conduct departmental self-audits to identify and address safety concerns in its laboratories. Further, Channel Islands should amend its chemical plan to include specific expectations about how often departments and its EH&S office will conduct self-audits.

3. To comply with statutory reporting requirements on its efforts to serve students with disabilities, by June 2018, the Chancellor's Office should establish and report on a system for evaluating state-funded programs and services for disabled students, including its gathering of outcome data, staff and student perceptions of program effectiveness, and data on the implementation of the program.

4. To ensure that students with disabilities have equal access to instructional materials, by June 2018, the Chancellor's Office should develop guidance for the community colleges on periodically monitoring the accessibility of instructional materials and on providing training to all instructors in making their materials accessible to students with disabilities.

5. To ensure that community colleges' websites comply with accessibility guidelines, by September 2018, the Chancellor's Office should provide guidance to colleges on establishing policies and procedures to monitor the accessibility of their websites. Additionally, by September 2018, the Chancellor's Office should provide guidance on best practices for colleges to use in preventing their websites from containing inaccessible information.

7. To assist all community colleges in increasing transparency of their shared governance decision-making processes, by September 2018, the Chancellor's Office should issue guidance to the community colleges on establishing procedures to document the attendees, input received, and agreements reached during department meetings, including those to consider technology equipment requests.

62. Sacramento should monitor the implementation of its new processes for inspecting safeguards to ensure that it completes monthly flushes of eyewashes and showers as state regulations require.

38. To ensure the health and safety of employees working with hazardous materials, San Diego should, by December 2018, make the required trainings available to employees and establish procedures for ensuring that the employees have received all required trainings.

42. To ensure the health and safety of employees working with hazardous materials, San Diego should, going forward, regularly monitor employee training records to ensure that all employees have received the required trainings.

56. To ensure the health and safety of students in a laboratory setting, beginning in the Fall 2018 semester, San Diego should perform reviews at least annually to ensure that all departments are using the student training acknowledgement forms and are complying with the retention requirement.

67. San Diego should continue to implement its new policy to regularly review open work orders to ensure that it closes work orders in a timely fashion.

73. San Diego should ensure that it documents all self-audits it conducts, including when it does not identify any violations during the audit. Further, San Diego should continue to implement and follow its new process to include reviews of safeguard inspections as a part of its self-audits.

52. To ensure the health and safety of students in a laboratory setting, beginning in the Fall 2018 semester, Sonoma should require departments to have those students required to wear PPE sign the student safety training acknowledgement forms to demonstrate that they have received proper laboratory safety training.

71. Sonoma should immediately begin following its policies to conduct departmental self-audits to identify and address safety concerns in its laboratories. Further, Sonoma should ensure that its self-audits review whether timely flushes of eyewashes and showers have occurred. As part of self-audits, Sonoma's departments should ensure that fume hoods have received annual inspections. Finally, Sonoma's EH&S department should regularly review whether departments are conducting self-audits.

4. Once it has developed the health and safety reporting template and campuses have used it to submit their reports, the Chancellor's Office should assess the data and information in the reports to identify trends, risks, and best practices.

5. Once it has developed the health and safety reporting template and campuses have used it to submit their reports, the Chancellor's Office should develop recommendations for improving campus health and safety and follow up on the campuses' implementation of any corrective actions related to these recommendations.

6. Once it has developed the health and safety reporting template and campuses have used it to submit their reports, the Chancellor's Office should incorporate the risks identified in its assessments into the University Auditor's audit plan to ensure that the University Auditor evaluates problem areas related to campus health and safety.

7. Once it has developed the health and safety reporting template and campuses have used it to submit their reports, the Chancellor's Office should follow up with campuses that fail to submit the required annual health and safety reports and take appropriate steps to ensure compliance with this requirement.

10. To ensure the health and safety of employees working with hazardous materials, the Chancellor's Office should prescribe the frequency for which the campuses provide refresher laboratory safety training to employees.

58. As part of the uniform health and safety-reporting template that we recommend that it develop, the Chancellor's Office should require campuses to annually report on the timeliness of their inspections of safeguards, engineering controls, and ventilation systems and identify the reasons for any delays.

59. Based on campuses annual reports on the timeliness of their inspections of safeguards, engineering controls, and ventilation systems, the Chancellor's Office should follow up with campuses that report untimely inspections and should require that the campuses develop action plans to ensure that they complete inspections as often as state regulations require.

3. To improve the oversight of CSU's management personnel, the Chancellor's Office should work with campuses, bargaining unit representatives, the Public Employment Relations Board, and others as necessary to come to an agreement on the appropriate classification of coaches. The Chancellor's Office should take into account the concerns that San Diego State has raised about the labor market for these employees.

10. The Chancellor's Office should finish developing the Common Human Resources System and implement it as scheduled by December 2019.

5. To determine the amount of money that it can reallocate to campuses and to ensure that it publicly presents comprehensive and accurate budget information, by April 2018 the Office of the President should implement our recommended budget presentation shown in Figure 11 on page 40. Specifically, the Office of the President's budget presentation to the regents should include a comparison of its proposed budget to its actual expenditures for the previous year. It should also include all its expenditures and identify changes to the discretionary and restricted reserves. The Office of the President should combine both the disclosed and undisclosed budgets into one budget presentation.

22. To determine the amount of money that it can reallocate to campuses and to ensure that it publicly presents comprehensive and accurate budget information, by April 2019 the Office of the President should continue to present a comprehensive budget based on the presentation in Figure 11 to the regents, the Legislature, and the public.

23. To ensure that its staffing costs align with the needs of campuses and other stakeholders, by April 2019 the Office of the President should set targets for any needed reductions to salary amounts using the results from its public and private sector comparison and adjust its salaries accordingly.

32. To determine the amount of money that it can reallocate to campuses and to ensure that it publicly presents comprehensive and accurate budget information, by April 2020 the Office of the President should evaluate its budget process to ensure that it is efficient and has adequate safeguards that ensure that staff approve and justify all budget expenditures. If the Office of the President determines that its safeguards are sufficient, it should begin developing a multiyear budget plan.

34. To ensure that its staffing costs align with the needs of campuses and other stakeholders, by April 2020 the Office of the President should adjust its salary levels and ranges to meet its established targets.

36. To ensure that its staffing costs align with the needs of campuses and other stakeholders, by April 2020 the Office of the President should reallocate funds to campuses when adjustments to its salaries and benefits result in savings.

38. To ensure that its staffing costs align with the needs of campuses and other stakeholders, by April 2020 the Office of the President should implement phases four and five of CalHR's best practice workforce planning model by implementing its workforce plan strategies and annually evaluating the completed workforce plan strategies against defined performance indicators and revising the plan where necessary.

39. To ensure that its staffing costs align with the needs of campuses and other stakeholders, by April 2020 the Office of the President should report to the regents on the amount of funds it reallocates to campuses as a result of implementing our recommendations.

8. To ensure that the university achieves its goals of obtaining services at the lowest cost or best value and of providing vendors with fair access to contracting opportunities, the Office of the President should revise the university's contract manual to incorporate the best practices found in the State Contracting Manual for limiting the use of amendments to repeatedly extend existing contracts.

9. To ensure that the university achieves its goals of obtaining services at the lowest cost or best value and of providing vendors with fair access to contracting opportunities, the Office of the President should revise the university's contract manual to narrow the exemption from competition to only selected professional services, similar to the State Contracting Manual.

15. To maximize benefits from the systemwide procurement initiative and to ensure that the university uses those benefits for its teaching, research, and public service missions, the Office of the President should study ways to measure actual procurement benefits—possibly focusing this effort on benefits from larger dollar amounts—and if such measurement is not possible, it should clearly disclose to the regents and the public that the amounts it reports are based on estimates.

16. To maximize benefits from the systemwide procurement initiative and to ensure that the university uses those benefits for its teaching, research, and public service missions, the Office of the President should, if actual benefits are measurable, implement a process to monitor and report annually to the regents the estimated and actual benefits.

4. To ensure prompt resolution of sexual harassment complaints against faculty respondents, the Office of the President should do the following:

After the Academic Senate develops written requirements to specify exact time frames, complete an annual review of all cases involving Senate faculty to determine the length of time the adjudication process lasted. If an adjudication process takes longer than the time frames specified, the Office of the President should work with the Regents and the Academic Senate to develop further measures to enforce a more prompt adjudication process. The Office of the President should complete its first review by October 2020.

24. To address any patterns or systemic problems of sexual harassment, by July 2019 the systemwide office should work with each campus to develop and implement processes and data reports to assist the campus in regularly identifying patterns and systemic problems related to sexual harassment and in instituting sexual harassment prevention education and training in those areas that need it.

25. To address any patterns or systemic problems of sexual harassment, by July 2019 the systemwide office should work with each campus to implement ongoing data quality control processes in order to ensure sexual harassment complaints data are accurate and complete.

26. To address any patterns or systemic problems of sexual harassment, by July 2019 the systemwide office should identify and review campuses' complaints data to identify outliers in their use of the formal, informal, and administratively closed processes.

1. To meet its commitment to California residents, the university should replace its "compare favorably" policy with a new admission standard for nonresident applicants that reflects the intent of the Master Plan. The admission standard should require campuses to admit only nonresidents with admissions credentials that place them in the upper half of the residents it admits.

2. To meet its commitment to California residents, the university should amend its referral process by taking steps to increase the likelihood that referred residents ultimately enroll.

3. To ensure that campuses' interpretations of admission standards do not adversely impact residents, the university should implement a thorough process to annually evaluate the qualifications of students who apply and students who are admitted. These evaluations should highlight instances when campuses admit nonresidents who are less qualified than residents and should include corrective action steps. Moreover, this evaluation should include resident and nonresident undergraduate enrollment in majors at each campus. The university should make the results of this evaluation—including details of the academic qualifications of students who applied and who were admitted—publicly available.

8. To ensure the reasonableness of the compensation the university pays its executives, it should include—to the extent possible— all items of compensation when setting or adjusting salaries and benefits, when conducting surveys and studies, and when comparing the compensation packages of its executives to those in similar positions outside the university.

10. To improve the transparency and timeliness of its annual compensation report, the university should streamline the process it uses to prepare the report so it can be issued by April of each year.

12. To maximize the savings and new revenue from the Working Smarter initiative and ensure that the university uses them for its academic and research missions, the Office of the President should immediately require that the campuses fully participate in all projects unless they can provide compelling evidence demonstrating a harmful effect.

13. To maximize the savings and new revenue from the Working Smarter initiative and ensure that the university uses them for its academic and research missions, the Office of the President should, by June 30, 2016, to the extent possible, implement a process to centrally direct these funds to ensure that campuses use them to support the core academic and research missions of the university.

14. To maximize the savings and new revenue from the Working Smarter initiative and ensure that the university uses them for its academic and research missions, the Office of the President should ensure that it substantiates that projects are actually generating savings and new revenue and that it can demonstrate how the university uses these funds.

15. To ensure that its recruiting efforts benefit residents, the university should prioritize recruiting residents over nonresidents. In particular, the university should focus its recruiting efforts broadly to ensure that it effectively recruits resident underrepresented minorities. For example, the university could establish a limit on the amount of funds it dedicates to nonresident recruiting. Further, it should develop a process to better track its nonresident and resident recruiting expenditures.

16. To determine if the campuses are using funds to further the goals of the University of California system and the Legislature, the Office of the President should begin regularly monitoring and analyzing how campuses are using both state funds and nonresident supplemental tuition. If, after the close of the fiscal year, the Office of the President determines that campuses are not using state funds and/or nonresident supplemental tuition in accordance with those goals, the Office of the President should take steps to correct the campuses' spending decisions as soon as possible.

17. To ensure that it spends state funds prudently for programs that do not directly relate to educating students, the university should track spending from state funds for programs that do not relate to educating students.

18. To ensure that it spends state funds prudently for programs that do not directly relate to educating students, the university should reevaluate these programs each year to determine whether they continue to be necessary to fulfill the university's mission.

19. To ensure that it spends state funds prudently for programs that do not directly relate to educating students, the university should explore whether the programs could be supported with alternate revenue sources.

22. To ensure that its rebenching efforts lead to equalized per-student funding among the campuses, the university should adopt a methodology that it can use, at least every three to five years, to update its weighting system to ensure the weight factors take into account campuses' actual costs of instruction, using the cost study that we recommend in Chapter 1 and other revenue sources if necessary.

23. To ensure that its rebenching efforts lead to equalized per-student funding among the campuses, the university should exclude from its rebenching calculation all state funding it uses for programs that do not directly relate to educating students. The university should exclude these programs only after it has evaluated them in accordance with the recommendation we made previously.

7. To ensure the ongoing accountability of the Office of the President, the regents should require it to implement our recommendations and report periodically on its progress.

14. To ensure that the Office of the President's staffing levels are justified and that costs are reasonable and align with the needs of campuses and other stakeholders, the regents should require the Office of the President to implement our recommendations and report periodically on its progress.

15. To ensure that the Office of the President is engaging in a thorough review of its systemwide and administrative costs and implementing our recommendations, the regents should develop a contract for an independent third party that can assist the regents in monitoring implementation of the three-year corrective action plan for the Office of the President. The independent third party should have expertise in higher education, public administration, and public finance. Moreover, the independent third party should have complete access to the Office of the President's documentation and its staff so that it has sufficient and appropriate information to verify the Office of the President's actions. The independent third party should report to the regents on the Office of the President's progress, challenges, and barriers to success at least quarterly.

4. UC Davis should collect all late fees that its licensees owe.

9. To help ensure that university faculty and staff do not mishandle student reports of incidents, all faculty and staff should receive training annually, consistent with their role, on their obligations in responding to and reporting incidents of sexual harassment and sexual violence.

21. All universities should provide their education on sexual harassment and sexual violence to incoming students as close as possible to when they arrive on campus but no later than the first few weeks of their first semester or quarter. Further, universities should provide periodic refresher educational programs, at least annually, to all students on campus to ensure that they are aware of how to handle and report incidents of sexual harassment and sexual violence.

1. To strengthen its administrative reviews and help ensure that school food authorities comply with the Buy American requirement, Education should update its written procedures to include a requirement that reviewers collect and retain evidence for all items they evaluate for compliance with the Buy American requirement. This update should occur no later than October 1, 2017.

4. To make its appeal process more accessible to families who may not receive a satisfactory resolution from its contractors, Education should, by October 2018, require that its contractors share key information in their communications with families about the process for appealing notices. The required information should include valid grounds for a family to file an appeal as well as information or documentation Education would need in order to review the family's appeal of adverse decisions regarding their child-care services. Education should also require contractors to incorporate this information into contractually mandated staff training and into publicly available policies and procedures.

11. To ensure that its contractors can effectively make program improvements and maintain successes in ways that are meaningful to their stakeholders, Education should adopt measures to ensure its contractors follow the terms of their contracts by demonstrating that their board members conduct a critical appraisal of each education program.

15. To ensure the appropriate use of state grant funds, Education should determine, to the extent possible, the amount of supplemental plan funds that did not comply with federal requirements, and it should require 4Cs to reimburse the State for improper payments of state funds it made to the supplemental plan.

23. To better understand the condition of school libraries statewide and to raise stakeholders' awareness of the State Education Board's adopted model standards, Education should identify school districts that reported employing significantly fewer teacher librarians in fiscal year 2015-16 than in previous years and verify the accuracy of their fiscal year 2015-16 reports.

8. To ensure that districts, county offices, and schools receive guidance on a variety of safety issues and to comply with state law, CDE and DOJ should resume their partnership activities, as required by state law. Further, the partnership should update the 2002 handbook, "Safe Schools: A Planning Guide for Action," and distribute it to all districts and county offices. If CDE or DOJ determine the need for additional funds to implement the legislative recommendations or to reestablish the partnership's activities, they should request those funds from the Legislature.

3. To ensure that all LEAs comply with federal special education requirements, Education should require them to include directly in a student's IEP document reasons for any changes to student placement or services.

8. To enable it to review additional areas of its special education program for quality assurance, Education should collect information about the frequency of the provision of each service contained in all students' IEPs. Education should then use this information to annually review the frequency of mental health services and follow up with SELPAs when it observes a significant reduction in the frequency of services.

9. To ensure that LEAs comply with federal and state requirements, Education should require all LEAs to use the IEP document to communicate the rationale for residential treatment and any potential harmful effects of such placement.

18. Education should analyze and report to the Legislature, by May 30, 2016, on the outcomes for students receiving mental health services statewide, including outcomes across the six performance indicators we identified, in order to demonstrate whether those services are effective. Once it has reported this statewide information, Education should provide each LEA throughout the State a report regarding the outcomes for the students the LEA served.

20. To ensure that the State knows the amount LEAs spend to provide mental health services for student IEPs, before the start of the 2017-18 fiscal year, Education should develop, and require all LEAs to follow, an accounting methodology to track and report expenditures related to special education mental health services.

29. To ensure that the State provides special education and related services to all eligible students, Education should investigate the difference between the estimated number of school aged children statewide who have a severe emotional disturbance and the number receiving mental health services through an IEP and determine the reason for such a discrepancy. Education should then take any steps necessary to assist LEAs in identifying and providing services to children who are severely emotionally disturbed.

3. To ensure that it consistently processes complaints and appeals in a timely manner and that it investigates and reviews all UCP complaints and appeals in compliance with state law and regulations, by July 2017 Education should designate a central office to receive all complaints and appeals. This central office should distribute complaints and appeals to the correct divisions for investigation or review.

4. To ensure that it consistently processes complaints and appeals in a timely manner and that it investigates and reviews all UCP complaints and appeals in compliance with state law and regulations, by July 2017 Education should designate a central office to receive all complaints and appeals. This central office should establish a single database to record and track all investigations of complaints and reviews of appeals. This database should capture all data necessary for Education to effectively make informed decisions related to UCP complaints or appeals. At a minimum, the database should capture the date on which Education received each complaint or appeal, the date on which it forwarded the complaint or appeal to the appropriate division for investigation or review, and the date on which it sent the decision to the complainant. The database should also include the type of complaint or appeal, the LEA involved, and the decision.

5. To ensure that it consistently processes complaints and appeals in a timely manner and that it investigates and reviews all UCP complaints and appeals in compliance with state law and regulations, by July 2017 Education should designate a central office to receive all complaints and appeals. This central office should track the divisions' progress in processing complaints and appeals to ensure the divisions meet all UCP requirements, including documenting exceptional circumstances that constitute good cause for extending investigations beyond 60 days.

6. To ensure that it consistently processes complaints and appeals in a timely manner and that it investigates and reviews all UCP complaints and appeals in compliance with state law and regulations, by July 2017 Education should designate a central office to receive all complaints and appeals. This central office should work with divisions to establish policies and procedures for the divisions to follow when investigating UCP complaints and reviewing appeals. The procedures should identify the individuals or units responsible for investigating complaints and reviewing appeals, the steps and time frames for conducting investigations and reviews, the requirements for issuing decisions, and the documentation that should be retained in the files.

7. To ensure that it consistently processes complaints and appeals in a timely manner and that it investigates and reviews all UCP complaints and appeals in compliance with state law and regulations, by July 2017 Education should designate a central office to receive all complaints and appeals. This central office should establish and distribute a standard investigation report format that includes the required elements for the divisions to use when processing UCP complaints.

8. To ensure that it consistently processes complaints and appeals in a timely manner and that it investigates and reviews all UCP complaints and appeals in compliance with state law and regulations, by July 2017 Education should designate a central office to receive all complaints and appeals. This central office should monitor the divisions' decisions and reports on complaints and appeals to ensure that they comply with requirements.

16. To increase the efficiency and effectiveness of LEAs' UCP processes, Education should work with those LEAs throughout the State that receive a disproportionately high number of non-UCP complaints through the UCP process to assess the potential benefits of establishing similar mechanisms.

22. To ensure that its regulations are consistent and align with state and federal requirements, Education should revise its regulations to allow LEAs to extend investigations under exceptional circumstances that constitute good cause if the LEAs document and support with evidence the reasons for the extensions.

23. After it makes the recommended regulatory changes to allow extensions under exceptional circumstances, Education should review LEAs' extensions to investigations as part of its Federal Program Monitoring to ensure that LEAs' documentation is sufficient and that their reasons adequately justify such extensions.

28. Education should revise UCP regulations to formally establish uniform time limits for filing all types of complaints.

19. To ensure that it has complete disposition information, Justice should coordinate with the Judicial Council at least once a year to share information about court reporting gaps and to determine the need to distribute additional information to courts about reporting requirements and the manner in which to report. In addition, Justice should reconvene its advisory committee and meet on a regular basis to discuss, at a minimum, improving the frequency and timeliness with which courts report dispositions to Justice and law enforcement agencies report arrest information to Justice.

20. To ensure that it is receiving all arrest information from law enforcement agencies, at a minimum, Justice should consider trends in the number of arrest reports each law enforcement agency sends it and the number of reports that it might expect to receive from an agency given the agency's size, location, and reporting history. Whenever Justice identifies a law enforcement agency that it determines may not be reporting all required information, it should request that the agency forward all required arrest information.

1. To ensure that it receives complete and accurate data, DOJ should, by May 2019, develop and maintain a list of law enforcement agencies that it updates annually, obtain hate crime data from all law enforcement agencies, distribute additional guidance to those agencies on procedures for reporting hate crimes, and conduct periodic reviews of law enforcement agencies to ensure that the data they report are accurate. It should also seek the resources to implement these efforts, if necessary.

3. To increase the effectiveness of hate crime prevention and response efforts, DOJ should provide additional guidance to law enforcement agencies by adding region-specific data fields to the hate crime database, including items such as the zip code in which reported hate crimes took place and other fields that DOJ determines will support its outreach efforts.

4. To increase the effectiveness of hate crime prevention and response efforts, DOJ should provide additional guidance to law enforcement agencies by analyzing reported hate crimes in various regions in the State and sending advisory notices when it detects hate crimes happening across multiple jurisdictions. It should also seek the resources to implement these efforts, if necessary.

5. To increase the effectiveness of hate crime prevention and response efforts, DOJ should provide additional guidance to law enforcement agencies by creating and disseminating outreach materials so law enforcement agencies can better engage with their communities.

6. To increase the effectiveness of hate crime prevention and response efforts, DOJ should provide additional guidance to law enforcement agencies by creating and making available training materials for law enforcement agencies on how best to identify and respond to hate crimes.

7. To ensure that law enforcement agencies effectively engage with communities regarding hate crimes, DOJ should provide guidance and best practices for law enforcement agencies to follow when conducting hate crime outreach to vulnerable communities within their jurisdictions, such as collaborating with a county human rights commission. It should make the outreach materials available to law enforcement agencies and should include in them presentation materials for various types of communities, including immigrants and Muslims, among others. It should seek the resources to implement these efforts, if necessary.

9. To ensure that districts, county offices, and schools receive guidance on a variety of safety issues and to comply with state law, CDE and DOJ should resume their partnership activities, as required by state law. Further, the partnership should update the 2002 handbook, "Safe Schools: A Planning Guide for Action," and distribute it to all districts and county offices. If CDE or DOJ determine the need for additional funds to implement the legislative recommendations or to reestablish the partnership's activities, they should request those funds from the Legislature.

4. To report to the Legislature about the effectiveness of its RADS program and to better inform decisions about expanding the number of analyzed sexual assault evidence kits, Justice should amend its agreements with the counties participating in the RADS program to require those counties to report case outcome information, such as arrests and convictions for the sexual assault evidence kits Justice has analyzed under the program. Justice should then report annually to the Legislature about those case outcomes.

10. As the Legislature considers creating a public program for shared gang database oversight and accountability, Justice should guide the board and the committee to identify and address the shortcomings that exist in CalGang's current operations and oversight. The guidance Justice provides to the board and the committee should address, but not be limited to, developing best practices based on the requirements stated in the federal regulations, the state guidelines and state law, and advising user agencies on the implementation of those practices. The best practices should include, but not be limited to reviewing criminal intelligence, appropriately disseminating information, performing robust audit practices, establishing plans to recover from disasters, and meeting all of the State's juvenile notification law requirements. Justice should guide the board and the committee to develop these best practices by June 30, 2017.

11. As the Legislature considers creating a public program for shared gang database oversight and accountability, Justice should guide the board and the committee to identify and address the shortcomings that exist in CalGang's current operations and oversight. The guidance Justice provides to the board and the committee should address, but not be limited to, instructing user agencies that use CalGang to complete a comprehensive review of all the gangs documented in CalGang to determine if they meet the necessary requirements for inclusion and to purge from CalGang any groups that do not meet the requirements. Justice should guide the board and the committee to ensure that user agencies complete this review in phases, with the final phase to be completed by June 30, 2018.

12. As the Legislature considers creating a public program for shared gang database oversight and accountability, Justice should guide the board and the committee to identify and address the shortcomings that exist in CalGang's current operations and oversight. The guidance Justice provides to the board and the committee should address, but not be limited to, instructing all user agencies to complete a comprehensive review of the records in CalGang to determine if the user agencies have adequate support for the criteria associated with all the individuals they have entered as gang members. If the user agencies do not have adequate support, they should immediately purge the criteria—and, if necessary, the individuals—from CalGang. In addition, the user agencies should ensure that all the fields in each CalGang record are accurate. Justice should guide the board and the committee to ensure that user agencies complete this review in phases, with the final phase to be completed by September 30, 2019.

13. As the Legislature considers creating a public program for shared gang database oversight and accountability, Justice should guide the board and the committee to identify and address the shortcomings that exist in CalGang's current operations and oversight. The guidance Justice provides to the board and the committee should address, but not be limited to, instructing all user agencies to report to Justice every six months, beginning in January 2017, on their progress toward completing their gang and gang member reviews.

19. To promote transparency and encourage public participation in CalGang's meetings, Justice should post summary results from the committee's audits of CalGang records to its website unless doing so would compromise criminal intelligence information or other information that must be shielded from public release.

2. Unless the Legislature directs the board to eliminate the compliance fund's excess fund balance within a time frame of more than a year, the board should eliminate the excess fund balance by June 30, 2017 by using it to offset the licensing program's annual funding shortfall. The board should also limit the fund's future balance to no more than two months' worth of licensing program expenditures.

6. To help address California's e-commerce tax gap and further ensure out-of-state retailers' compliance with state law regarding nexus, Tax Administration should implement a two-year pilot of its authorized reward program for information resulting in the identification of unreported sales and use taxes.

1. The authority should amend its regulations to bring them into accord with the 2004 act, thus allowing any eligible hospital to apply for the 2004 act's funds that remained as of June 30, 2014.

1. To ensure that the compensation the AOC provides is reasonable, the Judicial Council should adopt procedures that require a regular and thorough review of the AOC's compensation practices including an analysis of the job duties of each position to ensure that the compensation aligns with the requirements of the position. This review should include comparable executive branch salaries, along with a justification when an AOC position is compensated at a higher level than a comparable executive branch position.

3. To ensure that its compensation structure is reasonable, the AOC should mirror the executive branch's practices for offering leave buyback programs in terms of frequency and amount.

4. To increase its efficiency and decrease its travel expenses, the AOC should require its directors and managers to work in the same locations as the majority of their staff unless business needs clearly require the staff to work in different locations than their managers.

11. To reduce its expenses, the AOC should cease its excessive reimbursements for meals by adopting the executive branch's meal and travel reimbursement policies.

14. To ensure that it spends funds appropriately, the AOC should develop and implement controls to govern how its staff can spend judicial branch funds. These controls should include specific definitions of local assistance and support expenditures, written fiscal policies and procedures as the rules of court require, and a review process.

19. The AOC should conduct a comprehensive survey of the courts on a regular schedule—at least every five years—to ensure that the services it provides align with their responses. The AOC should re-evaluate any services that the courts identify as being of limited value or need.

20. To justify its budget and staffing levels, the AOC should conduct the steps in CalHR's workforce planning model in the appropriate order. It should begin by establishing its mission and creating a strategic plan based on the needs of the courts. It should then determine the services it should provide to achieve the goals of that plan. The AOC should base its future staffing changes on the foundation CalHR's workforce planning model provides. Finally, the AOC should develop and use performance measures to evaluate the effectiveness of this effort.

21. To ensure that it provides services to the trial courts as efficiently as possible, the Judicial Council should explore implementing a fee-for-service model for selected services. These services could include those that are little used or of lesser value to the trial courts, as identified in our survey that we discuss in Chapter 3.

22. To justify the budget and staff level of the AOC, the Judicial Council should implement some or all of the best practices we identified to improve the transparency of AOC spending activities.

3. To improve the usefulness of the Judicial Council's semiannual reports until a statutory requirement is enacted, the AOC should work with the Judicial Council to pursue a cost-effective method to do the following:

- Include new contracts and the complete history of contracts amended during the reporting period in the semiannual reports, including the date of the original contract; the original contract amount and duration; all subsequent contract amendments; and the date, amount, and duration of each such amendment. The AOC should present this information beginning with the semiannual report covering the July 1, 2014, through December 31, 2014, reporting period.

4. To improve the usefulness of the Judicial Council's semiannual reports until a statutory requirement is enacted, the AOC should work with the Judicial Council to pursue a cost-effective method to do the following:

- Begin tracking additional information in its data systems for inclusion in the semiannual reports. This information should include whether a contract was competitively bid, the justification if it was not competitively bid, and whether the contract was with a Disabled Veteran Business Enterprise. For information technology contracts, the AOC should identify whether the contract was with a small business. The AOC should present this information beginning with the semiannual report covering the July 1, 2014, through December 31, 2014, reporting period.

5. The AOC should implement all of the best practices related to general and business process application controls as outlined in the U.S. Government Accountability Office's Federal Information System Controls Audit Manual no later than December 31, 2014, thereby strengthening and continuously monitoring the effectiveness of the controls over its information systems. In addition, the AOC should immediately begin implementing improvements to its controls over access to its information systems and place these improvements into effect by February 2014. Finally, the AOC should provide guidance and routinely follow up with the superior courts—requiring updates every six months until all identified issues are corrected—to ensure that they make the necessary improvements to their general and business process application controls.

6. The AOC, the Supreme Court, and the first, second, and fourth districts should implement procedures to ensure that they follow a competitive process for their procurements when required.

13. The AOC should revise the judicial contracting manual to require judicial entities to maintain documentation on their determinations of fair and reasonable pricing for purchases under $5,000.

17. The AOC should revise the judicial contracting manual to require that judicial entities maintain documentation for their evaluation and selection process used for competitive procurements. The AOC should also strengthen its procedures to ensure that bid evaluations are conducted properly and calculated correctly.

20. The AOC, HCRC, Supreme Court, and fourth and fifth districts should implement procedures to ensure that required noncompetitive procurement processes, such as preparing justifications and obtaining approval for sole-source procurements, are properly documented. Additionally, the AOC should ensure that it prepares the appropriate documentation when it amends a contract that it has competitively solicited and the amendment includes a change that was not evaluated in the original competitive process.

25. The AOC should implement procedures to ensure that its internal controls over payments are followed and that procurements are approved before ordering and receiving goods and services.

11. The Secretary of State should adopt regulations establishing clear criteria for mistakes in election-related materials that constitute reportable errors and require counties to report these errors to it after each election.

12. Beginning in December 2018, the Secretary of State should implement annual risk-based reviews of a selection of county election officials' offices to ensure their compliance with state election laws and regulations.

13. To inform and enhance the guidance it provides to county election officials, the Secretary of State should analyze error reports and its risk-based review results to focus its guidance on topics most relevant to improving elections throughout the State.

6. To assign purchasing cards only to appropriate staff, ensure that the State Bar's records of employees' credit limits reflect those established with the bank, and to verify that staff use purchasing cards only for allowable and necessary expenses, the State Bar should immediately develop a policy that requires justification of the business needs for employees to receive purchasing cards, and use this policy to limit the number of staff issued a purchasing card.

7. To assign purchasing cards only to appropriate staff, ensure that the State Bar's records of employees' credit limits reflect those established with the bank, and to verify that staff use purchasing cards only for allowable and necessary expenses, the State Bar should immediately restrict the use of purchasing cards to its original purpose, which was for low-dollar and frequently occurring purchases. For purchases above $5,000, the State Bar should require the vendor to bill for payment.

9. To ensure that its costs are reasonable and appropriate, the State Bar should update its meal and catering policy to align with the meal policy of the State's Executive Branch and should require individuals attending committee meetings for the State Bar to comply with standard meal per diem rates.

7. To ensure that DWR manages WaterFix in an effective manner, DWR should complete both the economic analysis and financial analysis for WaterFix and make the analyses publicly available as soon as possible.

9. In order to prepare for the potential approval of WaterFix and to ensure that the project is managed properly during the design and construction phase, DWR should develop and update when necessary the associated program management plan for the design and construction phase of the project.

7. To reduce the risk of fraudulent applications, by September 2017 DMV should seek interagency agreements with the health boards responsible for licensing providers authorized to certify disabilities on placard applications. The agreements should include, but not be limited to, a review by medical experts of a sample of placard applications each quarter to ensure that the disability certifications meet state requirements. For any application that does not meet state requirements, DMV should require that the applicant and his or her provider submit the information needed so that the application meets state requirements. DMV should cancel the placards of those who do not respond within 90 days.