Report 2010-102 Recommendations and Responses in 2012-041

Report 2010-102: Administrative Office of the Courts: The Statewide Case Management Project Faces Significant Challenges Due to Poor Project Management

Department Number of Years Reported As Not Fully Implemented Total Recommendations to Department Not Implemented After One Year Not Implemented as of Most Recent Response
Administrative Office of the Courts 1 37 10* 4

Recommendation To: Administrative Office of the Courts

To better manage costs of future IT projects, the AOC should estimate costs at the inception of projects.

Response

The AOC has two artifacts for estimating costs at the inception of a project, the Business Case and the Project Assessment Form, both which have been developed as part of the Enterprise Methodology and Process (EMP) program. A key component of the EMP program is development and implementation of a standard Solution Development Life Cycle (SDLC). SDLC describes a phase-by-phase methodology for projects categorized as medium or large, including standards, processes, and artifacts. While no new projects in either category, medium, or large, are currently envisioned in the current budget climate, upcoming applicable maintenance and operations (M&O) efforts and future projects will be required to adhere to the SDLC.

The primary purpose of development of a business case, in support of Judicial Council-approved recommendations for AOC realignment, is to provide a financial justification for undertaking a project, entailing cost analyses for a recommended solution and for alternatives that were explored as well. Costs are broken out into several categories (hardware, software, etc.), as well as project vs. ongoing costs. A copy of the business case template is attached.

The purpose of the Project Assessment Form is to gather high-level information about a request to categorize the project as small, medium, or large, based on estimated level of effort, duration, and cost. Costs are broken out into several categories (hardware, software, etc.), application vs. infrastructure, and project vs. ongoing. A copy of the Project Assessment Form is included below.

Recommendation To: Administrative Office of the Courts

To better manage costs of future IT projects, the AOC should employ appropriate budget and cost management tools to allow it to appropriately budget, track, manage, and estimate costs.

Response

The AOC has two artifacts for budgeting, tracking, managing, and estimating costs, the business case and the project assessment forms, both developed as part of the Enterprise Methodology and Process (EMP) program. A key component of the EMP program is development and implementation of a standard Solution Development Life Cycle (SDLC) that describes a phase-by-phase methodology for projects categorized as medium or large, including standards, processes, and artifacts. While no new projects in either category are currently envisioned in the current budget environment, upcoming applicable maintenance and operations (M&O) efforts and future projects will be required to adhere to the SDLC. The Business Case and Project Assessment Form templates are attached.

In addition to completing the business case artifact itself, development of which supports Judicial Council-approved recommendations for AOC realignment, the SDLC calls for revisiting the business case after project inception to facilitate cost management and tracking once an effort is underway.

Another key tool used for budgeting, tracking, managing, and estimating costs is the Information and Technology Services Office's (formerly Information Services Division) zero based budgeting process. Specifically, Information and Technology Services prepares a five-year budget for each Information and Technology Services program. Each program budget is shared with the Fiscal Services Office (formerly Finance Division) and other business partners before being used as a basis for the Information and Technology Services' annual budget allocation. The core purpose of this process is to identify key needs for each project going forward while providing sufficient funding for baseline activities. The other purpose is to provide a budget monitoring tool for project managers throughout the fiscal year. The budget projections produced through this process are also used to inform the Judicial Council, related working groups, and advisory committees. To develop, track and monitor their budgets, each program uses information provided by the Oracle Financial system, which captures all AOC allocations, encumbrances, and expenditures for Information and Technology Services programs.

Finally, the AOC makes use of a project portfolio management (PPM) tool for weekly project reporting, one element of which includes budget health. Formal criteria for the budget health indicator have been established, and are as follows: green, project is tracking to approved budget, which may include one that has been re-baselined per a formally approved change; yellow, a budgetary risk has been identified for which a mitigation strategy is in place; red, a budgetary risk has been identified for which there is not a mitigation strategy in place. An explanation of any yellow or red budget indicators must be provided in writing, and a justification for any indicators that were previously yellow or red but have been reset to green must also be noted.

Recommendation To: Administrative Office of the Courts

To better manage costs of future IT projects, the AOC should ensure that cost estimates are accurate and include all relevant costs, including costs that superior courts will incur.

Response

The AOC has two artifacts to ensure that all relevant costs of an effort are considered, the Business Case and the Project Assessment Form, both of which have been developed as part of the Enterprise Methodology and Process (EMP) program. A key component of the EMP program is development and implementation of a standard solution development life cycle (SDLC) that describes a phase-by-phase methodology for projects categorized as medium or large, including standards, processes, and artifacts. While no new projects in either category are currently envisioned in the current budget environment, upcoming applicable maintenance and operations (M&O) efforts and future projects will be required to adhere to the SDLC.

The Business Case and Project Assessment Form templates are embedded below.

Both the business case, development of which supports Judicial Council-approved recommendations for AOC realignment, and the project assessment form, include a comprehensive list of cost categories, including hardware, software, data center, contract services, one-time vs. ongoing costs, and others. The project assessment form includes costs likely to be incurred by entities external to the AOC, such as the superior courts, justice partners, and law enforcement agencies. For example, these may include expenses related to time required of internal staff, potential software and hardware costs, or contracting vendors for design, development or testing. Further, the SDLC calls for revisiting the business case after project inception to validate the accuracy of initial estimates. The AOC makes use of a project portfolio management (PPM) tool for weekly project reporting, one element of which includes budget health, ensuring that significant variances from initial cost estimates are identified and addressed. The SDLC also requires a post-project review, one part of which includes an analysis of the accuracy of cost estimates and the identification of any lessons learned to improve cost management on future efforts. The post-project review template is embedded below.

Recommendation To: Administrative Office of the Courts

To better manage costs of future IT projects, the AOC should disclose costs that other entities will likely incur to the extent it can reasonably do so.

Response

The AOC has two artifacts for ensuring that all relevant costs of an effort are considered, the business case and the project assessment form, both of which have been developed as part of the Enterprise Methodology and Process (EMP) program. A key component of the EMP program is development and implementation of a standard solution development life cycle (SDLC) that describes a phase-by-phase methodology for projects categorized as medium or large, including standards, processes, and artifacts. While no new projects in either category are currently envisioned, given the current budget environment, upcoming applicable maintenance and operations (M&O) efforts and future projects will be required to adhere to the SDLC.

The Business Case and Project Assessment Form templates are embedded below.

Both the business case, development of which supports the Judicial Council-approved recommendations for AOC realignment, and the project assessment form, include a comprehensive list of cost categories, including hardware, software, data center, and contract services, one-time vs. ongoing costs, and costs likely to be incurred by entities external to the AOC, such as the superior courts, justice partners, and law enforcement agencies. For example, this might include expenses related to time required of internal staff, potential software and hardware costs if available, and contracting vendors for design, development or testing.

Recommendation To: Administrative Office of the Courts

To better manage costs of future IT projects, the AOC should update cost estimates on a regular basis and when significant assumptions change.

Response

The AOC has two artifacts for ensuring that cost estimates are updated when significant assumptions change, the business case and the project assessment form, both of which have been developed as part of the Enterprise Methodology and Process (EMP) program. A key component of the EMP program is development and implementation of a standard solution development life cycle (SDLC) that describes a phase-by-phase methodology for projects categorized as medium or large, including standards, processes, and artifacts. While no new projects in either category are currently envisioned in the current budget environment, upcoming applicable maintenance and operations (M&O) efforts and future projects will be required to adhere to the SDLC.

The Business Case and Project Assessment Form templates are embedded below.

The SDLC calls for revisiting the business case, development of which supports Judicial Council-approved recommendations for AOC realignment following project inception, to validate and update accuracy of initial estimates. Likewise, the AOC makes use of a project portfolio management (PPM) tool for weekly project reporting, one element of which includes budget health, ensuring that significant variances from initial cost estimates are identified and addressed. Formal criteria for the budget health indicator have been established, and are as follows: green, the project is tracking to the approved budget,; yellow, a budgetary risk has been identified for which a mitigation strategy is in place; red, a budgetary risk has been identified for which there is no mitigation strategy in place. An explanation of any yellow or red budget indicators must be provided in writing, and a justification for any indicators that were previously yellow or red but have been reset to green must also be noted.

The annual budget process also provides an opportunity to adjust budget projections to reflect changes in program strategy and assumptions. Specifically, Information and Technology Services prepares a five-year budget for each Information and Technology Services program. Each program budget is shared with the Fiscal Services Office (formerly Finance Division) and other business partners before being used as a basis for the Information and Technology Services' annual budget allocation. The core purpose of this process is to identify key needs for each project going forward while providing sufficient funding for baseline activities. The other purpose is to provide a budget monitoring tool for project managers throughout the fiscal year. The budget projections produced through this process are also used to inform the Judicial Council, related working groups and advisory committees. To develop, track and monitor their budgets, each program uses information provided by the Oracle Financial system which captures all AOC allocations, encumbrances and expenditures for Information and Technology Services programs. As the program strategy and assumptions change, Oracle is updated accordingly, to reflect the status of the project.

Recommendation To: Administrative Office of the Courts

To better manage costs of future IT projects, the AOC should disclose full and accurate cost estimates to the Judicial Council, the Legislature, and stakeholders from the beginning of projects.

Response

The Judicial Branch Technology Committee, in collaboration with working groups and advisory committees, is establishing a branch governance structure that will enable communication to the Judicial Council, the legislature and stakeholders, With the March 27, 2012 Judicial Council decision to end CCMS, the Judicial Council tasked the Technology Committee with overseeing the council's policies concerning technology. The Technology Committee is responsible, in partnership with the courts, to coordinate with the Administrative Director of the Courts and all internal committees, advisory committees, commissions, working groups, task forces, justice partners and stakeholders on technological issues relating to the branch and the courts. The committee is responsible to ensure compliance with council policies and that specific projects proceed on schedule, and within scope and budget.

In May 2012, the committee established the Judicial Branch Technology Initiatives Working Group, consisting of Presiding Judges, Court Executive Officers, and Court Information Officers, to assist in the overall effort of defining and confirming requirements for branchwide technology initiatives to address the short-term critical technology needs for the Judicial Branch, and a long-term strategic vision for the Judicial Branch. As a result of this effort, the Technology Committee will develop a governance structure for technology programs that will provide the oversight, monitoring, transparency and accountability recommended by the BSA and further supported by the Judicial Council recommendations published August 2012. Project implementations will be subject to the approval of the Technology Committee.

In addition to the project oversight provided by the governance structure , the superior courts, the appellate courts, and the Administrative Office of the Courts are all subject to the Judicial Branch Contract Law which requires the approval from the California Technology Agency for projects of the Judicial Council or the courts with total costs estimated at more than five million dollars.

We believe the above measures will provide the necessary oversight to ensure communication of costs at the onset of projects.

Recommendation To: Administrative Office of the Courts

To better manage costs of future IT projects, the AOC should ensure that it has a long-term funding strategy in place before investing significant resources in a project.

Response

The Judicial Branch Technology Committee, in collaboration with working groups and advisory committees, is establishing a branch governance structure that will enable communication to the Judicial Council, the Legislature and all stakeholders, With the March 27, 2012 Judicial Council decision to end CCMS, the Judicial Council tasked the Technology Committee with overseeing the council's policies concerning technology. The Technology Committee is responsible in partnership with the courts for coordinating with the Administrative Director of the Courts and all internal committees, advisory committees, commissions, working groups, task forces, justice partners and stakeholders on technological issues relating to the branch and the courts. The committee is responsible for ensuring that council policies are complied with and that specific projects proceed on schedule and within scope and budget.

In May 2012, the committee established the Judicial Branch Technology Initiatives Working Group consisting of Presiding Judges, Court Executive Officers, and Court Information Officers, to assist in the overall effort of defining and confirming requirements for branchwide technology initiatives, to address the short-term critical technology needs for the Judicial Branch and a long-term strategic vision for the Judicial Branch. As a result of this effort, the Technology Committee will develop a governance structure for technology programs to provide the oversight, monitoring, transparency and accountability recommended by the BSA and further supported by the Judicial Council recommendations published August 2012. Project implementations will be subject to the approval of the Technology Committee.

In addition to the project oversight provided by the governance structure , the superior courts, appellate courts, and the Administrative Office of the Courts are subject to the Judicial Branch Contract Law, which requires the approval from the California Technology Agency for projects of the Judicial Council or the courts with total costs estimated at more than five million dollars.

We believe these measures will provide the necessary oversight to ensure a long term funding strategy.

Recommendation To: Administrative Office of the Courts

Although the Judicial Council has the legal authority to compel the courts to adopt CCMS, to better foster superior court receptiveness to deploying CCMS, the AOC should continue to work with the superior courts that have deployed the civil system to ensure it is addressing their concerns in a timely and appropriate manner.

Response

Currently the Information Technology and Information Services Office (ITSO) [formerly the Information Services Division (ISD) of the Administrative Office of the Courts (AOC) conducts weekly conference calls with five superior courts that have deployed CCMS V3. The V3 support project manager facilitates weekly meetings which are attended by court project managers, technical analysts, and operational staff. During these meetings court representatives discuss operational issues and prioritize items for the next software release. Following established processes, any enhancements and defects exceeding a pre-defined level of effort (LOE) are escalated to the governance committee for approval. In addition to the weekly group meetings, weekly meetings are held with each individual court, providing an opportunity to discuss issues specific to their court.

To further support the courts, metrics are maintained to track compliance to service level agreements, as well as application performance and reliability. Over the past 12 months there has been only one severity 1 (critical) issue recorded. The CCMS V3 application has been extremely stable.

Regarding the future, the Judicial Council Technology Committee created the Judicial Branch Technology Initiatives Working Group and developed multiple workstreams to address various technology issues facing the branch. Their goal is to make recommendations to the Judicial Council to specifically address the technology issues. One such workstream is the V/2V3 Maintenance workstream tasked with determining how the courts that are currently using V2 or V3 will be supported. The V2/V3 workstream is chaired by the Court Executive Officer for Orange County and the Court Information Technology Officer from Fresno. It is a collaborative effort. Representatives from V3 and V2 courts including Orange, Sacramento, San Diego, Ventura, San Joaquin, and Fresno, with participation by AOC Information and Technology Services staff. The courts are preparing a proposal regarding the future of the V2 and V3 case management systems. The document addresses maintenance and support, governance, funding, addition of courts, hosting, and life expectancy of the case management systems. Current Service Level Agreements (SLAs) are being met.

In consideration of current support activities and those planned for the future, we believe the concerns of the courts using the civil case management system, CCMS V3, are being addressed in a timely and appropriate manner.

Recommendation To: Administrative Office of the Courts

Although the Judicial Council has the legal authority to compel the courts to adopt CCMS, to better foster superior court receptiveness to deploying CCMS, the AOC should work with superior courts to address concerns about hosting data at the California Court Technology Center (Technology Center). Further, the AOC should take steps to ensure that superior courts do not lose productivity or efficiencies by hosting data at the Technology Center.

Response

The CCTC provides consistent, cost-effective, and secure hosting services, including ongoing maintenance and operational support, data network management, desktop computing and local server support, tape back-up and recovery, help desk services, email services, a dedicated service delivery manager, and a disaster recovery program. In the event of a significant interruption of court services, the Disaster Recovery program ensures that infrastructure and network services, and trial and appellate court applications hosted in the CCTC can be safely and securely backed-up, redirected, and restored.

CCTC services allow the courts to rely on the skills and expertise of their maintenance and support teams to configure and install software and hardware upgrades and address other minor and critical issues. Courts also benefit from enterprise hardware and software license agreements, as well as bulk volume discounts in purchasing. In addition, centralized changes (e.g. hardware and software patches) are much easier and more efficient to install, while the central help desk support provides the courts a single point of contact.

To address the needs of the courts, each program has a delivery manager who works directly with the courts to address day-to-day issues and concerns, as well as extended challenges. Weekly, the delivery manager meets with CCTC staff on behalf of the courts to address any service issues.

Regarding the future, the Judicial Council Technology Committee created the Judicial Branch Technology Initiatives Working Group and developed multiple workstreams to address various technology issues facing the branch. Their goal is to make recommendations to the Judicial Council to specifically address the technology issues. One such workstream is the V/2V3 Maintenance workstream tasked with determining how the courts currently using V2 or V3 will be supported going forward. Another workstream is the Technology Roadmap, tasked with defining the technology vision for the branch. Both workstreams are collaborative efforts led by trial court executives and sponsored by judicial officers. The issue of hosting will be addressed by both workstreams as they address the question of local versus central hosting. At the conclusion of the project, recommendations will be provided to the Judicial Council.

In consideration of current support activities and those planned for the future, we believe the concerns of the courts hosted at the California Courts Technology Center are being addressed to guard against potential loss of productivity and or efficiencies. System reliability and availability at the CCTC has been stable and help desk tickets are being handled expeditiously. Performance is governed contractually through strict Service Level Agreements (SLAs) and monitoring, including financial consequences to the Technology Center vendor, SAIC, should SLAs ever be missed. The AOC has never had to exercise this contract clause due to missed SLAs.

The Technology Roadmap Workstream is working on the future of judicial branch technology.

Recommendation To: Administrative Office of the Courts

To ensure that future major IT projects receive appropriate independent oversight over technical aspects and project management, the AOC should obtain IV&V and IPO services at the beginning of the projects and ensure this independent oversight is in place throughout and follows best practices and industry standards appropriate for the size and complexity of the project.

Response

The AOC continues to use the guidelines and framework established for graduated project oversight of information technology projects, including independent oversight, from the Information Technology Project Oversight Framework handbook when appropriate. The oversight level will be according to the evaluation of criticality and the risk level of the project. This will be initiated during the concept stage of the project. The AOC will also continue to operate under industry guidelines and standards of the Institute of Electrical and Electronics Engineers (IEEE) for verification and validation (V&V) activities as established in Standard 1012-2012. Independent V&V activities will be performed with the established standard's guidelines. For example, if a vendor is performing project oversight work, a separate vendor will perform the independent V&V activities. This will be done within the review and according to the recommendations of the California Technology Agency, as required by Government Code section 68511.9.

Current Status of Recommendations

All Recommendations in 2012-041