Report 2014-116 Summary - February 2015

California Department of Consumer Affairs' BreEZe System

:

Inadequate Planning and Oversight Led to Implementation at Far Fewer Regulatory Entities at a Significantly Higher Cost

HIGHLIGHTS

Our audit concerning the California Department of Consumer Affairs' (Consumer Affairs) planning, development, and implementation of BreEZe—an information technology (IT) system envisioned to support all primary functions and responsibilities of its regulatory entities—revealed the following:

RESULTS IN BRIEF

The California Department of Consumer Affairs (Consumer Affairs) encompasses 40 boards, bureaus, committees, and a commission (regulatory entities) that regulate and license professional and vocational occupations to protect the health, safety, and welfare of the people of California. Annually, these regulatory entities process more than 350,000 applications for professional licensure and an estimated 1.2 million license renewals. The regulatory entities establish the minimum qualifications and levels of competency for licensure, register or certify practitioners, investigate complaints, and discipline violators. Although these entities are responsible individually for activities related specifically to the professions they oversee and they are semiautonomous bodies whose members are appointed by the governor and the Legislature, Consumer Affairs establishes general administrative policies for them and provides them with administrative support.

Historically, the regulatory entities have used multiple computer systems to fulfill their required duties and meet their business needs. However, significant issues with these systems reportedly resulted in excessive turnaround times for licensing and enforcement activities, impeding the ability of the regulatory entities to meet their goals and objectives. In 2009, after undertaking several unsuccessful efforts to develop or procure an information technology (IT) system that would improve the capabilities of the regulatory entities it administratively supports, Consumer Affairs proposed, and the California Department of Technology (CalTech) approved, BreEZe—a system Consumer Affairs envisioned would support all of the primary functions and responsibilities of its regulatory entities.1 Unfortunately, this has not been the case.

The work Consumer Affairs undertook on the BreEZe project has lacked adequate planning. Although an up-to-date assessment of business needs is critical to the successful development of an IT project, Consumer Affairs failed to properly perform such an assessment for its regulatory entities when developing the system requirements, resulting in requirements that did not adequately reflect their individual needs. According to our IT expert, system requirements define a business problem to be solved and specify what the system should do. For example, a system requirement for a regulatory entity could be that the system allow the entity to record the date it receives an application. In planning the BreEZe system, Consumer Affairs should have taken steps to ensure that the system requirements were based on the current business needs of its regulatory entities, so that the resulting system would aid the entities in conducting their business operations and in fulfilling their regulatory responsibilities. Instead, when developing the requirements for BreEZe, Consumer Affairs relied on requirements for earlier projects that were abandoned.

Because Consumer Affairs did not properly determine the business needs of its regulatory entities, it incorrectly assumed, for example, that the entities could use similar business processes to process applications and issue licenses. This misconception, coupled with the fact that Consumer Affairs wanted BreEZe to be developed quickly, informed Consumer Affairs' decision to select an existing commercial "off-the-shelf," or COTS, system as the foundation for BreEZe. Consumer Affairs believed that this type of product, rather than a custom-developed system, would require only moderate modifications and resources to implement. These faulty assumptions have led to significant project delays and a substantial increase in the estimated costs of the project, from $28 million in 2009 to $96 million as of January 2015, for implementation of a system that will include only half of the regulatory entities originally planned for BreEZe. Thus, it appears that Consumer Affairs' selection of this COTS product may not have been the most appropriate and most cost-effective decision.

In part, because the foundation of BreEZe—its system requirements—was inadequately developed, the BreEZe project has experienced delays at key stages of the project. The most extreme delay involved the key milestone of user acceptance testing—testing that future users of the system conduct to confirm that the system operates as its requirements specify. User acceptance testing for the 10 regulatory entities included in the first implementation of BreEZe (phase 1) was originally planned to occur over an eight-week period; instead it spanned 11 months, from the end of November 2012 to October 2013, significantly exceeding the original time frame. This likely occurred in part because the BreEZe system had almost 1,700 unresolved system defects at the beginning of user acceptance testing. According to our IT expert, many of these defects were likely attributable to the poor development of the system requirements. Although user acceptance testing is one of the final and more critical procedures undertaken before system implementation to ensure that the system operates appropriately, in this case it morphed into a redesign of the requirements and a rework of the system. Specifically, in conducting the testing of the system, some of the 10 regulatory entities included in the first phase of implementation, as well as Consumer Affairs itself, learned that the system did not operate as they expected or needed. Had Consumer Affairs performed a complete, current assessment of the regulatory entities' needs when determining the system requirements for BreEZe, some of the delays the project has experienced might have been avoided.

Further, although CalTech began providing independent oversight of the BreEZe project approximately one year after the project's inception, neither CalTech nor Consumer Affairs responded appropriately to the significant and persistent concerns that the CalTech staff and consultants charged with overseeing the project were raising. In addition to having the statutory authority to suspend or terminate IT projects, state law assigns responsibility for IT project oversight to CalTech; this project oversight mainly consists of two types of independent oversight. Independent verification and validation (IV&V) is used to ensure that a system satisfies its intended use and user needs. Independent project oversight (IPO) is used to ensure that effective project management practices are in place and in use. In their reports from December 2010 through September 2014 on the BreEZe project, the CalTech IV&V consultant and the IPO specialist raised nearly 180 significant concerns relating to project management, staffing, system requirements, and vendor performance. According to our IT expert, the volume and significance of these concerns should have prompted both CalTech and Consumer Affairs to analyze fully the costs and benefits of suspending or terminating the project versus proceeding. However, although Consumer Affairs officials and CalTech management were fully aware of these concerns, neither group took sufficient action to ensure that these concerns were appropriately addressed; instead, they allowed the project to continue for more than three years without significant intervention.

Given CalTech's authority and the numerous concerns the IV&V consultant and the IPO specialist raised about the project, we question why CalTech did not take steps to ensure that Consumer Affairs heeded its advice. For instance, CalTech could have formally warned Consumer Affairs that it would suspend the project if Consumer Affairs did not bring the project back into alignment with its planned scope, cost, and schedule. As an example, the estimated cost to complete the project had almost tripled to $78 million and the project had experienced significant delays in its schedule before completion of user acceptance testing. We believe these problems, along with the significant cost increases the project had already experienced, should have been enough to elicit CalTech's greater involvement in the project. Instead, it approved Consumer Affairs' Special Project Report (SPR) 2, which requested additional funding for the project, in October 2013.2

Consumer Affairs submitted SPR 3 to CalTech in June 2014; in it, it requested additional funding and estimated the costs to complete the project through phase 2 at $118 million. However, it was not until after Consumer Affairs informally estimated the cost of completing the project had risen to $300 million that same month that CalTech changed its oversight approach on the BreEZe project.3 Although CalTech approved SPR 3 in July 2014, according to the BreEZe project director, Consumer Affairs withdrew its submission of SPR 3 upon direction from CalTech and the California Department of Finance in September 2014.

As discussed previously, CalTech has the authority and responsibility to oversee IT projects. If CalTech had chosen to suspend the project, BreEZe development would have been paused temporarily, giving Consumer Affairs additional time to conduct a cost-benefit analysis and correct fundamental problems, such as requirements issues, that occurred during planning and development. However, in October 2014 the CalTech director—who has overseen the BreEZe project since Consumer Affairs executed its contracts with the project vendor, Accenture LLP (Accenture)—told us that CalTech has not halted BreEZe for several reasons: because BreEZe is moving in the right direction, because the system's problems are not incurable, and because the system is working and functional.4 However, Consumer Affairs' SPR 3.1, which it submitted to CalTech in January 2015, indicates the project is not moving in the right direction and proposes a rescoping of the project because of significant concerns relating to staffing and increasing project costs, and because its contracts with Accenture are no longer financially feasible for Consumer Affairs.

For these reasons, among others, the future implementation of BreEZe is uncertain at best and, as it relates to the regulatory entities originally included in the final phase (phase 3), likely unfeasible. As of January 2015, 10 regulatory entities had implemented the system, with the first of three phases occurring in October 2013. Another eight regulatory entities are included in phase 2, which is currently planned for March 2016. However, Consumer Affairs has indicated that it needs additional staffing to successfully implement BreEZe at the phase 2 regulatory entities, and as of January 2015 lacked the funding to fill those positions. Additionally, it is unknown whether or when the remaining 19 phase 3 regulatory entities will implement BreEZe. Specifically, CalTech officials indicated that it completed renegotiating Consumer Affairs' design contract with Accenture on December 1, 2014, and according to Consumer Affairs' director, these 19 regulatory entities had been removed entirely from the project. Although the director of Consumer Affairs maintains that the department intends to implement BreEZe at those 19 regulatory entities, it lacks a plan to do so. In fact, SPR 3.1 indicates that the project will end after the phase 2 regulatory entities implement BreEZe, and only after its successful implementation of that phase will Consumer Affairs reassess the best implementation approach for the phase 3 regulatory entities. However, the director of Consumer Affairs acknowledged that the department has not assessed the extent to which the business needs of the 19 regulatory entities will require changes to the system. Moreover, Consumer Affairs has not conducted a formal cost-benefit analysis to determine whether BreEZe is the most cost-beneficial solution for meeting those needs.

Additionally, the contracts Consumer Affairs executed with Accenture for developing BreEZe do not adequately protect the State. Consumer Affairs executed the BreEZe contracts with Accenture in September 2011, under the direction of the California Department of General Services (General Services). Although its role at that time was to administer state IT procurements and conserve the fiscal interests of the State, General Services and Consumer Affairs agreed to revise the contracts' terms and conditions during the procurement process, at Accenture's request, in ways that significantly increased risk to the State. During the request for proposal (RFP) bidding period (RFP phase), General Services provided every potential bidder with the opportunity to submit a protest for issues such as the selection of prequalified bidders or RFP requirements before submitting a bid and to have General Services review its concerns. During the RFP phase in the BreEZe procurement process in January 2011, only Accenture submitted a protest, in which it proposed modifications to the State's standard IT general provisions and model contract language (standard IT contract).5 Of the 44 modifications to the State's standard IT contract that Accenture proposed, General Services accepted 18, proposed its own revisions to 19, and rejected just seven. Subsequently in April 2011, in accordance with state law, Consumer Affairs entered into a negotiation with Accenture during which further changes were made to the contract, with General Services' approval. However, some of those accepted changes to the standard IT contract's terms and conditions decrease Consumer Affairs' ability to obtain rights to work product that Accenture builds if Consumer Affairs terminates the contracts early, and they reduce Consumer Affairs' financial protections in the event of intellectual property rights violations.

Although General Services cited reasons for approving the modified terms and conditions in the BreEZe contracts, we question the prudence of some of the decisions it and Consumer Affairs made, as they increased Consumer Affairs' financial risks related to these contracts. CalTech's current authority over procurements for IT projects, a role that was not in place at the time the BreEZe contracts were being negotiated, together with its authority for approving and overseeing IT projects, position it well to ensure that future IT procurements do not jeopardize the State's financial interests.

Various stakeholders of the Board of Registered Nursing (BRN), one of the 10 phase 1 regulatory entities, raised concerns about the timeliness with which it has processed applications after implementing BreEZe in October 2013. According to BRN, it has faced significant delays in processing license applications and has been forced to modify its business processes since implementing the BreEZe system. However, although BRN asserted that it was exceeding the maximum time frames for processing certain applications and was facing a backlog of applications after implementing the system, we found little evidence demonstrating that it consistently tracks the information needed to support such claims.

For the selection of applications we reviewed, BRN processed these applications, on average, well within the allowable maximum time frames. However, we did determine that as of September 2014, BRN had a significant number of applications that were pending its review—more than 7,000, of which 63 had already exceeded the respective maximum processing time frames. Yet because BRN does not formally track this information, it cannot adequately assess its workload.

Additionally, BRN indicated that it has faced, and continues to face, obstacles in its implementation of the BreEZe system; for example, the system requires that staff take additional steps to enter applicant information. However, BRN does not track the information needed to assess the impact of such obstacles. Further, because it believes its efficiency in processing applications has decreased since implementing BreEZe, it has requested additional staff it believes it needs to process applications within required time frames. However, this request is based on data from the two fiscal years preceding BRN's implementation of BreEZe. Thus, because the analysis BRN used to support its need for the additional positions does not reflect its current workload and business processes since implementing the BreEZe system, the additional positions it requested are not adequately justified.

Most of the executive officers of the 10 phase 1 regulatory entities are generally dissatisfied with their BreEZe experience because it has not met their expectations. We interviewed the executive officers of each of the regulatory entities that have implemented the system regarding various aspects of their experience with the project, including their satisfaction with BreEZe and their overall experience with the system. Each regulatory entity reported experiencing issues with certain aspects of the BreEZe project. For example, the majority were unsatisfied with the testing they were able to conduct before implementing the system, and most found the training to be inadequate. In addition, all 10 of the executive officers indicated that BreEZe's reporting capability was unsatisfactory. Of greater concern, most executive officers reported that BreEZe has decreased their regulatory entity's operational efficiency.

RECOMMENDATIONS

CalTech

To help ensure the success of the BreEZe project going forward, CalTech should ensure that Consumer Affairs responds promptly to, and adequately addresses, concerns the IPO specialist and the IV&V consultant raise.

If Consumer Affairs receives the necessary funding and resources to successfully implement BreEZe at the phase 2 regulatory entities and the project continues to face escalating costs, CalTech should require Consumer Affairs to analyze the costs and benefits of moving forward with the project as planned versus suspending or terminating the project.

To ensure that future IT project procurements do not jeopardize the State's financial interests, CalTech should document its reasons for approving any deviations from standard contract language.

Consumer Affairs

Consumer Affairs should develop a process to ensure that it undertakes all required oversight activities with respect to BreEZe so that it can prevent or identify and monitor any problems as they arise. This includes taking steps to sufficiently respond to any concerns the IPO specialist and the IV&V consultant raise.

To ensure that BreEZe is a cost-effective solution to meet the business needs of the phase 3 regulatory entities, should it elect to pursue implementing BreEZe at these entities, Consumer Affairs should first complete a formalized cost-benefit analysis. This analysis should include an assessment of the potential changes those regulatory entities may require be made to the BreEZe system and the associated costs.

Consumer Affairs should continue to work with the phase 1 regulatory entities to ensure that the issues they are facing with BreEZe are being resolved in a timely manner.

BRN

To ensure that it has adequate data to effectively use its resources and manage its workload, BRN should do the following:

Conduct an analysis no later than June 30, 2015, of its application processing since implementing BreEZe to identify its workload capability. To the extent that it determines additional resources are necessary, BRN should submit a request for these resources that is appropriately justified.

AGENCY COMMENTS

Consumer Affairs and BRN agreed with our recommendations and outlined the actions they plan to take to implement them. Although CalTech states that our report's recommendations are for the most part appropriate and in line with actions and initiatives that it has already undertaken, it explained that it has general concerns with the report and did not indicate whether it agrees with our recommendations. Our comments on CalTech's response begin on page 125.

1 Although Consumer Affairs consists of 40 regulatory entities, only 37 of these entities were originally scheduled to implement BreEZe. Specifically, the Bureau of Real Estate and the Bureau of Real Estate Appraisers were brought under Consumer Affairs as a result of the governor's reorganization plan, effective July 2013, after the BreEZe project was approved and underway. According to Consumer Affairs, it planned to implement BreEZe at these regulatory entities once the system was fully implemented at the 37 regulatory entities. Another entity, the Arbitration and Certification Program, does not issue licenses and will not be included in BreEZe.

2 An SPR provides a summary of proposed changes to the original project cost, schedule, or scope. An SPR is generally required when the project costs or total financial program benefits deviate or are anticipated to deviate by 10 percent or more, or a major change occurs in project requirements or methodology.

3 The BreEZe project team developed the estimate informally and not in the same manner as an SPR requires.

4 There are three contracts related to the BreEZe project—one contract for design, development, and implementation; another contract for maintenance support; and a third contract for the system license. When we discuss a specific contract, we identify it as either the design, maintenance, or system license contract.

5 At the time of the BreEZe procurement, General Services had several modules of standard contract language related to IT contracts.