The Governor of California
President pro Tempore of the Senate
Speaker of the Assembly
Sacramento, California 95814
This letter reports on the Financial Information System for California (FI$Cal) project’s substantial deviations from its plan approved in the sixth Special Project Report (SPR) and the FI$Cal project office’s (project office) plan to create a seventh SPR to address those deviations. SPRs provide a basis for understanding and agreement among stakeholders and justify any changes to those agreements. This letter also reports on other challenges facing the project and provides an update on previously reported issues. Below is a list of the highlights contained in this letter:
Sources: Sixth Special Project Report, February 2016; October 2017 IPO Report; Accenture LLP’s fifth contract amendment; Public Consulting Group’s first contract amendment; and technology department staff.
* According to the IPO, the project office was only able to provide financial information through the month of July 2017 due to the timing of its month‑end closing activities.
FI$Cal is an information technology (IT) project that is intended to combine the State’s accounting, budgeting, cash management, and procurement operations into a single financial management system. The objective of the project is to implement a unified and consistent financial system that will eventually be used by most state entities. Accenture LLP (Accenture) is the project’s system integrator, which is the role responsible for merging the various components of FI$Cal into a single product and ensuring that those components function together. In addition, the California Department of Technology (technology department) provides Independent Project Oversight (IPO) and the project office contracts with Public Consulting Group for Independent Verification and Validation (IV&V) services. The purpose of IPO is to provide an independent review and analysis to determine whether the project is properly managed and on track to be completed within the estimated schedule and cost, and that it will provide the functionality the State requires. In contrast, the IV&V function provides technical oversight of an IT system’s products and processes as they are being developed and implemented. IV&V feedback provides an early warning if quality issues arise to limit later problems or system changes that might result in greater costs and schedule delays. Both oversight entities report their concerns and recommendations to the project office monthly. In addition, FI$Cal is governed by a steering committee composed of stakeholders from various departments and the chair of the Customer Impact Committee, which acts as the primary customer representative for all departments. Furthermore, state law established the Department of FI$Cal in July 2016, which provides a permanent administrative structure for the project. We provide some key facts about FI$Cal in the text box.
In our January 2017 report, we raised concerns that the project office continued to experience significant scheduling challenges, despite its release in early 2016 of a sixth SPR that extended the project’s completion date by two years. We also reported that as of October 2016, the project office was behind schedule in its implementation of the State Controller’s Office’s (State Controller) and State Treasurer’s Office’s (State Treasurer) control agency functionality (control functionality). These two control agencies maintain the State’s financial records that other state entities use to reconcile their accounts and to determine the State’s financial status. In May 2017, the steering committee voted to postpone the implementation of the State Controller’s and State Treasurer’s control functionality, which the project’s sixth SPR scheduled for a July 2017 release. This postponement was in part a result of multiple delays in the design and testing phases of the State Controller’s and State Treasurer’s release. The project office has indicated that it currently plans to begin releasing the two agencies’ control functionality in phases during the 2018 release. However, we have concerns with this approach, which we describe in the following section.
In addition, we reported in January 2017 that the project office had failed to respond promptly to its oversight entities’ concerns and recommendations, many of which had been outstanding for more than a year. We also noted that the project office did not adequately address the technology department’s approval conditions for the sixth SPR, which required the project office to develop, among other things, detailed plans on the following: deploying FI$Cal to the remaining state entities, transferring technical knowledge from Accenture staff to state staff responsible for the project’s operation and management (knowledge transfer), and transferring operational control and maintenance of the system from Accenture to the State. Furthermore, we questioned the IV&V’s independence due to the fact that the project office is primarily responsible for selecting and paying for IV&V services. We provide an update on these concerns later in this report.
Although the project has experienced improvements in the area of entities closing their month‑ and year‑end financial statements, some entities continue to experience difficulties with this process. The State Controller uses the year-end financial statements to prepare the State’s Comprehensive Annual Financial Report, which informs stakeholders—such as credit agencies that rate the State’s bonds—of the State’s financial position and how the State manages public resources. Although the IPO reported that the project has made positive progress with entities closing their month-end statements, some entities are still not closing their year‑end statements in a timely manner. The IPO stated that as of October 2017, 48 of the 58 entities deployed prior to the 2017 release have closed their year‑end financial statements on time. The project’s chief deputy director (chief deputy) stated that the project office has created a dedicated team to support state entities in closing their month-end and year-end statements. However, given the difficulties that the project office has had with smaller entities being unable to close out their financial statements on time, the project office will likely face challenges during the 2018 release when deploying FI$Cal to large state entities such as the Department of Public Health, which has approximately 3,600 authorized positions and accounts for its activities in more than 40 different funds.
Finally, in our April 2012 letter report, we recommended that the Legislature require the project office to track costs for state departments’ subject matter expert staff, monitor the benefits that FI$Cal is projected to provide, and report the cost and reason for any significant and unanticipated customizations that the project office makes to the FI$Cal software. The project office is already required to include in its annual report to the Legislature, among other things, a description of any significant software customizations and the reasons for them, but not the associated cost unless it is significant or causes a deviation to budgeted or overall project costs. We are unaware of any legislative action taken to address our recommendations.
The project office plans to issue its seventh SPR in early 2018, roughly two years after it last extended its schedule by two years and increased its estimated project cost by $237 million in its sixth SPR. According to the chief deputy’s statements at the October 2017 steering committee meeting, the seventh SPR will focus primarily on three areas: the State Controller’s and State Treasurer’s control functionality, the transition to maintenance and operations of previously implemented functionality, and the project’s release schedule for remaining state entities. Figure 1 shows the historical changes to FI$Cal’s budget and estimated completion dates since the fourth SPR, as well as future anticipated changes.
Sources: FI$Cal’s fourth through sixth Special Project Reports, October 2017 steering committee meeting minutes, and interviews with FI$Cal project staff.
* Information for SPR #7 is based on statements made by the chief deputy director at the October 2017 steering committee meeting and discussions with project management.
Despite previous postponements of the State Controller’s and State Treasurer’s control functionality, the project office determined that it would need to further postpone the two agencies’ control functionality and begin working on a new SPR. According to the chief deputy, the seventh SPR will describe the project office’s plan to deploy the two agencies’ control functionality. Although the project office does not plan to make significant changes to the State Treasurer’s release, it plans to change the scope of the State Controller’s release. Rather than transitioning completely to FI$Cal as previously planned, the project office plans to integrate FI$Cal with the State Controller’s current financial system during the 2018 release. Specifically, the project office plans to build multiple interfaces so the FI$Cal system can exchange data with the State Controller’s current system, with the intention that the two systems operate in sync until the State Controller transitions completely to FI$Cal at a future, as yet unspecified, date. According to the chief deputy, this approach was necessary to mitigate the risk associated with the complexity of the State Controller’s control functionality. Because this work will require additional software development, the project office plans to renegotiate its contract with Accenture to encompass this additional work. Although the sixth SPR did not anticipate such costs, the chief deputy asserts that the project office will not need to request additional project funding in the seventh SPR because it has achieved significant cost savings. Nevertheless, the IPO believes that the project office’s plan to build multiple interfaces constitutes a considerable increase in the project’s scope of work. However, according to the IPO, the technology department will assess the sufficiency of the project’s funding as part of its review and approval of the project’s seventh SPR.
In addition, the project may encounter further challenges if one of the IV&V’s concerns materializes. Specifically, the IV&V has expressed a concern that the FI$Cal system and the State Controller’s current system could encounter reconciliation errors. If such reconciliation errors occur and result in material differences between the two systems, the IV&V stated there would likely be a significant amount of work to identify and correct the errors. The chief deputy acknowledged that this approach will require the project office to exercise a high level of discipline as it implements best practices to address this risk. Nevertheless, if the State Controller’s control functionality is not fully deployed during the 2018 release, the project’s total cost may increase.
The project office plans to end project related activities by July 2019, even though some planned entities might not have fully implemented FI$Cal by that time. According to the chief deputy, the project’s upcoming SPR will not extend the project’s schedule beyond the July 2019 completion date established in the sixth SPR. However, as we have previously reported, we continue to be troubled about the number and size of entities that have yet to implement FI$Cal. The project office currently plans to bring 79 state entities onto FI$Cal during the 2018 release, which is the largest rollout to date. Given the large number and the relative complexity of some state entities, the project office may face challenges in implementing FI$Cal at all of them.
According to the chief deputy, the project is on track for deploying FI$Cal to the remaining state entities and it does not plan to include any further releases after 2018 in its seventh SPR. Although the project office believes that it will successfully deploy FI$Cal to the remaining state entities, its oversight entities have reported concerns about the 2018 release: the IV&V reported in October 2017 that the 2018 release contains some of the largest entities in the State and is more complex than the 2017 release and it further suggested that the project may not have adequate resources to support the implementation of all the entities in the release. Similarly, the IPO reported as of October 2017 that the project’s high staff vacancy rate of 20 percent, with 294 filled positions out of 368 authorized positions, could potentially impact project work and lead to challenges in meeting the project’s scope and schedule. The chief deputy stated that the 2018 release will be manageable because he believes that the project office has addressed most of the technical hurdles in prior releases. However, he acknowledged that implementing the significant number of departments during the upcoming release will be a challenge for other nontechnical reasons, such as training users and identifying role assignments in the system for state staff.
Furthermore, the 2018 release includes significantly larger and more complex entities than previous releases, such as the Department of Health Care Services, the Department of Social Services, and the Employment Development Department. Additionally, as Figure 2 shows, the project office has yet to implement FI$Cal at nearly half of the planned state entities, which represent 67 percent of the total positions and accounts for more than 90 percent of the total funding in the fiscal year 2017–18 State Budget for the state entities scheduled to use FI$Cal.1
The project office’s plan to begin deploying the State Controller’s and State Treasurer’s control functionality at the same time as the 2018 release may further exacerbate the risk of state entities missing that release. In its sixth SPR, the project office identified implementing state entities at the same time as control agencies as a significant risk. In addition, the project office identified that its resources are stretched in preparing for the 2018 onboarding of entities. Given these issues, our information technology expert (IT expert) believes there is a critical risk that the project’s resources will be overwhelmed when it tries to release both the large number of state entities and the two control agencies’ functionality during the 2018 release.
We are also concerned that the seventh SPR may propose a time-based approach to defining project completion, rather than the scope‑based approach previously established in past SPRs. Previous SPRs included within the project’s scope the implementation of FI$Cal at all state entities except those that intended to convert to FI$Cal when their own business management systems become obsolete or those that have a statutory provision allowing them to not use FI$Cal. The chief deputy has indicated that the Department of FI$Cal would handle any scheduled entities that were not implemented in the 2018 release on a case‑by‑case basis as part of its ongoing maintenance and operations activities. Maintenance and operations include the costs associated with the ongoing upkeep of a system and the costs associated with continued use of a system, which may include personnel and other resources. For example, the project office plans to use maintenance and operations to address its ongoing work with state entities’ previously implemented functionality. However, maintenance and operations costs for implementing FI$Cal after July 2019 at additional state entities may not be tracked as part of the project’s costs. Instead, if a state entity does not complete its implementation of FI$Cal during the 2018 release, its implementation costs may lose their identity and be captured as part of the Department of FI$Cal’s ongoing maintenance and operations after the project has ended. This will make it difficult for stakeholders to determine the true final cost and timeline of the project. The chief deputy has stated that the upcoming SPR will address the tracking of costs related to entities that miss the 2018 release. Nevertheless, if the upcoming SPR does not address the future costs of entities that are unlikely to implement FI$Cal fully until after the 2018 release, the true costs of the FI$Cal project as initially defined may be obscured. Although the deputy state chief project officer of the technology department asserts that these costs would be addressed through the Governor’s budget process, we believe this process would require more effort on the part of stakeholders, thus diminishing the transparency of the project’s complete cost.
Sources: Fiscal Year 2017–18 State Budget, documents from FI$Cal’s website, October 2017 IPO Report, FI$Cal’s Special Project Report #6, and discussions with IPO staff.
Note: According to the sixth special project report, the number of departments and number of users, among other factors, drive the effort of a release. We determined the number of entities by counting each of the unique department codes that the project office listed in its release documentation and compared those entities to the fiscal year 2017–18 State Budget. If the project office identified a state entity as having implemented or planning to implement FI$Cal’s functionality across multiple waves or releases, we counted it along with its total funding and its total authorized positions in each respective wave or release. A total of five state entities implemented or plan to implement FI$Cal across multiple waves or releases.
* According to the IPO, the project office was only able to provide financial information through the month of July 2017 due to the timing of its month‑end closing activities.
† The number of positions does not reflect the number of actual users; however, it gives an indication of the size of the state entity.
We continue to track concerns raised by the IPO and IV&V and the resulting recommendations those entities have made to the project office. Although the project office has taken action to address some of the oversight entities’ concerns and recommendations, it has left several unaddressed and has been very slow in addressing others. By failing to respond adequately, the project office exposes itself to risks that could threaten a successful system implementation and result in quality and reliability issues as well as additional schedule delays and cost overruns.
Although the project office appears to have resolved some of the IPO’s concerns, it has not adequately addressed others. For example, the IPO noted that as of October 2017, the project office has not fully resolved two concerns that the IPO originally identified more than a year ago. Specifically, one of the IPO’s concerns from May 2016 was that the project office had not provided sufficiently detailed time and cost analyses for changes to its workload. The other concern from September 2016 related to the project office’s lack of regularly updated schedules. Although the IPO notes that this issue may be partly addressed in the seventh SPR, we are uncertain that the upcoming SPR will address all of the IPO’s previous concerns. As of October 2017 the IPO reported that the project office has still not sufficiently addressed three of the five conditions the technology department established for approving the project’s sixth SPR more than a year ago; namely, developing an updated project schedule that allocates resources to tasks and milestones, providing a detailed plan for deploying FI$Cal to remaining state agencies, and providing a detailed plan for the transfer of operational control and maintenance of the system from Accenture to the State. The IPO further reported that it was reviewing the project’s updated plan for the transfer of operational control and maintenance. Given that the IPO has not signed off on most of the technology department’s approval conditions for the sixth SPR, we would expect the technology department to withhold its approval of the seventh SPR until the IPO determines that the project office has fully addressed the issues related to these outstanding conditions.
We also reviewed the concerns and recommendations reported by the IV&V as of October 2017 and found several issues that the project office had not adequately addressed. The IV&V reported that it had 11 open concerns that could affect the project’s quality, manageability, cost, or schedule—four of which have been open for more than a year. In fact, one concern has been open since December 2012. These concerns include the transfer of technical knowledge from Accenture to state staff, as well as issues with ensuring the proper planning and design of the integration between FI$Cal and the State Controller’s existing system. In Table 1, we summarize select concerns raised by the IV&V.
|Date Issue opened||Category||Condition||IV&V Summary Recommendation||time Outstanding as of October 30, 2017|
|Knowledge Transfer to FI$Cal Personnel||Accenture is making changes to certain key software configurations settings.||Ensure Accenture provides proper documentation that will allow the State to maintain and upgrade software without the help of consultants after Accenture completes the project.||4 years, 10 months|
|Security||There is no established process to perform periodic reviews of FI$Cal accounts to determine the appropriateness of group membership or to identify terminated user accounts.||The State should establish a policy of periodic reviews of accounts for appropriateness and conduct the initial review.||3 years, 4 months|
|March 2015||Knowledge Transfer to FI$Cal Personnel||Accenture resources are not consistently providing knowledge transfer to FI$Cal staff. IV&V is concerned that FI$Cal staff are not receiving the necessary knowledge transfer due to Accenture resource constraints.||IV&V recommends that Accenture and FI$Cal management ensure that essential knowledge transfer activity sessions are being scheduled and provided on a regular basis.||2 years, 7 months|
|Knowledge Transfer to FI$Cal Personnel||Only one Accenture resource has knowledge of the project's key function related to current user roles and workflow. There is no documentation of how the associated data are created and used in the testing environment.||Request that Accenture provides documentation that describes how workflow data are created and used in the testing environment so that the State can maintain and support key software when the single Accenture resource leaves FI$Cal.||2 years, 4 months|
|Maintenance and Operations||The project is delaying the maintenance upgrades of the accounting function. Delaying and accumulating maintenance upgrades may result in avoidable production errors and more complex and time‑consuming upgrades in the future.||Regularly review available upgrades and plan for more timely deployment of desired enhancements and new features.||4 months|
|Functional and Technical Design||There is insufficient review of technical documentation before implementing changes in the production system, which is the version of the system used by state entities. This may lead to implementing avoidable problems and documenting the production system inadequately, thus making the system more difficult to maintain when the State takes over.||All proposed design changes should be reviewed and approved by project staff and oversight entities before entering the production system. Project staff should review documents for completeness and accuracy to ensure that the State has a smooth transition when it takes over maintenance of the applications. If project staff is unable to perform a full review of the documentation, the State should ensure that Accenture updates the documents within 90 days of the design changes being made.||2 months|
|Maintenance and Operations Readiness||The project is not effectively testing its proposed changes to the production system, which is resulting in errors.||Perform testing on more of the proposed changes and establish criteria for that testing.||1 month|
|Data Conversion and Interfaces||The project's proposal to integrate FI$Cal with the State Controller’s Office's (State Controller) current financial system is complex. If appropriate steps are not taken to prevent, detect, and recover from synchronization errors, FI$Cal may encounter data integrity issues and extended system downtime.||Create a comprehensive mitigation strategy that addresses, but is not limited to, the prevention, identification, and recovery of data errors. The plans should also include the processes and software necessary to support this complex proposal. Finally, the plans should also include the approach to coordinate maintenance between FI$Cal and the State Controller’s legacy system.||1 month|
|2018 Deployment of State Entities||The 2018 release involves more departments and is more complex than the prior release. Inadequate project resources to support the 2018 release of state entities could lead to longer resolution times for individual entity’s implementation issues and could threaten the success and timeliness of entities going live with FI$Cal.||Determine the resources needed to support the 2018 release and implement plans to obtain sufficient resources immediately.||N/A
(Opened in October 2017)
Source: California State Auditor’s analysis of Independent Verification and Validation Reports for December 2016 through October 2017.
Note: The table contains unresolved findings reported by the new IV&V as medium risk or above, as well as other unresolved findings that we had previously reported. The IV&V defines a medium risk finding as having a possibility of moderate impact on project quality, manageability, cost, or schedule—some disruption is likely, and mitigation strategies should be implemented as soon as possible.
Although the project office appears to have made some progress in transferring technical knowledge from Accenture to state staff, this issue continues to be a problem for the project. As of October 2017, three of the IV&V’s 11 concerns are related to knowledge transfer. For example, in one case opened in December 2012, the IV&V reported that the State had made the task of reviewing and approving documentation describing Accenture’s configuration of the software a lower priority. Although the IV&V recently reduced the risk rating of these knowledge transfer findings, the project office’s efforts to clear these findings may have been hindered, in part, by resource constraints. Nevertheless, if the project office does not ensure the necessary transfer of knowledge between Accenture and state staff, we believe the State will have to continue to rely on external contractors for support after the project is complete. This would likely end up costing the State much more than if state staff were adequately involved in knowledge transfer activities and were able to effectively operate and maintain FI$Cal after Accenture’s departure.
By not fully addressing the concerns raised by its oversight entities, the project office places itself at greater risk of experiencing further delays and additional costs. Moreover, we have concerns with the IPO’s and IV&V’s oversight of the project, as we discuss further in the next section.
Since our last report, we have identified new concerns regarding the effectiveness of the IPO’s and IV&V’s oversight of the project. For example, we identified multiple unplanned outages that occurred in FI$Cal—incidences that made the system unavailable to users during regular business hours. Our IT expert explained that outages can be expensive for the State because of the large number of FI$Cal users who are unable to perform their work in the system. In fact, from July 2017 through November 2017, we identified at least 17 system outages during regular business hours, with outage periods ranging from approximately 12 minutes to more than five hours. Despite the importance of this problem, the IV&V did not raise any specific concerns regarding these outages. This issue went unreported for four months following the first outage that we identified, until the IPO eventually reported on this issue at the end of November 2017. According to the IPO manager, the IPO reports did not mention these outages for several months because outages and the causes of outages were previously tracked by the IV&V team. He further stated that after the IPO noticed that the IV&V was no longer tracking these things, the IPO began tracking them. The IPO stated that it requested the required analyses of the causes of these outages from the project office, but it had not yet received the analyses as of October 2017. When we spoke with the IV&V consultant, he stated that the IV&V was responsible for reviewing the project office’s process for identifying the root cause of the outages but not the outages themselves. However, by not reporting on these outages in a timely manner, the project’s oversight entities are failing to inform stakeholders of potential issues with FI$Cal and the associated costs.
Recent changes in the staffing of the IV&V may present further challenges. In September 2017, a new IV&V project manager replaced the manager who had been assigned to the project for more than seven years. This loss of institutional knowledge may prove challenging for the IV&V. The IPO also reported in October 2017 that it had concerns with the IV&V’s level of staffing. Specifically, the IPO stated that the current number of IV&V staff may not be sufficient for its workload associated with the 2018 release, as there would only be two IV&V team members remaining at the end of November instead of three as its contract requires: a project manager and two staff-level employees. Without adequate staffing the IV&V may be less effective in providing technical oversight to the project.
Further, we have previously reported that in order to preserve the independence of IV&V services, we believe the project office should not directly manage the IV&V contract and that this responsibility should reside with another state entity, such as the Government Operations Agency. Making this change may better ensure that IV&V services are provided by an organization that can maintain adequate independence, including keeping the organization separate from the organization responsible for implementing the system, which is the project office. This ensures that the IV&V can deliver findings and recommendations to stakeholders without restriction, fear of retaliation, or coercion. In light of the issues we discuss in this report, we believe that responsibility over the contract for IV&V services should not reside with the project office and should instead be held by another agency.
Finally, the IPO was late in submitting seven of its monthly oversight reports since we issued our last report. Reviewing these reports represents a key part of our monitoring efforts. The Statewide Information Management Manual requires IPO consultants to issue their oversight reports by the 10th working day of the month for a project as critical as FI$Cal; however, the IPO has become increasingly late in submitting its reports. A chief with the technology department’s project approvals and oversight branch stated that delays in its reporting occurred because the complex nature of FI$Cal’s current replanning efforts required additional focus and review from technology department management and critical partners to ensure accuracy of the reports. Nonetheless, these delays hinder the project office’s and others’ abilities to promptly identify and address potential problems that could negatively affect the project’s cost, schedule, and scope.
As part of our statutory role, we continue to attend monthly oversight meetings and monthly steering committee meetings. We also obtain briefings from the project office after it holds executive working group meetings.2
Pursuant to Government Code section 11864, the California State Auditor (State Auditor) is required to independently monitor the FI$Cal project throughout its development, as the State Auditor deems it appropriate. Our independent monitoring includes, but is not limited to, monitoring the contracts for IPO and IV&V services, assessing whether concerns about the project raised by the IPO and IV&V staff are appropriately addressed by the FI$Cal steering committee and the FI$Cal project office, and assessing whether the FI$Cal project is progressing within schedule and budget. We are required to report on the project’s status at least annually. This is the 14th report we have issued since we began our monitoring in 2007 and our seventh report since the project began the design, development, and implementation phase in June 2012.
We will continue to monitor and report on these topics in addition to others that come to our attention, at a minimum, on or before January 10 each year.
ELAINE M. HOWLE, CPA
January 9, 2018
Nicholas Kolitsos, CPA, Audit Principal
Hunter Wang, CFE
Joseph L. Porche, Staff Counsel
For questions regarding the contents of this report, please contact Margarita Fernández, Chief of Public Affairs, at 916.445.0255.
1 The number of positions does not reflect the number of actual users; however, it gives an indication of the size of the state entity. Go back to text
2 Executive working group meetings are meetings of the project’s executives to discuss significant project issues before they are formally presented to the steering committee. Go back to text
Back to top