Report 2010-122 Recommendations and Responses in 2012-041

Report 2010-122: California Department of Transportation: Its Capital Outlay Support Program Should Strengthen Budgeting Practices, Refine Its Performance Measures, and Improve Internal Controls

Department Number of Years Reported As Not Fully Implemented Total Recommendations to Department Not Implemented After One Year Not Implemented as of Most Recent Response
Department of Transportation 1 11 7 5

Recommendation To: Transportation, Department of

To improve accountability internally and with the public, Caltrans should create and incorporate an analysis of support cost budget variances in its quarterly report to the agency and in its annual report to the Legislature and the governor. The analysis should report on the number of completed projects with budget variances and on the number of open projects for which the estimates at completion predict budget variances. Further, the analysis should report on the overrun and underrun ratios for those projects, and the portions of the variances due to rates and hours. Also, Caltrans should include in its strategic plan a measurable goal for reducing variances.

Response

This recommendation will be fully implemented when the report is transmitted to the Legislature on November 15, 2012. The performance measure that targets support expenditures that are within a specified range of the support budget was not included in the 2011 annual report to the Legislature due to timing issues, but will be included in the 2012 annual report.

Caltrans has not included in its report an analysis of variances due to rates or hours because the largest impact to project costs are currently due to changes in scope of work such as additional permit requirements and mitigation requirements. As the audit report noted, the primary cause for cost overruns in the projects reviewed was the increase in hourly rates. Caltrans does not anticipate this being the case any time in the near future. However, Caltrans does perform a separate analysis of rates each year for planning purposes.

Recommendation To: Transportation, Department of

To improve performance metrics related to the support program, Caltrans should devise, use, and publicize a consistent method for reporting the support-to-capital ratio on its Web site and in other reports to the public. Further, Caltrans should recalculate past support-to-capital ratios using the method devised to allow for comparison across years.

Response

This recommendation was fully implemented in December 2011. However, Caltrans' status updates provided a combined response to 1.2.a and 1.2.b. The recommendation to 1.2.b was not fully implemented by the one-year status update. We provide a separate status for 1.2.b.

In response to recommendation 1.2.a, Caltrans updated the website with re-calculated ratios and a consistent methodology in December 2011. The website was updated in September 2012, with the 2012 results. Please see Attachment 2A-1 which consists of the Annual Overall Program Measure. The updated information can be found at the following link:

http://onramp.dot.ca.gov/hq/projmgmt/index.jsp?pg=106

Recommendation To: Transportation, Department of

To improve performance metrics related to the support program, Caltrans should develop goals—and publicly report on the progress against those goals—for the support-to-capital ratio, based on project type—State Transportation Improvement Program or the State Highway Operation and Protection Program—and project size.

Response

Caltrans established support-to-capital ratio goals based on project size. These goals and results can be found at the following link:

http://onramp.dot.ca.gov/hq/projmgmt/index.jsp?pg=106

Caltrans also developed an annual measure which includes all programmed STIP and SHOPP projects that completed construction during each fiscal year for which Caltrans was responsible for development and construction management. A copy of the Five Capital Cost Ranges is attached as Attachment 2B-1. This measure was published in the 3rd Quarterly Project Delivery Report. A copy of the report is included as Attachment 2B-2 and can also be found at the following link:

http://www.dot.ca.gov/hq/transprog/ctcbooks/2012/0612/033_3.9_Q3_Project_Delivery_Rept.pdf

The 4th Quarterly Project Delivery Report is included as Attachment 2B-3 and can be found at the following link:

http://www.dot.ca.gov/hq/transprog/ctcbooks/2012/0912/26_3.11.pdf

Recommendation To: Transportation, Department of

To improve performance metrics related to the support program, Caltrans should continue to explore the use of additional metrics, such as a measure based on a productivity index as described in a March 2011 draft study by the University of California, Davis.

Response

As Caltrans reported in the one-year status update, the additional metric was produced by the target date of July 2012. However, the Production Study identified four potential measures (a copy of the Production Study is attached.) Caltrans has reviewed the four measures and believes that measure four is a viable measure. Measure four is similar to the support-to-capital goal and uses hours to mitigate labor cost charges and indexes capital to mitigate capital escalation changes.

However, Caltrans is not prepared to adopt this as a full measure until the following is done:

• Additional years of data are needed to ensure the method reflects long-term production information.

• Additional analysis is needed to establish trends based on project groups that are based on the capital value.

• Additional analysis is needed to establish trends based project groups that are based on the capital value.

• Additional effort is needed to evaluate the pre-construction phase. The construction phase hours were evaluated first to see if there was any potential for a reasonable measure.

Once the issues discussed above are complete, Caltrans will have a better time estimate for full implementation of this recommendation. In the meantime, Caltrans is moving forward with the following timeline:

• Evaluate and determine a potential measure for project groups by capital value by January 2013.

• Evaluate and determine potential pre-construction measure by July 2013.

Caltrans believes that the effectiveness of a long-term measure will not be known for several years. The existing data sample was based on the past four years but additional years are necessary in order to have a complete set of data. Caltrans is also considering the impacts on productivity by the ongoing furloughs and the constraints of limited overtime hours.

Recommendation To: Transportation, Department of

To ensure that it monitors the status of projects, Caltrans should implement earned value management throughout its districts in a manner similar to the implementation in the Los Angeles district. To allow for performance evaluation of project work, Caltrans should ensure that these performance metrics are available at the task level for both active and completed projects. Caltrans should instruct districts to aggregate this information for all projects by task level, to better assess the effectiveness and efficiency of support costs by task level. Caltrans should also make available to project managers graphical displays of project cost and schedule performance.

Response

As noted in Caltrans' one-year response, a Project Delivery Directive (PD-08) on Earned Value Management (EVM) was issued on April 5, 2012. PD-08 provided staff with a framework in terms of policy, definitions, roles and responsibilities for consistent implementation of EVM. Full implementation of EVM is contingent on the roll-out of the Project Resource and Schedule Management (PRSM) application which will provide earned value metrics and reports at the task level for all projects in the system. Caltrans is on track for a complete roll-out of PRSM by June 2013. Currently, Caltrans utilizes reports which graphically depict cost and schedule performance for all projects.

Recommendation To: Transportation, Department of

To better address costs associated with the support program, Caltrans should ensure that the PRSM system contains strong controls that ensure employees only charge time to projects and phases for which they are assigned.

Response

This recommendation will be fully implemented by June 2013, when the new Project Schedule and Resource Management (PRSM) system is fully implemented.

Recommendation To: Transportation, Department of

To better address costs associated with the support program, Caltrans should commission an independent study of the costs and benefits of using consultants to address temporary increases in workload and, if the study reveals cost savings, use consultants. To the extent possible, Caltrans should also use temporary staff appointments for temporary increases in workload when consultants are unavailable.

Response

As reported in the one-year response, Caltrans Division of Research and Innovation contracted for an independent study to identify options or tools to improve decision-making processes regarding resource mix during workload peaks and valleys. This final report is expected to be completed by March 2013, at which time this recommendation will be fully implemented.

Current Status of Recommendations

All Recommendations in 2012-041