Report 2020-104 All Recommendation Responses

Report 2020-104: Calbright College: It Must Take Immediate Corrective Action to Accomplish Its Mission to Provide Underserved Californians With Access to Higher Education (Release Date: May 2021)

Recommendation for Legislative Action

Legislature
To ensure that Calbright provides educational and economic opportunities to Californians and is accountable for its performance, the Legislature should do the following:

1. Require Calbright to demonstrate substantive compliance with our audit recommendations.

2. Require the California State Auditor (State Auditor) to provide an update to the Legislature by no later than December 2022 about Calbright's progress in implementing those recommendations.

3. Adopt a sunset provision that would eliminate Calbright as an independent community college district if the State Auditor determines that Calbright has not demonstrated substantive compliance with those recommendations by December 2022.

4. If it eliminates Calbright, the Legislature should explore other options for providing competency-based education for California adults who face barriers to traditional postsecondary education.

Description of Legislative Action

AB 2820 (Medina and Quirk-Silva, 2022) would make the California Online Community College Act inoperative on January 1, 2024, and would appropriate CalBright's funding to other community college purposes. As of June 10, 2022, this bill is in the Senate Education Committee.

AB 1432 (Low, 2021) would have made the California Online Community College Act inoperative at the end of 2022-23 academic year. This bill died in the Senate in 2021.

California State Auditor's Assessment of 1-Year Status: Legislation Introduced

As of June 10, 2022, AB 2820 (Medina and Quirk-Silva, 2022) is pending in the Senate Education Committee.

Description of Legislative Action

AB 1432 (Low, 2021) would make the California Online Community College Act inoperative at the end of the 2022-23 academic year. This bill is pending in the Senate.

California State Auditor's Assessment of 6-Month Status: Legislation Introduced

Description of Legislative Action

As of July 10, 2021, the Legislature has not taken action to address this specific recommendation. However, AB 1432 (Low) would make the California Online Community College Act inoperative at the end of 2022-23 academic year. Additionally, the Budget Act of 2021 (SB 129, Skinner), which is enrolled and pending the Governor's action, would redirect funds appropriated to Calbright should AB 1432 be enacted.

California State Auditor's Assessment of 60-Day Status: No Action Taken

Recommendation #2 To: Calbright College

To provide greater accountability regarding its spending and to ensure that it effectively uses the public funds it receives to accomplish the goals for which it was created, Calbright should do the following:

1. By November 2021, incorporate into its implementation plan a spending plan that details how and when it expects to spend the funds the Legislature allocates to it. At a minimum, the spending plan should identify the estimated costs to accomplish the tasks set forth in its implementation plan and a timeline for when it expects to incur those costs. The spending plan should also describe Calbright's strategy for staying within its budget while completing necessary start-up activities and achieving its milestones.

2. Calbright should annually review the spending plan and make adjustments as necessary. It should also annually report to the Board of Governors on its spending to date and explain how its spending has furthered its progress in achieving its mission.

Annual Follow-Up Agency Response From October 2023

In November 2021, Calbright incorporated into its Implementation Plan an aligned Spending Plan. The California State Auditor's office indicated this recommendation would be complete following the first annual review of the Spending Plan and report to the Board of Trustees.

The first annual review of the Spending Plan (SP) took place late 2022 and early 2023, which led immediately into the College's FY 2023-2024 budget planning. Calbright's annual budget is the manifestation of the SP and advances the strategy and priorities outlined in the internal and deliberative planning document. The review of the SP included consultation with Calbright's department heads and pertinent directors to ensure alignment in forward-looking planning, and was also updated to reflect prior fiscal years' actual spending. This review includes evaluation of anticipated personnel and resource needs, departmental priorities as they relate to the College's operations and Strategic Vision, and progress against deliverables in the Implementation and Spending plans. This review and update served as the foundation for initial development of Calbright's FY 2023-2024 budget, which aligns with and advances the College's priorities and Board-approved Strategic Vision.

Calbright's budget is created in collaboration with College leadership, department heads, chief financial officer, and Board of Trustees and is a multi-month process. Activities also include regular public updates to the Board of Trustees as well a formal adoption of the budget in the summer as part of regularly scheduled public Board meetings. The College also provides quarterly financial updates in Board meetings to ensure budget compliance and also presents, for approval by the Board, contracts the College seeks to engage in that exceed $100,000, and provides monthly updates on all external contract expenditures processed in the prior month as well as all significant expenditures made by the College.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Calbright has previously provided its implementation plan, which we found to include a spending plan that projected the costs associated with its planned activities for setting up the college. For this update, Calbright provided documentation showing its annual budget and quarterly updates to the board on that budget. It also provided its spending plan, which had been updated to include actual expenditures for fiscal years 2022 and 2023.

Annual Follow-Up Agency Response From November 2022

In November 2021, Calbright incorporated into its Implementation Plan (IP) an aligned Spending Plan, which provides greater accountability regarding the College's spending and ensures an effective use of public funds. The California State Auditor's office has indicated that this recommendation is currently partially implemented and will be considered fully implemented when the first annual review of the Spending Plan and report to Calbright's Board of Trustees is complete. Calbright anticipates conducting an annual review of the Spending Plan on schedule and reporting to the Board of Trustees during the first quarter of 2023.

The first annual review of the Spending Plan will begin in November 2022. Following an internal review, the College will provide a report to the Board of Trustees about how the Plan has advanced progress toward achieving the College's mission, and any necessary adjustments made to the Plan as part of the review process. As noted above, the College anticipates this report to the Board will take place during the first quarter of 2023.

In addition to an annual review of the Spending Plan, the College also provides a quarterly financial update as part of regular Board meetings, and will continue to present an annual budget for Board approval that is designed to achieve the College's mission.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

1-Year Agency Response

In November 2021, Calbright incorporated into its Implementation Plan (IP) an aligned Spending Plan, which provides greater accountability regarding the College's spending and ensures an effective use of public funds. This recommendation is fully implemented at this time, with the caveat that an annual review and report to our Board of Trustees will be completed each year going forward. To accomplish this, the College plans to provide a quarterly financial update as part of regular Board meetings, and to also continue presenting an annual budget for Board approval that is designed to achieve the College's mission.

The Spending Plan identifies estimated costs to accomplish the tasks set forth in the IP and a timeline through the end of FY2025 for when we expect to utilize these funds. These costs include both labor and related expenses (e.g., salaries and benefits for full-time staff and faculty) and operating costs.

The spending plan, like Calbright's board-approved Strategic Vision goals, is based on a conservative assumption of flat annual funding from the state at $15 million (not adjusted for inflation), which materially constrains future growth. This is a reduction from the $20 million annual funding initially provided, even though statutory and other expectations have not been adjusted accordingly. Any additions or reductions in actual annual funding levels through FY2025 will affect the pace and scope of activities identified in the IP.

Calbright will review this Spending Plan at least annually and make adjustments as needed to ensure budget compliance while also completing necessary start-up activities and achieving institutional milestones specified in Calbright's founding legislation. Data on actual spending and revised projections of specific tasks will be included in the years ahead. Additionally, as we review our IP at least every six months, any modifications or adjustments to the IP will be reflected in updates to the spending plan.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

As we stated in the last evaluation of this recommendation, we will assess Calbright's implementation of the annual review portions of this recommendation in November 2022.

6-Month Agency Response

To provide greater accountability regarding Calbright's spending and to ensure an effective use of public funds, the College has incorporated into its Implementation Plan (IP) an aligned spending plan that details how and when we expect to spend the funds allocated by the Legislature. The spending plan identifies estimated costs to accomplish the tasks set forth in the IP and a timeline over the next four fiscal years (FY2022-FY2025) for when we expect to utilize these funds. These costs include both labor and related expenses (e.g., salaries and benefits for full-time staff and faculty) and operating costs. The spending plan, like Calbright's board-approved Strategic Vision goals, is based on a conservative assumption of flat annual funding from the state at $15 million (not adjusted for inflation), which materially constrains future growth. This is a reduction from the $20 million annual funding initially provided, even though statutory and other expectations have not been adjusted accordingly. Any additions or reductions in actual annual funding levels will affect the pace and scope of activities identified in the IP.

To stay within budget while completing necessary start-up activities and achieving institutional milestones specified in Calbright's founding legislation, we will review our spending plan at least annually and make adjustments as needed to ensure fiscal efficiency and sustainability, incorporating data on actual spending and revised projections of specific tasks. Additionally, as we review our IP at least every six months, any modifications or adjustments to the IP will be reflected in updates to the spending plan. We will annually report to the Board of Trustees on our spending to date and explain how our spending has advanced progress in achieving Calbright's mission.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

The Implementation and Spending Plan that Calbright provided incorporates the estimated costs associated with each of the tasks included in the Plan, as well as a timeline for which it expects to complete those tasks.

Because this recommendation includes annual review of the spending plan and reporting to the Board of Governors, we will assess Calbright's full Implementation of this recommendation after Calbright's first opportunity to implement those elements of the recommendation, which will occur in November 2022.

60-Day Agency Response

Calbright is developing a spending plan that details how and when the college expects to spend allocated dollars. Subsequent to the final enactment of the 2021-2022 State Budget, the college is working with its Board of Governors (Calbright's Board of Trustees) to adopt its FY 2021-2022 budget consistent with the goals and objectives under the first year of the college's Strategic Vision, key legislative milestones, and other program and compliance requirements.

As noted in the response to Recommendation #6, the college has taken the next steps in the development of the 7-year implementation plan including a preliminary operational plan identifying key activities being undertaken and timeframes for advancing the goals set forth in the College's plan (Strategic Vision). This plan is based on the best available data in the field.

The development of the spending plan will track the development of the college's revised 7-year implementation plan also required by November 2021. The Chief Financial and Administrative Officer and Controller/Executive Director of Finance have begun a process to identify estimated costs associated with key activities under the operational plan. This process includes a budget review with an eye toward capital outlays and additional estimates of staffing to match the growth in students over the next several years. These activities will be essential to the success of our students, and the college will be working to ensure that there are sufficient resources to meet these needs while the college grows.

Longer-term cost estimates are anticipated to follow 2021-2022 budget and policy deliberations, which are expected to wrap up by August 2021, and will also inform longer-term discussions.

California State Auditor's Assessment of 60-Day Status: Pending

We will assess Calbright's spending plan when Calbright provides that plan in November 2021.

Recommendation #3 To: Calbright College

To ensure that its hiring process is fair and results in the hiring of well-qualified staff, Calbright should, by November 2021, finalize its development of human resources policies and procedures for recruitment and hiring that comply with state law and regulation.

6-Month Agency Response

To ensure that Calbright's hiring process is fair, continues to align with state law and regulations, and results in the hiring of well-qualified staff, the College has strengthened its human resources policies and procedures. In accordance with Title 5 of the California Code of Regulations, Section 53003, all Calbright employees participate in an Equal Employment Opportunity and hiring process training which includes an overview of the education code as well as state and federal nondiscrimination laws; the educational benefits of workforce diversity; the elimination of bias in hiring decisions; a detailed review of the Calbright recruitment and hiring process; and best practices in serving on Calbright's hiring screening committees. To foster inclusion and to promote open positions, we advertise positions on our website and the community colleges' registry as well as external sites such as LinkedIn, Idealist, Higher Ed Jobs, and job boards that focus on reaching diverse audiences and candidates.

In accordance with Education Code section 87101 (subd.b) and (c), Calbright's Board of Trustees approved the College's Equal Employment Opportunity (EEO) plan in September 2021. Additionally, Administrative Procedure (AP) 7120 Recruitment and Hiring, which outlines Calbright's recruitment and hiring procedures, has been approved through the College's AP governance structure. In addition to AP 7120, we have also prioritized and approved nine HR related APs that have implications for fair treatment and equity in the hiring process. Human Resources has developed standard operating procedures based on the approved Administrative Procedures and will publicly post to Calbright's Boarddocs, and will continue to monitor and evaluate policies to reflect future changes in laws, regulations, and best practices.

California State Auditor's Assessment of 6-Month Status: Fully Implemented

Calbright provided both its EEO plan and its formalized recruitment and hiring policy.

60-Day Agency Response

Calbright is in the process of further strengthening its hiring process and human resources policies and procedures in line with federal and state laws and regulations. In accordance with Title 5 of the California Code of Regulations, Section 53003, all Calbright employees participate in an Equal Employment Opportunity training which includes an overview of the education code as well as state and federal nondiscrimination laws; the educational benefits of workforce diversity; the elimination of bias in hiring decisions; a detailed review of the Calbright recruitment and hiring process, and best practices in serving on Calbright's screening committee. In accordance with Education Code section 87101 (subd. b) and (c) Calbright College has developed an Equal Employment Opportunity (EEO) Plan which is going through legal review and will be presented to the Board of Trustees in October 2021. Draft Administrative Procedure (AP) 7120 Recruitment and Hiring outlines Calbright's recruitment and hiring procedure and is currently under legal review and will be ready for final approval in October 2021. Beyond AP 7120, we are identifying a list of priority HR related AP's that have implications for recruitment and hiring. Calbright recently revised our AP process to streamline, centralize, and strengthen our approach to ensure timely and thorough completion of the colleges AP's. We expect to finalize these AP's on time.

Currently, all screening committees consist of a committee chair, EEO representative, subject matter experts, and department members who review all applications to ensure applicants meet the minimum job qualifications (or its equivalent). The screening committees then interview the most qualified to make hiring recommendations. Guidelines to ensure a fair and equitable process are reviewed each time a committee convenes to start the screening and interview process.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #4 To: Calbright College

To ensure that it uses state resources responsibly, Calbright should do the following:

1. Immediately commence the process of hiring a procurement director with significant experience in public sector procurement.

2. By July 2021, complete the development of a fully functioning procurement process that aligns with state law, regulations, and to the extent practicable, the State Contracting Manual. That process should include policies and procedures that ensure that all of Calbright's contracts provide clear expectations of the work that contractors will perform. It should further include strong contract management processes for ensuring that contractors perform that work satisfactorily before receiving payment.

3. By August 2021, provide training on relevant laws, policies, and procedures related to procurement to all staff involved in approving or managing contracts.

60-Day Agency Response

Calbright has taken all the necessary steps to fully implement this recommendation. Calbright immediately commenced the process to hire a procurement coordinator with significant experience and expertise in public sector procurement. The procurement coordinator position was authorized at the Board of Trustees Executive Committee meeting on April 21, 2021. The position description was then subsequently posted on the Calbright website and several job boards. Due to insufficient applications, the position description was then reposted in June 2021. We continue to monitor and actively promote the position description and are exploring the possibility of leveraging a loaned staff from the California Community College system or someone with significant experience in public sector procurement to support this work in the interim.

To implement the recommendation to develop a fully functioning procurement process that aligns with state law, regulations, and to the extent applicable, the State Contracting Manual, Calbright developed a "Procurement, Purchasing, and Contracting Handbook." We obtained best practices from other California Community Colleges and the Foundation for California Community Colleges, including staff involved in procurement processes; identified best practices from the State Contracting Manual regarding contracting and procurement practices; and leveraged subject matter experts and consulted with counsel in the development of the Handbook with policies and procedures that align with state law, regulations, and best practices. Building on the Handbook, Calbright has scheduled training for all employees on July 26, 2021, which will be conducted in conjunction with legal counsel specialized in District laws, policies, and procedures related to procurement and contracting. Going forward, all hires will be trained as part of an interactive onboarding process on relevant laws and Calbright's procurement policies and procedures.

California State Auditor's Assessment of 60-Day Status: Fully Implemented

Calbright provided its procurement handbook, which includes policies for establishing expectations of the work that contractors will perform and contract management processes. Calbright also provided evidence that many of its managerial and executive staff attended a training on procurement laws, policies, and procedures. Finally, Calbright provided evidence that it has posted a position for a procurement coordinator.

Recommendation #5 To: Calbright College

To ensure that the compensation it provides its employees is reasonable, Calbright should establish a pay schedule for all employees by November 2021 that does the following:

1. Includes salary, benefits, and all other forms of compensation.

2. Establishes compensation packages that are comparable to those for similar positions within the community college system.

Annual Follow-Up Agency Response From October 2023

Part of the language in this recommendation indicates that Calbright must "establish a pay schedule for all employees." As past responses to this recommendation note, this was completed in October of 2021. Since the initial California State Auditor report in the spring of 2021, the auditor's office indicated that to fully implement this recommendation, all pay schedules must be implemented, as opposed to "established."

To further comply with this additional requirement, Calbright had to first complete collective bargaining with its labor units. At the College's May 24, 2022 Board of Trustees meeting, a Collective Bargaining Agreement with the Calbright Faculty Association (the College's local chapter of the California Teachers Association) was ratified and adopted. This agreement included salary schedules and compensation. At the College's June 15, 2022 Board of Trustees meeting, salary schedules for Confidential, Management/Administrators/Academic Administrators were adopted by the Board. At the College's March 21, 2023 Board of Trustees meeting, a Collective Bargaining Agreement with the California School Employees Association and Calbright Classified staff was ratified and adopted. This agreement included salary schedules and compensation. At the College's June 21, 2023, Board of Trustees meeting a salary schedule was adopted for Adjunct Faculty.

Calbright has implemented this recommendation and offers competitive salary and benefit schedules for all employee groups. Ongoing salary reviews will occur on a regular basis in order to maintain competitive salaries and benefits for Calbright employees. Salary schedules are posted on the College's website in the careers section.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Calbright provided its salary schedules for its staff including faculty, management, and classified staff.

Annual Follow-Up Agency Response From November 2022

The language in this recommendation indicates that Calbright must "establish a pay schedule for all employees." This was completed in October of 2021. To ensure employee compensation is competitive and comparable to those for similar positions within the California Community Colleges (CCC) system, Calbright commissioned Stan McKnight & Associates, Inc. (a firm specializing in higher education salary analysis) to conduct research and analysis of peer CCCs and Districts to benchmark Calbright's salaries. McKnight's analysis also took into account our unique characteristics and needs with respect to certain positions or job titles. On 10/20/21, McKnight presented their findings, including recommended pay schedules and salary ranges (executive, academic administrators/administrators/management, and confidential) for our staff positions to Calbright's Board of Trustees.

At the College's June 15, 2022 Board of Trustees meeting the Confidential, Management/Administrators/Academic Administrators salary schedules were adopted and salary schedules are currently implemented for both academic and non-classified staff. Academic employees are represented by the California Teachers Association (CTA) and the College's inaugural agreement with CTA was ratified in May 2022.

Since the initial California State Auditor report in the spring of 2021, the auditor's office indicated that to fully implement this recommendation, all pay schedules must be implemented, as opposed to "established."

While the salary schedule for classified staff is established, Calbright's classified employees are represented by the California School Employees Association (CSEA) and the collective bargaining process with CSEA is currently underway. The College anticipates a final agreement to be reached during the first quarter of 2023. Once the agreement is ratified, salary schedules for all Calbright faculty and staff will have been implemented.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

1-Year Agency Response

To ensure employee compensation is competitive and comparable to those for similar positions within the California Community Colleges (CCC) system, Calbright commissioned Stan McKnight & Associates, Inc. (a firm specializing in higher education salary analysis) to conduct research and analysis of peer CCCs and Districts to benchmark Calbright's salaries. McKnight's analysis also took into account our unique characteristics and needs with respect to certain positions or job titles. On 10/20/21, McKnight presented their findings, including recommended pay schedules and salary ranges (executive, academic administrators/administrators/management, and confidential), for our staff positions to Calbright's Board of Trustees. The analysis for benchmarking also addressed base salary (prior to longevity), establishing fair and consistent hiring ranges, and pay percentage increase by step within salary ranges.

Calbright academic employees are represented by the California Teachers Association (CTA) and classified employees by the California School Employees Association (CSEA). Salary schedules will be implemented in compliance with collective bargaining obligations. The College anticipates presenting its first ratified contract with CTA to its Board of Trustees in May 2022, and to begin collective bargaining with CSEA in the summer of 2022. Pay schedules for both CTA and CSEA are established, with the CSEA schedule currently moving through the formal bargaining process.

Calbright has established a compensation package that, where applicable, is comparable to those for similar positions within the CCC system, and includes: medical, dental, vision, California state pension plans, and A&D and life insurance. We offer a variety of voluntary benefits common in the CCC system, such as 403b/457 retirement plans, supplemental insurance plans, and flexible and healthcare spending accounts.

California State Auditor's Assessment of 1-Year Status: Pending

We recognize the requirements Calbright faces with respect to collective bargaining mean that it cannot unilaterally address issues of compensation. Accordingly, while Calbright has taken important first steps, we will assess this recommendation as Pending implementation until Calbright completes its negotiation process and finalizes its salary schedules.

6-Month Agency Response

To ensure employee compensation is competitive and comparable to those for similar positions within the California Community Colleges (CCC) system, Calbright commissioned McKnight & Associates, Inc. (a firm specializing in higher education salary analysis) to conduct research and analysis of peer CCCs and Districts to benchmark Calbright's salaries. McKnight's data analysis also took into account our unique characteristics and needs with respect to certain positions or job titles. On October 20, 2021, McKnight presented their findings—including recommended pay schedules and salary ranges (executive, academic administrators/administrators/management, and confidential)—for our staff positions to Calbright's Board of Trustees. The analysis for benchmarking also addressed base salary (prior to longevity), establishing fair and consistent hiring ranges, and pay percentage increase by step within salary ranges. Calbright cannot unilaterally create a salary schedule for represented employees outside the negotiations process. We are currently negotiating with the California Teachers Association for its academic employees and are also in the process of unit recognition of our classified employees with the California School Employees Association. Salary schedules will be implemented in compliance with collective bargaining obligations.

Calbright has established a compensation package that, where applicable, is comparable to those for similar positions within the CCC system. The College contributes to a comprehensive package that is standard throughout the CCC system and consists of core employee benefits including: medical, dental, vision, California state pension plans, and A&D and life insurance. Additionally, we offer a variety of voluntary benefits common in the public sector, such as retirement plans (403b/457), insurance plans, and flexible and healthcare spending accounts. Calbright is planning to offer a comprehensive Employee Assistance Plan to employees in 2022.

California State Auditor's Assessment of 6-Month Status: Pending

Calbright provided evidence of the work that McKnight & Associates has performed toward the development of pay schedules and noted that it is engaged in the collective bargaining process for its employees. We recognize the requirements Calbright faces with respect to collective bargaining mean that it cannot unilaterally address issues of compensation. Accordingly, while Calbright has taken important first steps, we will assess this recommendation as Pending implementation until Calbright completes its negotiation process and finalizes its salary schedules.

60-Day Agency Response

Calbright Board Policy 7130 (Compensation) requires that salary schedules, compensation, and benefits, including health and welfare benefits, for all classes of employees be established by the Board of Trustees.

To develop Calbright's salary schedules for faculty, classified staff, and administrators, the college has retained the services of McKnight Associates, Inc. (a consulting firm specializing in HR for colleges, universities, and other organizations) in May, 2020. McKnight Associates reviewed 20 peer institutions within the California Community College system for the purposes of benchmarking Calbright's salaries to those identified institutions. McKnight Associates used the data from Calbright's peer institutions, while also taking into account Calbright's unique characteristics and needs, to present data analysis on salary benchmarks for Calbright employees.

Calbright is establishing a process that will ensure our compensation structures and benefits programs provide all employees a reasonable and competitive pay schedule, health and supplemental benefits, and retirement plans. The adoption of salary schedules will be implemented in compliance with applicable collective bargaining obligations.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #6 To: Calbright College

To adequately address its foundational purpose for existing, Calbright should immediately develop a robust implementation plan that aligns with best practices and translates its mission into actionable goals and strategies. It should complete that plan and begin implementing it by November 2021. At a minimum, Calbright should include in its implementation plan all of the following:

1. Its goals, which should include both its goals for completing the setup of the college and its student outcome goals. It should develop its student outcome goals based, at a minimum, on a comparison of the student outcomes for multiple other reasonably comparable educational programs.

2. The major steps necessary to achieve its goals.

3. The estimated resources and specific deliverables that each step will require.

4. The due dates and assigned staff for each deliverable or major step.

5. The criteria it will use for measuring its success and monitoring its progress.

6. A strategy and timeline for ending its reliance on the Foundation.

After completing its implementation plan, Calbright should review the plan at least every six months, and revise and update it as needed to account for major changes relevant to the college's implementation. By July 2022, Calbright should demonstrate that it has made consistent progress in implementing its plan.

To improve its accountability for its actions toward fulfilling its mission, Calbright should annually report to the Legislature its progress related to each step in its implementation plan.

Annual Follow-Up Agency Response From October 2023

Calbright has created an Implementation Plan (IP) that the auditor's team has approved. This internal and deliberative planning document requires review and updates in six-month cycles. The first review was completed in the first calendar quarter of 2023 and a progress update was provided to Calbright's Board, in a public meeting, in the spring of 2023.

In addition, public updates to the College's Board are completed regularly and include both topic-specific presentations—such as ratification of a bargaining agreement or re-launch of the website—as well as more general updates, like holistic progress made in a prior calendar year or fiscal year. These are complemented by Calbright's engagement with Legislative offices and annual milestone report, which is published each summer and shared directly with the Legislature, Board, and placed on the College's website.

The Milestone Report provides an update on the College's progress with regard to the requirements outlined in Calbright's founding legislation, as well as priorities included in the IP. The high-level goals and deliverables articulated in the IP align with requirements in Calbright's founding legislation - the report includes discussion of enrollment and completion, student body demographics, academic program development, accreditation, labor market outcomes for graduates, and student survey data, as well as initiatives and improvements in areas like student success, staffing, and technology infrastructure. The Calbright team has also met with every Legislator's office to provide updates about the College directly, and will continue to do so. Calbright's president and CEO also participates in budget hearings in the spring with appropriate Senate and Assembly committees, which includes direct engagement from the College's leadership team with staff from the committees as well as elected officials' offices in order to answer questions they have regarding Calbright.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

We have previously determined that Calbright implemented the first part of our recommendation to develop an implementation plan containing all of the elements listed in the recommendation. For this update, Calbright provided documentation of an internal procedure for reviewing and updating the implementation plan, and documentation of its November 2022 update to that plan. It also provided its most recent milestone documents and a separate report to the Board of Governors to demonstrate that it has provided updates to the Board on its progress on the implementation plan. However, the documents provided do not include a complete update on its progress on the implementation plan. Although an update to the Board lists key tasks that Calbright had completed, it did not, for example, report on tasks that were in progress or on the horizon, or the degree to which Calbright had many any adjustments to expected completion timelines. Additionally, Calbright did not provide documentation demonstrating that it had reported on its progress on the implementation plan to the Legislature as we had recommended.

Annual Follow-Up Agency Response From November 2022

As reported in November 2021, Calbright has fully implemented all elements of Recommendation 6 with the exception of an annual review of the Implementation Plan. This final element requires a full year to pass since the Implementation Plan was created and the California State Auditor's office has indicated that this recommendation will be considered fully implemented when the first annual review of the Implementation Plan and report to Calbright's Board of Trustees is complete. Calbright anticipates all tasks will be finished during the first quarter of 2023.

The first annual review of the Implementation Plan will begin in November 2022. Following an internal review, the College will provide a report to the Board of Trustees about how the Plan has advanced progress toward achieving the College's mission, and any necessary adjustments made to the Plan as part of the review process.

Calbright has also written and shared a memo to the Foundation for California Community Colleges (FCCC) that details a strategy and timeline for how and when the College plans to end reliance on FCCC's support. As of October 2021, FCCC no longer supports Calbright in six of the eight workstreams in which it previously assisted, and by March 2024, any services requested by Calbright and provided by FCCC will likely be of a limited and technical nature.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

1-Year Agency Response

As reported in November 2021, Calbright has fully implemented the first six items under Recommendation 6, and will demonstrate that it has made consistent progress in implementing its plan by July 2022.

Calbright College has developed and already begun implementing a robust Implementation Plan (IP) covering January 2021 through December 2025, the remaining five years of our statutory seven-year startup period. Through the first quarter of 2022, the College completed each deliverable, including recognition of the establishment of a student government, creating a plan that integrates career preparation and development into every part of the student journey, and implementing a staff performance review process.

The IP aligns with best practices and translates our mission into actionable goals and strategies. It includes institution-level goals for our operational setup and student outcomes; activities, which are the major steps required to meet those goals, grouped by focus area; and assigned executive leads, estimated resources, and key deliverables with target completion dates for each focus area. The deliverables and completion dates are the mechanisms we are using to monitor progress and measure success. The IP is also aligned with our spending plan (Rec #2). Within each focus area, periodic reviews and revisions are built into specific activities and deliverables, and the IP will receive holistic review every six months.

Calbright has also written and shared a memo to the Foundation for California Community Colleges (FCCC) that details a strategy and timeline for how and when we plan to end reliance on FCCC's support. As of October 2021, FCCC no longer supports Calbright in six of the eight workstreams in which it previously assisted, and by March 2024, any services requested by Calbright and provided by FCCC will likely be of a limited and technical nature.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

We will assess Calbright's periodic review and revision of the plan, consistent progress in implementing the plan, and annual report to the Legislature - when its update on those steps becomes due later this year.

6-Month Agency Response

Calbright has fully implemented the first six items under Recommendation 6 and will demonstrate that it has made consistent progress in implementing its plan by July 2022

Calbright College has developed and already begun implementing a robust Implementation Plan (IP) covering January 2021 through December 2025, the remaining five years of our statutory seven-year startup period. The IP aligns with best practices and translates our mission into actionable goals and strategies. It includes institution-level goals for our operational setup and student outcomes; activities, which are the major steps required to meet those goals, grouped by focus area; and assigned executive leads, estimated resources, and key deliverables with target completion dates for each focus area. The deliverables and completion dates are the mechanisms we are using to monitor progress and measure success. The IP is also aligned with our spending plan (Rec. #2). Within each focus area, periodic reviews and revisions are built into specific activities and deliverables, and the IP will receive holistic review every six months.

Calbright has also written and shared a memo to the Foundation for California Community Colleges (FCCC) that details a strategy and timeline for how and when we plan to end reliance on FCCC's support. As of October 2021, FCCC no longer supports Calbright in six of the eight workstreams in which it previously assisted, including: Budgeting & Business Processes; Legal, Contracts, Risk Management and Governance; Purchasing; Information Technology; External Relations, Marketing & Communications Design; and Office Space. Our current engagement with FCCC is compliant with requirements that apply to the CCC system and consistent with best practices.

By March 2024, any services requested by Calbright and provided by FCCC will likely be of a limited and technical nature, as we near fully independent financial, technical, and administrative units.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

Calbright provided its Implementation and Spending Plan, which identifies its goals for student outcomes and the setup of the college. The Plan also includes the minimum items that we identified in the recommendation, including the major steps necessary to achieve its goals, due dates for accomplishing those steps, and specific deliverables. Calbright also provided its plan for ending its reliance on the FCC. Calbright's implementation of these plans should improve its ability to achieve its stated goals.

We will assess Calbright's implementation of the remainder of this recommendation - including its periodic review and revision of the plan, consistent progress in implementing the plan, and annual report to the Legislature - when its update on those steps becomes due in July 2022.

60-Day Agency Response

Calbright has completed a 3-year Strategic Vision for Calbright's future, including ambitious and achievable goals relating to program pathway development and scaling, and organizational maturity at scale. This Strategic Vision has been approved by the Calbright Board of Trustees and includes goals for completing the setup of the college and its student outcome goals. We are in the process of further developing this vision into a robust implementation plan with more specific detail. As envisioned by the Legislature, Calbright contracted with the Foundation for California Community Colleges during the seven-year start-up window. We have taken a number of steps towards setting up key organizational administrative functions and will develop a plan for reducing contractual administrative services during this period. Calbright has scheduled a series of meetings with the Foundation for California Community Colleges leadership over the next several months to formalize this plan.

California State Auditor's Assessment of 60-Day Status: Pending

We look forward to reviewing Calbright's implementation plan once it is complete. However, it is critical that Calbright build its implementation plan around goals that will help it to achieve its mission. As we stated in the audit report, although Calbright published a strategic vision containing goals for student enrollment, program completion, and labor market outcomes, it did not adequately support its goals by benchmarking them against multiple colleges or other points of comparison. Until Calbright demonstrates that it has established reasonable goals for completing the setup of the college and for student outcomes, Calbright cannot demonstrate that any implementation plan that it develops to achieve those goals is adequate to assisting it in achieving its mission.

Recommendation #7 To: Calbright College

To effectively reach and enroll the students the Legislature intended it to serve, Calbright should, by November 2021, do the following:

1. Develop and implement a specific plan for conducting outreach to individuals within its target student population. The plan should reflect its current outreach strategies and long-term goals, including strategies for reaching each group within its target population.

2. Establish methods for measuring whether it has successfully enrolled its target student population. These methods should include collecting and reviewing the information necessary to ensure that it is reaching its target student population, including data on student income level, veteran status, employment status, incarceration history, and reasons for enrolling in Calbright instead of a traditional community college.

By July 2022, Calbright should demonstrate that its efforts have been effective at reaching the population the Legislature intended it to serve.

Annual Follow-Up Agency Response From November 2022

Calbright's student body reflects the College's mission and legislative mandate and we continue to execute an outreach plan designed to engage with our target student population. As such, the College considers this recommendation fully implemented.

Calbright's founding legislation identifies our focus population as "working adults." In May 2021, the auditor's office further clarified our focus populations to include, among others: Californians who 25+ years old with no college degree, individuals who are unable to access traditional forms of education and job training, parents and caregivers, those who are unemployed, working adults, members of BIPOC communities, and Californians who are underrepresented in higher education and the workforce.

Calbright's outreach advances the dual charge of enrolling a statewide student body while also prioritizing key regions like the Central Valley, Inland Empire, and rural counties. An enrollment report that provides more context about our outreach endeavors, as well as details about our student body will be shared with the auditor's office. The following are some of the key statistics the College monitors when considering outreach effectiveness:

As of 10/31/22, enrollment stands at 1,422, a 194% increase from July 2021. As of 8/30/22, the time period for which most recent data is available, 92% of Calbright students are 25+ years old, more than double the CCC rate. 34% are parents/caregivers, more than triple the CCC rate, and 38% are unemployed. 54% of students are either unemployed or working part time and another 8.4% work more than 40 hours per week. Additionally, 88% of students cite the College's model as the reason their program was accessible. 33% of students identify as Latinx, 22% as Black, 13% as Asian, and 5% as Native American. In addition, enrollment spans the state and represents 44 of California's 58 counties, including 25 rural counties.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Calbright has established an outreach plan and demonstrated that it now has processes in place to collect and monitor data on whether it is reaching and enrolling the students in its target student population. It provided its periodic analysis of its enrollment, which shows that it has enrolled a diverse student body that is generally reflective of its target population, including students who are caregivers, who are unemployed, or who work part time.

1-Year Agency Response

Calbright has fully implemented Recommendation 7 and continues to enroll a student body that reflects the populations the Legislature intended it to serve. This includes: Californians who are at least 25 years old with no college degree, individuals who are unable to access traditional forms of education and job training, parents and caregivers, those who are unemployed, working adults, members of BIPOC communities, and Californians who are underrepresented in higher education and the workforce, to name only a few.

Calbright has developed and implemented an outreach plan, aligned with our 2021-2023 Strategic Vision, for conducting outreach to our target student populations. Since July 2021, Calbright's student body has increased by more than 110%, surpassing 1,000 students and coming from 40 counties across the state, including 22 rural counties. 92% are at least 25 years old, more than double the systemwide rate. 24% identify as Black, 33% as Latino, 19% as Asian, and 4% as Native American. In response to questions added to the supplemental section of the CCC system's application in October 2021, additional information about our students is also available: 90% of Calbright's students cite our unique model as the reason their program was accessible. 34% are parents/caregivers, which is triple the systemwide rate, 40% are currently unemployed, 18% work part-time, 8.4% work more than 40 hours per week, and 31% are displaced workers who recently lost their job or had a reduction in hours. The percentage of student immigrants, military veterans, and formerly incarcerated are 15.2%, 4.8%, and 2.7% respectively.

While continued monitoring, data review, and expansion of partnerships is necessary as Calbright continues to progress through its start-up period, these figures indicate that the College is meeting its mission with regard to enrolling its focus student population.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

Although Calbright has a documented outreach plan, it has not yet demonstrated improved success at reaching its target population.

6-Month Agency Response

Calbright has fully implemented the first two items under Recommendation 7, and will demonstrate that its efforts have been effective at reaching the population the Legislature intended it to serve by the stated deadline of July 2022.

Calbright has developed and implemented an outreach plan, aligned with our 2021-2023 Strategic Vision, for conducting outreach to our target student population. This plan includes both current and long-term goals and strategies for implementing, evaluating, and refining a multi-faceted approach consisting of digital/terrestrial ads and campaigns, in-person events, advancing current partnerships and developing new ones, media engagement, and expanding internal capacity. We will regularly update and revise strategies and goals as we continue to measure progress, determine needs, and refine planning. This plan is dynamic and will, at minimum, receive holistic review every six months and be revised as necessary.

Measuring Calbright's effectiveness at enrolling its focus student population is essential to the College's ability to deliver on our mission. We are constantly monitoring enrollment goals on our backend platform, and in October 2021 added two questions to the supplemental questions section of the CCC system's application (CCCApply). These questions join previously added questions regarding employment status, and will provide additional data to measure the effectiveness of the College's outreach strategies for reaching its target student population. The first new question asks applicants if they identify as an immigrant, individual who was formerly incarcerated, parent or caregiver, military veteran, and/or if they have recently experienced job loss or a reduction in hours/wages. The second asks why applicants are interested in Calbright.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

Calbright provided its new outreach plan, which describes recent outreach efforts and its plans for the coming year regarding how it will conduct outreach, including to students within its target student population. Calbright also provided evidence that it has added questions to its application to collect information critical to understanding whether applicants are members of its target population, such as whether the applicant is a parent or caregiver. Calbright still does not ask students about their income, which Calbright has stated it intends to collect from EDD data. Finally, the plan describes how Calbright will use the various data it now collects to evaluate its success at reaching students within its target population.

We will assess Calbright's ability to demonstrate that it has effectively enrolled students from its target population after it reports on its progress in July 2022.

60-Day Agency Response

Calbright has developed a draft outreach plan, aligned with our 2021-2023 Strategic Vision, that lays out a structure for expanding internal capacity and implementing, evaluating, and refining a multi-faceted outreach approach consisting of digital ads, partnerships, in-person events, and earned and paid media to reach prospective students in Calbright's focus population. Throughout the Strategic Vision period, we will refine our goals and evaluation periods to measure progress, determine needs, and refine planning. The draft outreach plan includes a detailed timeline of the steps to build awareness, expand strategic partnerships, and reach our target population over the Strategic Vision period.

Measuring Calbright's effectiveness in enrolling its targeted student population is essential to the college's ability to deliver on its mission. Calbright is constantly monitoring our enrollment goals on our backend platform, as well as administering a bi-annual student survey that measures factors such as: employment status, job history and future job expectations on role and salary. To reach our targeted student populations, Calbright is currently partnering with eight local Workforce Boards in targeted areas of California: SELACO (Southeast LA County), NOVA (Santa Clara and San Mateo counties), Fresno, San Bernardino, Merced, San Diego, and the Pacific Gateway (Long Beach). Additionally, we have worked and will continue to work, with organizations representing incumbent workers that would fall within our student population.

As we continue to pilot and evaluate our initial hypotheses regarding our student population, we aim to solidify our benchmarks by which we measure our effectiveness in enrolling our target population for each particular pathway.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #8 To: Calbright College

To ensure that it adequately prepares its target student population to obtain positive employment outcomes after graduation, Calbright should, by November 2021, develop and implement a process for selecting and expanding educational programs that will provide value to that population; that process should include the following:

1. Collaboration with employers and industry groups to inform the content of the programs.

2. Consideration of market demand for graduates of such programs.

3. Determination of whether the programs can help its target student population obtain positive employment outcomes including jobs, earning gains, and upward mobility.

4. Available resources for program implementation.

5. An evaluation of student demand for the programs.

Annual Follow-Up Agency Response From October 2023

As noted in previous updates, Calbright has enhanced its human-centered design practice to ensure program selection strategy is directly influenced by adult learners, their unique circumstances, needs, and lived experiences. The auditor's office has shared that it is satisfied with the information provided in past updates and has made an additional request for Calbright to provide its Standard Operating Procedure (SOP) that outlines the College's process for identifying, assessing, and selecting potential programs. The College had previously (in November 2022) provided a draft version of this SOP, and then received further guidance noting that a final version was necessary in order to complete this recommendation. A final version of the SOP, which includes steps the College takes to review available resources for the program (such as technology infrastructure, staffing, and licensing costs), was provided to the auditor's office in October 2023. This SOP will continue to be updated throughout the College's start-up period in order to reflect evolution in Calbright's development and operations.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Calbright provided its updated procedures for selecting educational programs, which now includes that Calbright will identify the costs associated with the educational program. It further includes as criteria for proceeding with a new program that Calbright has sufficient financial resources available for the program.

Annual Follow-Up Agency Response From November 2022

Calbright has enhanced its human-centered design practice to ensure program selection strategy is directly influenced by adult learners, their unique circumstances, needs, and lived experiences. The auditor's office has shared that it is satisfied with the information provided in past updates and has made an additional request for Calbright to provide its Standard Operating Procedure (SOP) that outlines the College's process for identifying, assessing, and selecting potential programs. The auditor's office has indicated that this recommendation will be fully implemented after the SOP is provided. The SOP was shared with the auditor's office on November 7, 2022.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Calbright provided its draft procedures for selecting educational programs. Those procedures include performing market research, identifying employer partnerships, and considering the benefits that a program will provide to Calbright's target student population. However, the procedures are still in draft form, and they do not establish a process for verifying that sufficient financial resources are available to establish the program. We will consider this recommendation to be fully implemented when Calbright finalizes a set of procedures that address all aspects of our recommendation, including verifying that resources are available to launch a program.

1-Year Agency Response

Calbright has enhanced its human-centered design practice to ensure program selection strategy is directly influenced by adult learners, their unique circumstances, needs, and lived experiences. The College's evolution in this area has led to the development of a workforce-driven equity-centered design approach for program selection which acknowledges that equity requires intent and focus.

The College's decision making leverages labor market insights to assess traditional supply and demand metrics as well as in-demand occupational skills, including workforce analysis to gauge hiring probability of our target population. We utilize data sources such as EMSI to inform program selection, including information on total job postings in California for roles related to specific programs; regional and state-wide demand for these roles; expected job growth within California over the next decade; and whether the median and range of salaries for those positions will foster economic mobility. In addition, we maintain an Advisory Council for each program area composed of external members who provide expert feedback and collaborate to help identify in-demand jobs and the technical and soft skills they require. Research on job postings' educational attainment requirements, as well as race equity within incumbent workforce by occupation, further validate that the College's programs will enhance both job prospects and economic mobility of our target population.

We are creating market demand by responding to critical gaps in the workforce that have historically been the responsibility of the employer to address. Our recent partnership with SEIU-UHW to develop a Licensed Vocational Nurse training program is one such example of this new approach. We are working in tandem with the labor organization to address a need felt acutely in every region of the state, expand the training pipeline, and diversify the frontline healthcare provider workforce.

California State Auditor's Assessment of 1-Year Status: Pending

Calbright provided documentation containing some information on factors it considers when researching new educational programs. However, it has not yet developed a clear, specific, and documented process describing the criteria that it will use to select new programs or the procedures it will follow.

6-Month Agency Response

Calbright has developed and implemented a process for selecting and expanding educational/career pathways of value. This process begins with determining labor/workforce needs based on employment supply and demand, as well as available Calbright resources to expand programs, and includes quantitative validation as well as engagement with employers, advisory councils, regional workforce development boards, and worker associations to understand urgent needs and anticipate future opportunities.

Calbright maintains an Advisory Council for each program area composed of external members who, through regular meetings, provide expert feedback and collaborate to help identify in-demand jobs and the technical and soft skills they require. To date Calbright has engaged with industry leaders, employers, and advocacy groups including from the technology and healthcare sectors to both inform and validate our program pathways and career supports. To drive stronger outcomes for our students, we have also built partnerships with regional organizations and workforce development boards to align efforts to support the college's focus population.

The College utilizes data sources such as Burning Glass Labor Insights and Emsi to inform program selection, including information on total job postings in California for roles related to specific programs; regional and state-wide demand for these roles; expected job growth within California over the next decade; and whether the median and range of salaries for those positions will foster economic mobility. The College also conducts qualitative research interviews with individuals currently employed in these roles to identify pain points and key skills for their positions, as well as their career goals and aspirations.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

Calbright provided a document that it stated describes its process for selecting and expanding programs. The document provides a very high level overview of its process for selecting and launching new programs. It describes in greater detail Calbright's efforts to conduct market research and collaborate with labor market representatives. However, it provides little detail on how it will evaluate the appropriateness of potential programs for its target student population, whether the programs will provide positive employment outcomes for those students, and whether Calbright possesses sufficient resources to launch, maintain, and expand the program.

60-Day Agency Response

Calbright's process for selecting and expanding educational/career pathways starts with determining need based on employment supply and demand, and includes engagement with employers, Advisory Councils, Workforce Boards and worker associations to understand urgent needs and anticipate future opportunities. Calbright maintains an Advisory Council for each program area composed of external members who provide expert feedback. To date Calbright has engaged with a variety of industry and employers from technology and healthcare to both inform and validate our program pathways and career supports. Designing effective programs also requires us to understand employees, their environments and what makes them successful. To drive stronger outcomes for our students we have also built partnerships with regional organizations and workforce boards to align efforts to support the college's focus population.

We also conduct Labor market research to: align academic programs to support employment in high-demand areas; determine total job postings in California for relevant job roles; understand statewide and regional demand; identify top defining and distinguishing skills for job roles (to align with our curriculum); analyze the ratio of job postings to eligible graduates who could fill them; and assess salary data leading to improved economic mobility.

Calbright's process also includes determining need based on areas where our targeted student population does not have access to programming, yet there is a high demand for such a program. We also conduct qualitative interviews to glean first-hand insights into the needs of those currently in the workforce.

We have begun to document this process and are on schedule for full implementation by November 2021. Reflected in the attached documentation are specific examples for how the data and analysis occurs for a given program prior to Calbright Board of Trustees approval.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #9 To: Calbright College

To ensure that it is fulfilling its mission to help students obtain positive employment outcomes, including jobs in their field of study, earning gains, and upward mobility, Calbright should do the following:

1. By November 2021, develop and implement a specific plan that describes how it will assist its students in acquiring jobs, earning more income, or being upwardly mobile after graduation; the plan should include a path toward securing job placements for its students.

2. By the same date, also establish a method to collect and review data on student use of its career services, employment outcomes following graduation, and employer satisfaction with Calbright's preparation of its students.

3. By July 2022, Calbright should begin demonstrating that it has been successful at assisting its graduates in obtaining positive employment outcomes, including jobs in their field of study, earning gains, and upward mobility.

Annual Follow-Up Agency Response From October 2023

As agreed upon by the auditor's team in the spring of 2022, Calbright has fully implemented the first two elements of Recommendation 9.

To expand on the College's work supporting positive labor market outcomes for students, Calbright has developed an employer engagement plan and expanded its career services team. In the fall of 2023, the College announced the launch of a data analyst apprenticeship training pipeline in partnership with Bakersfield College. Complementing initiatives underway with workforce boards and labor unions, Calbright is also working with NextGen Policy and Los Angeles County to establish a public sector technology apprenticeship program.

To track labor market outcomes for graduates, Calbright has been added to LaunchBoard, a statewide CCC Chancellor's Office data system. However, meaningful employment data lags and may not be available for several years while information is populated. In addition, as is standard practice in higher education, Calbright conducts alumni surveys to measure labor market outcomes. The current version of the survey launched in the spring of 2022 and reflects a 28% response rate (55 responses) of completers who finished between 3/22 and 7/23 (197 certificates issued during this period). 89% of students who completed the survey answered the following question: "How long did it take you to see a positive impact of your coursework on your employment, after completing your program?" Of the students who responded, 61.2% indicated a positive impact within 12 months of program completion. Students, through the alumni survey, may also provide their income prior to enrollment and after graduation. Of those who did, nearly 48% either moved from unemployed to employed or earned a raise, and nearly 48% kept their current role. This is a rough comparison to the CCC system-wide figure that shows roughly 28% of adult learner graduates move from unemployed to employed after completing their credential.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

As previously noted, Calbright submitted a plan for providing career services to its students. It also tracks usage of those career services and includes questions about students' use of those services in its career survey. For this update, Calbright provided data on responses to its alumni survey. We shared with Calbright some concerns regarding the specificity of the information that it solicits from its graduates regarding the degree to which the certificate they earned from Calbright has helped them achieve positive employment outcomes, and would suggest that it ask more specific questions about what positive outcomes those graduates enjoyed. However, Calbright's survey data does show that some students reported experiencing a positive employment outcome as a result of Calbright's program.

Annual Follow-Up Agency Response From November 2022

As agreed upon by the auditor's team in the spring of 2022, Calbright has fully implemented the first two elements of Recommendation 9: developing and implementing a specific plan that describes how it will assist students with positive labor market outcomes and establishing a method to collect and review data on student use of support services.

To expand on the College's work supporting positive labor market outcomes for students, Calbright has developed an employer engagement plan which complements the strong partnerships we have established with regional organizations, workforce associations and boards, community based organizations, and government agencies. Through the third quarter of 2022, the plan highlights our focus on statewide outreach and engagement with cross-industry employers to increase visibility of Calbright programming and position our adult learners for optimal labor market outcomes.

To measure students' employment outcomes after graduation, Calbright currently conducts alumni surveys and has been added to LaunchBoard, a statewide data system supported by the CCC Chancellor's Office. However, employment data is a lagging metric and may not be available right away, due to the manner in which LaunchBoard collects it. Calbright is also a new college and does not yet have a large alumni base, so it will take time to accumulate alumni survey data - the College will focus on this throughout the 2022-2023 fiscal year. There are strong anecdotal data of positive labor market outcomes, many of which are noted on the College's "Our Students" webpage, and continued measurement will be a focus as completion numbers accelerate during the 2022-2023 fiscal year.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

1-Year Agency Response

Calbright has fully implemented the first two items under Recommendation 9, and will complete the third by the stated deadline, July 2022.

To expand on the College's work supporting positive labor market outcomes for students, Calbright has developed an employer engagement plan which complements the strong partnerships we have established with regional organizations, workforce associations and boards, community based organizations, and government agencies. Through the third quarter of 2022, the comprehensive plan highlights our focus on statewide outreach and engagement with cross-industry employers to increase visibility of Calbright programming and position our adult learners for optimal labor market outcomes. Building on this, we have also partnered with the Bay Area Council, a leading business association in California, to facilitate information sessions, job site visits, webinars, and multi-industry convenings with priority employers.

As shared in November 2021, Calbright has also developed and implemented a Career Services Plan (CSP) that describes our strategies for helping students and alumni obtain positive employment outcomes. We provide career readiness and career advancement services and resources to students throughout their educational journey, including career coaching, career readiness workshops, and industry events.

To measure students' employment outcomes after graduation, Calbright currently conducts alumni surveys and has been added to LaunchBoard, a statewide data system supported by the CCC Chancellor's Office. However, employment data is a lagging metric and may not be available right away, due to the manner in which LaunchBoard collects it. Calbright also plans to survey employers to track the outcomes of program completers, such as retention after upskilling, earnings gains, and upward mobility.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

As noted in the previous evaluation, we look forward to reviewing Calbright's success in assisting its graduates in obtaining positive employment outcomes in July 2022.

6-Month Agency Response

Calbright has fully implemented the first two items under Recommendation 9, and will complete the third by the stated deadline, July 2022.

Calbright has developed and implemented a Career Services Plan (CSP) that describes our strategies for helping students and alumni obtain positive employment outcomes. We provide career readiness and career advancement services and resources to students throughout their educational journey, including career coaching, career readiness workshops, and industry events. Career Services also collaborates with the Workforce Development team to establish partnerships with employers, industry advisory boards, workforce development boards, and other agencies to provide a path toward securing job placements for students.

To measure students' employment outcomes after graduation, we currently conduct alumni surveys and have taken steps to be added to LaunchBoard, a statewide data system supported by the CCC Chancellor's Office. LaunchBoard provides data on employment and wages by matching student records with the state's Unemployment Insurance wage file. We expect to be included in LaunchBoard in 2022, but employment outcomes data may not be available right away, due to external lags in data collection. Calbright also plans to survey employers to track the outcomes of program completers, such as retention after upskilling, earnings gains, and upward mobility. Using available data, Calbright will demonstrate that it has been successful at assisting its graduates in obtaining positive employment outcomes by July 2022.

Additionally, as discussed in the CSP, Calbright tracks students' use of career services through case notes, post-session surveys, and its alumni survey. To assess employer satisfaction with Calbright's preparation of students, we collect feedback from employer partners, such as through our advisory councils, and plan to conduct surveys of employers that are strategic enrollment partners for particular programs.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

Calbright provided a career services plan that describes career services and the various stages throughout a student's enrollment where those services will be offered to students. The career services that the plan describes include services such as resume building, job searching, and networking. Calbright also noted in the plan a method for tracking student usage of career services. Calbright listed one method for tracking student success in achieving positive employment outcomes - an alumni survey - however, the survey that Calbright provided was a draft.

Although the career services plan briefly describes some work that Calbright has done to establish partnerships to increase employment opportunities for students, specifically with local Workforce Development Boards and a job search organization called Opportunity@Work, it has little description of plans for future partnerships or how those partnerships may lead toward job placements.

We look forward to reviewing Calbright's progress toward demonstrating that it has successfully aided graduates in obtaining positive employment outcomes when its next update is due in July 2022.

60-Day Agency Response

Calbright centers its work on ensuring students obtain positive employment outcomes. This work is shared across pathway development and career services.

Our design research approach to pathway development is centered on employer needs, market research, and user feedback. As referenced in our response to recommendation #8, Calbright's process starts with determining need based on employment supply and demand. We engage with employers, Advisory Councils, Workforce Boards and Unions to understand urgent needs. We conduct Labor Market Research on emerging trends and qualitative interviews to stay abreast of workforce realities. While validating our programs we use qualitative interviews to determine challenges, realities and aspirations of those currently in the workforce. Those insights inform our coursework and guide how we help our graduates to become workforce ready. To support our continuous improvement efforts, Calbright conducts ongoing outreach to industry and employers to stay current with urgent needs and trends and distributes student outreach surveys.

Career Services provides opportunities for our students to supplement their program coursework with connections to the workforce they are studying. Based on the needs that students communicate, Career Services also develops and facilitates workshops for resume writing, interviewing, and other skills development. Through a variety of surveys, including alumni surveys, we discover which students have received job offers as a result of their development with us, and we are able to continue to offer services to ones who are still seeking jobs.

Calbright has begun to document this process and is on schedule for full implementation.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #10 To: Calbright College

To ensure that it is providing students with the assistance they need to graduate, Calbright should, by November 2021, establish systems to monitor the effectiveness of its student support efforts. Specifically, it should take the following actions:

1. Establish a monitoring system to ensure that it provides each student with the supports it has identified in its student support plan.

2. Conduct an annual survey of enrolled students to assess their satisfaction with its support services and instruction and with their own progress toward their educational goals.

3. Every six months, evaluate the effectiveness of the student support plan, including reviewing data on its provision of support to its students, student progress, and its annual student survey. Following its review, it should adjust the plan as necessary.

4. Include in its annual report to the Board of Governors and the public the results of its annual student survey and the steps it has taken to address student feedback.

Annual Follow-Up Agency Response From October 2023

Calbright has developed and implemented a Student Support Plan, conducted a new annual survey to enrolled students (spring of 2022), reported data from the survey to the College's Board of Trustees (summer of 2022), and will continue to evaluate and refine the Student Support Plan on a six-month cycle, as well as report annual student survey results to the Board each year. The auditor has indicated that the information provided in past updates is satisfactory, and has also requested additional information about how the College uses student progress and engagement information to inform its support activities.

Among other mechanisms, Calbright utilizes heat maps to inform outreach activities. These heat maps categorize student activity and alert the support team to students who have been inactive for distinct lengths of time (such as 10-21 days). Each week, these reports are pulled and students are then called, texted and emailed in order to assist them with persistence. Three weeks after being contacted, data is tracked to see which students re-engaged with their coursework and to evaluate the effectiveness of the outreach. From April 2023 to August 2023, for example, on average 26.7% of students completed a substantive academic activity (a module or quiz, for example) following outreach, while 47.2% did some kind of activity or reading within their coursework.

In addition, and in response to student feedback, Calbright also launched a timeline pilot in the spring of 2023 that created a non-binding structure around students' progress in the College's CRM program. In the first month after enrolling, students offered timelines as part of the pilot completed roughly 10% more program assignments, compared to students in the control group. Nearly four months into using timelines, the pilot group completed roughly 15% more assignments than the control group. The timeline structure will continue to be implemented across programs throughout 2023 and 2024.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Calbright provided documentation showing that it has incorporated some data tracking and managerial oversight to monitor students' receipt of its various support services. The documentation describes evaluating whether certain student support efforts resulted in students making further progress in their programs. It also has reported on three different student surveys, and included information in its milestone report to the Board of Governors on actions it has taken in response to student feedback.

Annual Follow-Up Agency Response From November 2022

Calbright has developed and implemented a Student Support Plan, conducted a new annual survey to enrolled students, reported data from the survey to the College's Board of Trustees, and will continue to evaluate and refine the Student Support Plan on a six-month cycle. The auditor has indicated that the information provided in past updates is satisfactory, and has also requested additional information about how the College uses student progress and engagement information to inform its support activities.

Calbright uses a variety of methods to monitor the effectiveness of its student support efforts. Over the summer of 2022, Calbright faculty and staff created heat maps that are built utilizing student progress and engagement data within the College's learning platforms. Heat maps allow student-facing members of the team to quickly reference student progress, design personal and group outreach endeavors, and also help identify common hurdles and understand who is moving forward and who may need additional support.

Early indicators show that these heat maps and associated outreach activities are making an impact: During the month of September 2022, counselors reached out to students who had been inactive for 21-30 days and, as a result, 42% of the students logged back in and engaged with their coursework. In October 2022, the re-engagement rate increased to 52%.

Per the auditor's additional request, the Calbright team will provide a demonstration of College's heat maps during a meeting on November 7, 2022.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Calbright demonstrated that it has tools, which it refers to as heat maps, available to monitor students' progress and identify when a student's inactivity suggests that the student may need additional support. However, Calbright lacks formally documented procedures to establish when and how it will use those tools to ensure that students receive the support they should. As we describe in the report, if Calbright does not monitor its staff's adherence to its student support plan, it will lack assurance that it is supporting students in the way it intends. Additionally, a formalized process for monitoring that support will better enable it to evaluate the effectiveness of the support it provides. We will consider this recommendation to have been fully implemented when Calbright demonstrates that it has implemented all four elements of the recommendation, including formalizing a process for monitoring the support it provides to students and evaluating the effectiveness of that support.

1-Year Agency Response

Calbright has developed and implemented a Student Support Plan: a learner-centered approach to providing timely resources and support throughout the student journey. This plan includes resources and support provided by the enrollment, counseling, instructional, and career services teams, as well as coordination between those teams. This plan will be evaluated every six months, and annual student survey results, as well as steps taken to address student feedback, will be shared in the College's annual report to the Board of Trustees.

Calbright uses a variety of methods to monitor the effectiveness of its student support efforts. The College tracks contacts with prospective students and creates "Cases" at various stages of the application process, which signal Student Support Specialists to reach out to students via phone, text, and email. Instructors and counselors also monitor student progress—through Caseload Explorer and other systems—to target outreach (e.g., to students who are nearly finished with their program, or to those who have not attempted any activity within a certain timeframe). In their one-on-one outreach to students, counselors use Recommended Schedules for Program Completion to help set goals for coursework and make sure students are on track to achieve them. Early indicators show that the student support plan is making an impact: In October 2021, 88% of Calbright students had engaged in their program in the past 180 days. During the first quarter of 2022, 88% had been engaged during the past 120 days, a 33% improvement.

Additionally, Calbright has conducted surveys of enrolled students to assess their satisfaction with their program as well as the College's support services, and plans to administer a revised survey by Q2 2022. This new annual survey will be sent to all students and cover satisfaction with our support services and instruction, with students' own progress toward their educational goals, and with program enhancements.

California State Auditor's Assessment of 1-Year Status: Pending

Calbright has updated its student support plan to include more accountability measures for ensuring that the college identifies when students may need additional support and providing such support. However, the college has not yet provided evidence that it has implemented those measures.

6-Month Agency Response

Calbright has developed and implemented a Student Support Plan: a learner-centered approach to providing timely resources and support throughout the student journey. This plan includes resources and support provided by the enrollment team, counseling team, instructional team, and career services team, as well as coordination between those teams.

Calbright uses a variety of methods to monitor the effectiveness of its student support efforts. The College tracks contacts with prospective students and creates "Cases" at various stages of the application process, which signal Student Support Specialists to reach out to students via phone, text, and email. Instructors and counselors also monitor student progress -- through Caseload Explorer and other systems -- to target outreach (e.g., to students who have not attempted any activity within a certain timeframe). In their one-on-one outreach to students, counselors use Recommended Schedules for Program Completion to help students set goals for their coursework and make sure they're on track to achieve them. The effectiveness of the Student Support Plan will be evaluated at least every six months, and the plan will be adjusted as needed.

Additionally, Calbright has conducted surveys of enrolled students to assess their satisfaction with their program as well as the College's support services, and plans to administer a revised survey by Q2 2022. This new annual survey will be sent to all students and cover satisfaction with our support services and instruction, with students' own progress toward their educational goals, and with upcoming program enhancements. Results will be presented to the College's Board of Trustees and the public.

California State Auditor's Assessment of 6-Month Status: Pending

Calbright provided a document that it stated is its Student Support Plan. The plan provides a timeline for contacting students who have applied but not yet completed the steps to become fully enrolled in the program. It also describes at a high level the support services that it will provide to students during their enrollment in the program. However, the Plan lacks specificity with regard to a monitoring system for ensuring that college identifies when students may need additional support, and for ensuring that the college provides that support. It also does not call for an annual survey of enrolled students.

60-Day Agency Response

Calbright utilizes data to monitor student progress and academic activity as well as inform our support of students towards completion. This data helps inform counselors and instructors on which students may need additional support and guidance. The platforms we utilize for the data are Salesforce Customer Relationship Management (CRM) and Strut Learning Management System (LMS).

Calbright conducts student surveys at various intervals to assess student feedback on instruction, student support services, and technology. These surveys include alumni surveys (program completion), mid-program surveys (mid-program), and end-of-term surveys (end of each six-month term). Additional feedback sources include one-to-one meetings with counselors and instructors, engagement with Calbright's support teams, Career Services interactions, informal webchats, instant messaging channels dedicated to student feedback, and what we call Calbright Cafe meetings.

Cross-functional teams meet weekly to analyze data and evaluate the effectiveness of the student support plan and recommend modifications to consider and enact. Results from surveys are shared across the college and we will be presenting our findings.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #11 To: Calbright College

To assist its students in completing its programs, Calbright should immediately establish an efficient process to recognize previous training and experience and allow students to bypass areas of curriculum in which they have demonstrated competence.

Annual Follow-Up Agency Response From October 2023

Credit for prior learning (CPL) is a critical tool to help adult learners succeed. It is used to evaluate and formally recognize learning that has occurred outside of the traditional academic environment and to grant college credit, certification, or advanced standing toward further education or training. CPL helps students value the skills and assets they bring to the table and well-designed CPL can boost college completion and workforce success particularly in underserved communities.

Calbright's competency-based education naturally allows students to move quickly through material they already know. In addition, we currently utilize a "prerequisite challenge form" in our IT programs so a student with prior IT experience may advance directly into our Cybersecurity program and bypass the prerequisite Introduction to IT program. Another example of Calbright's recognition of prior learning can be found in the College's newest Data Analysis program, which allows students to take a pre-assessment to evaluate their current knowledge. If they convey mastery in that pre-assessment, they may move directly to the final summative assessment for a competency and, if they pass that assessment, they are able to move ahead without engaging with all (or any) of the activities within the curriculum of that competency - they are able to "test out" so to speak.

To formally award credit for prior learning, Calbright had to first obtain accreditation, which it received in July 2023, one-and-a-half years ahead of the April 2025 deadline outlined in the College's founding legislation. The College will now begin the process of developing plans to offer credit, not only for prior learning, but also for its programs so that students can transfer credit into Calbright as well as carry Calbright credits to other institutions. Calbright expects to operationalize and implement this plan in late 2025, near the end of its seven-year start-up period.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From October 2023

Credit for prior learning (CPL) is a critical tool to help adult learners succeed. It is used to evaluate and formally recognize learning that has occurred outside of the traditional academic environment and to grant college credit, certification, or advanced standing toward further education or training. CPL helps students value the skills and assets they bring to the table and well-designed CPL can boost college completion and workforce success particularly in underserved communities.

Calbright's competency-based education naturally allows students to move quickly through material they already know. In addition, we currently utilize a "prerequisite challenge form" in our IT programs so a student with prior IT experience may advance directly into our Cybersecurity program and bypass the prerequisite Introduction to IT program. Another example of Calbright's recognition of prior learning can be found in the College's newest Data Analysis program, which allows students to take a pre-assessment to evaluate their current knowledge. If they convey mastery in that pre-assessment, they may move directly to the final summative assessment for a competency and, if they pass that assessment, they are able to move ahead without engaging with all of the activities within the curriculum of that competency - they are able to "test out" so to speak.

To formally award credit for prior learning, Calbright had to first obtain accreditation, which it received in July 2023, one-and-a-half years ahead of the April 2025 deadline outlined in the College's founding legislation. The College will now begin the process of developing plans to offer credit, not only for prior learning, but also for its programs so that students can transfer credit into Calbright as well as carry Calbright credits to other institutions. Calbright expects to operationalize and implement this plan in late 2025, near the end of its seven-year start-up period.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From November 2022

Credit for prior learning (CPL) is a critical tool to help adult learners succeed. It is used to evaluate and formally recognize learning that has occurred outside of the traditional academic environment and to grant college credit, certification, or advanced standing toward further education or training. CPL helps students value the skills and assets they bring to the table and well-designed CPL can boost college completion and workforce success particularly in underserved communities.

Calbright's competency-based education naturally allows students to move quickly through material they already know. In addition, we currently utilize a "prerequisite challenge form" in our IT programs so a student with prior IT experience may advance directly into our Cybersecurity program and bypass the prerequisite Introduction to IT program.

Calbright has also developed an implementation framework for a comprehensive approach to CPL, including a Board Policy, that provides a streamlined student experience and adheres to state compliance rules and regulations. This framework is conditioned on high-quality assessment design and the technical implementation necessary for students to benefit from CPL. This structured approach includes conducting research, implementing new technology, determining assessment methods, defining administrative procedures, and providing training and support to optimize the process for students. Calbright's Implementation Plan includes a timeline for the development, implementation, and review of a Credit for Prior Learning process.

To formally award credit for prior learning, Calbright must first obtain accreditation from an accreditor recognized by the U.S. Department of Education, which we are actively seeking to secure and for which we are two years ahead of schedule.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

1-Year Agency Response

Credit for prior learning (CPL) is a critical tool to help adult learners succeed. It is used to evaluate and formally recognize learning that has occurred outside of the traditional academic environment and to grant college credit, certification, or advanced standing toward further education or training. CPL helps students value the skills and assets they bring to the table and well-designed CPL can boost college completion and workforce success particularly in underserved communities.

Calbright's competency-based education naturally allows students to move quickly through material they already know. In addition, we currently utilize a "prerequisite challenge form" in our IT programs so a student with prior IT experience may advance directly into our Cybersecurity program and bypass the prerequisite Introduction to IT program.

Calbright has also developed an implementation framework for a comprehensive approach to CPL, including a Board Policy, that provides a streamlined student experience and adheres to state compliance rules and regulations. This framework is conditioned on high-quality assessment design and the technical implementation necessary for students to benefit from CPL. This structured approach includes conducting research, implementing new technology, determining assessment methods, defining administrative procedures, and providing training and support to optimize the process for students. Calbright's Implementation Plan includes a timeline for the development, implementation, and review of a Credit for Prior Learning process.

To formally award credit for prior learning, Calbright must first obtain accreditation from an accreditor recognized by the U.S. Department of Education, which we are actively seeking to secure and for which we are two years ahead of schedule.

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

Credit for prior learning (CPL) is a critical tool to help adult learners succeed. It is used to evaluate and formally recognize learning that has occurred outside of the traditional academic environment, to grant college credit, certification, or advanced standing toward further education or training. By meeting learners where they are, CPL helps students value the skills and assets they bring to the table and well-designed CPL can boost college completion and workforce success particularly in underserved communities.

Calbright currently utilizes a challenge method to assess prior learning in our IT programs. We have implemented a process to allow students with prior IT experience to advance directly into the Cybersecurity program, bypassing the prerequisite of our IT program, through the use of our "prerequisite challenge form."

Calbright has also developed an implementation framework for a comprehensive approach to CPL that provides a streamlined student experience and adheres to state compliance rules and regulations. This framework is conditioned on high-quality assessment design and the technical implementation necessary for students to benefit from CPL. This structured approach includes conducting research, implementing new technology, determining assessment methods, defining administrative procedures, and providing training and support to optimize the process for students. Calbright's Implementation Plan includes a timeline for the development, implementation, and review of a Credit for Prior Learning process.

To formally award credit for prior learning, Calbright must first obtain accreditation from an accreditor recognized by the U.S. Department of Education, which we are actively seeking to secure.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

Recognition for prior learning results in the awarding of credit, certification, or advanced standing toward further education, based on equivalent learning outside the classroom. To formally award credit for prior learning, Calbright must first obtain accreditation, currently in process through a U.S. Department of Education recognized accreditor. In the interim, Calbright has taken steps to implement a process for the recognition of prior learning.

Under a competency-based educational program, students demonstrate mastery of skills associated with the academic program typically through assessment and can hasten advancement through their programs with prior experience. Calbright has taken initial steps in its cybersecurity programs to allow students with prior experience to advance faster and is working to develop additional opportunities for students to accelerate progress through demonstrated mastery.

Calbright has a process in place to collect information from students about their prior IT experience, to support the advancement of students into the Cybersecurity program without completing the recommended IT support program or possessing CompTIA A+ certification. The experience is validated by the Cybersecurity instructional faculty. This process will be evaluated to determine how best to leverage the approach with other program pathways.

Calbright has researched best practices in the assessment of prior learning and has begun analyzing how our current course structures and content can be used to assess mastery of competencies in ways that can be leveraged to recognize prior learning. Process design for the recognition of prior learning has an estimated completion date of November 2022.

Full implementation for students also requires the college to accelerate development work on skills assessment, and rework existing onboarding and instructional technologies. The anticipated completion date of this work is May 2022.

California State Auditor's Assessment of 60-Day Status: Pending

All Recommendations in 2020-104

Agency responses received are posted verbatim.