Report 2018-114 Recommendation 9 Responses

Report 2018-114: Disabled Veteran Business Enterprise Program: The Departments of General Services and Veterans Affairs Have Failed to Maximize Participation and to Accurately Measure Program Success (Release Date: February 2019)

Recommendation #9 To: Veterans Affairs, Department of

To ensure that its outreach efforts are effective and result in a greater number of DVBE firms available that can provide the necessary goods and services awarding departments are seeking, CalVet should assess, at least annually, the effectiveness of its past outreach efforts in increasing the number of DVBE firms that become certified.

Annual Follow-Up Agency Response From October 2021

Through the establishment of two staff through an inter-agency agreement with DGS, and a restructured DVBE Advisory Council, CalVet conducted a review of its DVBE outreach. Based on the assessment it was determined that there are several areas of opportunity. The analysis revealed two primary findings:

- There was not an adequate means of tracking inquiries from current and prospective DVBEs made to CalVet.

- There was not an adequate means of determining direct feedback from awarding departments regarding the necessary goods and services they are seeking.

With this information CalVet created the Request for Response survey hosted on its website and a direct email that CalVet staff monitors on a daily basis. The Request for Response survey and the DVBE email are avenues for DVBE community to gain increase access to state advocates, as well as communicate with CalVet in an effort to improve communication. This allows CalVet the ability to refer current and prospective DVBEs directly to the Department of General Services (DGS) certification officers to begin the process of acquiring their certification, and maintain communication and follow up with the veteran throughout the process.

Additionally, CalVet developed the following surveys in order to gain further resolution on the needs of awarding departments to further increase outreach efficacy:

1. A certified DVBE survey that identifies goods and services that certified DVBEs offer.

2. A survey that identifies what goods and services awarding departments are struggling to find DVBE vendors

CalVet will conduct an annual assessment for DVBE outreach as well as use the information gathered from these surveys to connect awarding departments directly to the DVBEs that offer the goods and services they are seeking.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From October 2020

On April 1 2020, VSD established an Interagency Agreement (IA) with DGS to provide CalVet the means to implement the recommendations of this audit. The IA includes two full time analyst positions which were fully staffed on October 1, 2020. The essential functions of these positions will assist the department in analyzing outreach efforts by:

-Working directly with DVBE Advocates throughout the state to discover the procurement needs of awarding departments. CalVet has begun the process of training these staff working collaboratively with DGS. This will ensure CalVet's recruitment of DVBEs is strategic and works within business sectors that give DVBEs the greatest likelihood of success.

-Placing the two DVBE positions with the California Transition Program will provide greater support to these two positions through resources such as survey monkey, research staff, and outreach.

-The establishment of the IA includes language for data sharing between the two departments with the first request for new DVBE certification data anticipated to commence on November 1, 2020. This will allow for the identification of a reliable benchmark to measure the success of DVBE-specific outreach through regular periodic surveys.

-The DVBE Advisory Council has been revamped significantly to ensure programmatic issues brought forth to the council are relevant and cycle through an efficient process which produces actionable recommendations for the department to address.

It is through the DVBE Advisory Council that the efforts of these organizations and individual DVBEs be coordinated, which will ensure growth of the program. The DVBE Advisory Council will serve as a culturally competent entity that will perform analysis of DVBE outreach activities to increase program participation.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

1-Year Agency Response

CalVet underwent leadership changes that directly impacted the DVBE program in late 2019. These changes also include a deviation from the previously submitted DVBE Promotion and Outreach Enhancement Plan. In January 2020, the DVBE program responsibilities were transferred to the Veterans Services Division (VSD). VSD is establishing a new Interagency Agreement (IA) with DGS effective April 01, 2020 to provide CalVet the means to implement the recommendations of this audit. The IA includes 2 full time analyst positions whose essential functions will assist the department in analyzing outreach efforts by:

- Working directly with DVBE Advocates throughout the state to discover the procurement needs of awarding departments. This will ensure CalVet's recruitment of DVBEs is strategic and works within business sectors that give fledgling DVBEs the greatest likelihood of success.

- Performing analysis of certification and procurement data provided by DGS. This will allow for the identification of a reliable benchmark to measure the success of DVBE-specific outreach through regular periodic surveys.

- Coordinating a revamped DVBE Advisory Council to ensure that outreach strategies developed are culturally competent, relevant, and implemented through a community-based infrastructure.

DGS's annual analysis of their past outreach efforts recommends that they "continue to strengthen relationships with DVBE-focused organizations." It is through the DVBE Advisory Council that the efforts of these organizations and individual DVBEs be coordinated, which will ensure growth of the program. The DVBE Advisory Council will serve as a culturally competent entity that will perform analysis of DVBE outreach activities to increase program participation.

CalVet will also ensure our newly established internal audits unit will be an integral part in ensuring full and timely implementation of the recommendations provided moving forward.

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

CalVet held further internal Discovery Meetings with DGS regarding previous DVBE promotional and outreach efforts. We identified the need for a more robust DVBE marketing plan. CalVet then developed a 2019 DVBE Promotion and Outreach Enhancement Plan (the Plan) that includes annual assessments as recommended (Attachment A). CalVet is in the initial phases of implementing the Plan. CalVet will continue to conduct Discovery Meetings with DGS and underperforming departments.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

CalVet agrees with this recommendation. In July 2018, CalVet reassigned the DVBE State Advocate to the Division of the Deputy Secretary of Minority and Underrepresented Veterans and began assessing past outreach efforts with internal and external "Discovery Meetings." The intent of the discovery meetings is to provide effective communication amongst stakeholders from CalVet to DGS to the implementing agencies and departments.

More specifically, these meetings evaluate past outreach efforts in order to understand systematic challenges within awarding departments. The results from the internal and external Discovery

Meetings include:

1. Modifying and formatting monthly DVBE reports to enable CalVet to track valid and invalid DVBE certifications, inspect inconsistencies, and target key DVBE industries for awarding agency needs;

2. CalVet will develop outreach events based on one-on-one awarding agency Discovery Meetings. CalVet noted targeted matchmaking for both awarding agencies and DVBEs;

3. CalVet will be identifying approaches to non-traditional methods of veteran businesses outreach. Target areas include: architectural, engineering, general contracting, facility management, building, management, and supplier management organizations;

4. Currently, CalVet is assessing new tools to measure DVBE success with targeted online surveys and batch surveys to be performed after every outreach event and bi-annually. The surveys will be distributed in June 2019.

5. The CalVet DVBE Advisory Council was reconstituted with a reorganized set of advisory board members, to better consider and facilitate CalVet's efforts with: effective DVBE outreach, DVBE mentorship with seasoned prime DVBEs as mentors, and to facilitate efforts with aligning qualified DVBEs with agencies that need to increase their DVBE goals. The first meeting occurred in February and additional meetings have been scheduled for calendar 2019.

California State Auditor's Assessment of 60-Day Status: Pending

All Recommendations in 2018-114

Agency responses received are posted verbatim.