Use the links below to skip to the section you wish to view:
- Rehabilitation Failed to Formalize Procedures for Soliciting and Awarding Grants, Resulting in Inconsistencies and Inadequacies in the Grant Process
- Significant Gaps in the Evaluation of Grants Raised Questions About the Adequacy and Fairness of the Process
- Rehabilitation Did Not Always Follow Its Appeals Process as Suggested in Its Grant Manual, and Its Review Committees Did Not Always Ensure Fair Evaluations
- Scope and Methodology
Rehabilitation Failed to Formalize Procedures for Soliciting and Awarding Grants, Resulting in Inconsistencies and Inadequacies in the Grant Process
- Although federal regulations require Rehabilitation to have procedures and state regulations require it to have a format in place for its grant process, during our audit period of fiscal years 2014–15 through 2017–18, Rehabilitation had developed only a grant manual that contains procedures it does not require staff to follow. Instead, it had provided its staff with discretion in how they solicit, evaluate, and award grants. This lack of established procedures and level of discretion were key factors contributing to the many inconsistencies and shortcomings we identified in Rehabilitation’s grant process. Further, without formalized procedures, Rehabilitation has potentially violated federal and state regulations and does not know whether its staff and evaluators follow a consistent and fair grant process.
- In our review of the grant manual, we found that the procedures were largely useful and reasonable, and in the absence of any other sufficient direction provided by Rehabilitation to its staff, we used the grant manual and the minimal state regulations that pertain to Rehabilitation’s appeals process to perform our assessment.
- For the four grants we reviewed, we found that in developing its RFAs, Rehabilitation failed to take steps that are critical to ensuring a consistent, complete, and fair grant process. For example, in most instances, it did not clearly define staff responsibilities, obtain stakeholder input and feedback on the content of RFAs, and disclose all information pertinent to the grant process in its RFAs.
- Further, in part because of its haphazard and disorganized approach to managing documentation of the grant process, Rehabilitation did not fully respond to several requests it received for public records regarding certain grants.
Failure to Develop Formalized Procedures for the Grant Process
Requirements for Formalized Procedures
and a Format for the Grant Process
Code of Federal Regulations
Since at least 1992, “Each state shall have procedures for reviewing and approving applications for subgrants and amendments to those applications, for providing technical assistance, for evaluating projects, and for performing other administrative responsibilities . . .”
California Code of Regulations
Since at least 2001, “Rehabilitation must establish and maintain a format to evaluate, prioritize, and award applications for grants . . .”
State Plan for Independent Living
Since at least 2008, “Rehabilitation will comply with all applicable federal statutes and regulations in effect with respect to the three‑year period it receives funding under the state plan.”
Source: State and federal regulations, as well as state plans for the last 10 years.
Although Rehabilitation is aware that it is required to have procedures in place for the grant process, it has developed only draft procedures and has not required staff to follow these procedures. Federal law requires states to satisfy certain conditions to receive federal aid and, as shown in the text box, for more than 25 years, the Code of Federal Regulations has required each state to have procedures for reviewing and approving applications for the grants awarded to service providers. Further, state regulations require Rehabilitation to establish and maintain a format to evaluate, prioritize, and award applications for grants. However, current state regulations do not prescribe procedures or establish a particular format for awarding grants and offer only minimal direction for processing appeals. Additionally, the State Plan requires Rehabilitation to comply with all federal legal requirements and to be consistent with state law.
As a result of these requirements, we expected Rehabilitation to have developed uniform written procedures, approved by its management, that sufficiently describe the steps in its grant process—from development of the RFA to the final decision on any appeals. The chief deputy acknowledged that Rehabilitation is aware of the requirements that it have procedures and that it should have finalized the grant manual; however, she explained that she was not aware that Rehabilitation had not finalized the grant manual until around the time the Joint Legislative Audit Committee (Audit Committee) approved this audit. Upon learning this, she stated that the former chief of the Contracts and Procurement Division (former contracts chief) advised her that the division had not finalized the grant manual because Rehabilitation does not award many grants and the Contracts and Procurement Division had other, higher priorities.1 The chief deputy stated that, had she known that Rehabilitation was lacking such critical procedures, she would have expedited completion of the grant manual. Without formalized written procedures, Rehabilitation has potentially violated federal and state regulations. Further, it lacks assurance that its staff and evaluators follow a consistent and fair grant process and that the most qualified applicants receive awards.
The chief deputy acknowledged that Rehabilitation should formalize its procedures as a matter of good practice, but stated that she believes Rehabilitation has met the requirement to have procedures through sharing practices by word of mouth, institutional knowledge, and reference to its previous RFAs to describe the grant process. We disagree that word of mouth, institutional knowledge, and past practices are a substitute for having formalized procedures, which we would expect to be written and approved by Rehabilitation’s management. Further, we found that the RFAs we reviewed are not uniform, consistent, or comprehensive. We also recognize that the California Administrative Procedure Act (administrative procedure act) generally prohibits state agencies from issuing or using any guideline, manual, or instruction that meets the definition of a regulation without first engaging in the administrative procedure act’s rulemaking process.2 Because the procedures in the grant manual are intended to describe how Rehabilitation solicits, evaluates, and awards competitive grants, as well as to supplement its scant regulations governing the appeals process, we find that if finalized, the grant manual likely meets the administrative procedure act’s definition of a regulation. Consequently, we believe Rehabilitation should engage in the rulemaking process. This is particularly true because Rehabilitation’s RFAs, as described later in this report, omit key steps and do not adequately describe Rehabilitation’s comprehensive grant process. Therefore, we believe that to ensure transparency, fairness, and compliance with both state and federal requirements, Rehabilitation should promulgate regulations to govern its grant process. Once Rehabilitation promulgates regulations, it could further develop its grant manual to provide instruction to staff on how to meet the regulatory requirements.
Although the grant manual has been available to staff since approximately 2015, and Rehabilitation has provided the grant manual in response to two requests under the California Public Records Act (public records act) for grant policies and procedures, as well as supporting documents, Rehabilitation does not require staff to follow the current grant manual. Before Rehabilitation received the first appeal of the 2017 Systems Change grant award, which we describe here, Southern California Resource Services (SCRS), an applicant that was not awarded the grant, requested Rehabilitation's policies and procedures for conducting the grant process, and Rehabilitation provided SCRS with the grant manual. In doing so, it did not state to SCRS that the manual was a draft and did not represent its formal policies and procedures. Similarly, in response to a subsequent request from SCRS for Rehabilitation’s adopted grant solicitation manual, Rehabilitation again provided its draft grant manual. Although it included the word “draft” in the title of the electronic file that contained the grant manual, Rehabilitation did not explain that it does not require staff to follow the manual during the grant process. According to the former contracts chief, because the grant manual is in draft form, Rehabilitation does not consider it compulsory or enforceable—meaning that Rehabilitation does not require staff to follow it during the grant process. However, we find it troubling that Rehabilitation does not require staff to follow the grant manual but chose to present the manual to SCRS—a grant applicant—as its procedures. As of June 2018, the chief deputy advised us that Rehabilitation is working to promulgate regulations and finalize the grant manual.
In the absence of formalized procedures or other sufficient direction to Rehabilitation’s staff, and given that we found that the grant manual largely contains reasonable and useful procedures, we used the grant manual to assess the adequacy of Rehabilitation’s grant process. In general, the grant manual clearly defines each stage of the grant process and expectations of staff and evaluators. Further, the grant manual outlines process controls, which include the roles of staff and management who are responsible for overseeing aspects of the grant process to ensure that Rehabilitation carries it out appropriately. Nevertheless, we did identify multiple areas for improvement in the draft manual, as certain key procedures provide too much flexibility and discretion to program staff. For example, the grant manual states that before program staff begin developing the RFA, they should do some planning, including making decisions regarding how much weight each component of the scoring criteria, such as the applicant’s experience and effectiveness in providing relevant services and its financial position, will receive in the total score. However, by not explicitly requiring program staff to establish a weight for each component and to include that weight in each RFA, Rehabilitation risks not fully disclosing how it scores applications. Similarly, the grant manual states that staff should consider including information in the RFA about whether correspondence regarding an appeal can be made by email, how Rehabilitation will acknowledge receipt of an appeal, and who will send the acknowledgment. If this were a requirement, Rehabilitation would have greater assurance that applicants will have the information necessary for filing an appeal.
Lack of Clearly Defined Roles and Responsibilities in the Grant Process
In its grant manual, Rehabilitation suggests that, at the start of the development of each RFA, management and staff participating in the grant process conduct a kickoff meeting, wherein they are educated on the steps in the grant process and their roles and responsibilities in the process. The grant manual also suggests that participants in the kickoff meeting document the expected deliverables during the grant process and which staff are responsible for each deliverable. If conducted appropriately, kickoff meetings ensure that those involved in the grant process understand Rehabilitation’s expectations of them, document these expectations, and know how to carry out the grant process adequately as planned.
However, Rehabilitation could not demonstrate that it held these kickoff meetings or documented participants’ roles and responsibilities for the four grants we reviewed, as shown in Table 2. For example, for the 2017 Systems Change grant, Rehabilitation did not schedule a kickoff meeting before the grant process began, which officials stated began in November 2016, and instead held the meeting roughly two months later in January, just one week before it released the RFA—a key document in the grant process that requires significant coordination on the part of staff to prepare. Rehabilitation also did not document deliverables or staff roles and responsibilities during the grant process. According to the former chief of independent living, assistive technology, and traumatic brain injury programs (chief of independent living), who has since assumed another position at Rehabilitation but at the time was responsible for administering the 2017 Systems Change grant process, due to scheduling conflicts, it was not feasible to hold this meeting before starting the grant process without delaying the RFA. Further, the independent living deputy director, who was responsible for overseeing the entire process for the 2017 Systems Change grant, was not involved in the meeting when it was eventually held. As another example, for the 2014 OIB grant, the former OIB program manager, who has since retired, did not recall holding a kickoff meeting or otherwise defining staff roles. Rehabilitation’s failure to ensure that it clearly defined and documented participants’ roles and responsibilities in the grant process, and to ensure that key management responsible for overseeing the grant process were present at kickoff meetings, created an environment in which the shortcomings and inefficiencies in its grant process that we describe throughout this report were able to occur.
Various Rehabilitation staff also did not receive ethics training, which includes conflict‑of‑interest training, before becoming involved in the grant process, as shown in Table 2. The grant manual specifies that Rehabilitation employees in positions that involve awarding grants or contracts must receive ethics training, which includes conflict‑of‑interest training, every other year. Further, state law requires certain state employees involved in making governmental decisions—including most of those involved in the grant process—to attend training on relevant ethics laws and regulations within six months of assuming their position and every two years thereafter. These trainings include topics to help employees recognize any personal or private interests that may affect their ability to perform their job fairly and impartially. Because staff involved in the grant process may make decisions regarding grant awards, this training is of particular importance.
Despite these requirements, Rehabilitation could not demonstrate that five key staff members involved in the 2017 Systems Change grant—the independent living deputy director, chief deputy, former contracts chief, deputy director of specialized services, and an attorney—had completed the required ethics training within two years of the start of the grant process. Similarly, Rehabilitation could not demonstrate that key staff who worked on the 2017 OIB grant, including the chief deputy, former contracts chief, and deputy director of specialized services, had completed the required ethics training within two years of the start of the grant process. According to the chief of Rehabilitation’s human resources branch, a unit within her branch notifies managers of staff who need to complete ethics training; however, she explained that Rehabilitation expects managers to be responsible for ensuring that their staff receive the training. The administrative services deputy director, who oversees the human resources branch, stated Rehabilitation also expects managers, such as a deputy director or chief deputy, to ensure that they themselves receive the proper and required amount of training. By not ensuring that its employees receive ethics training as required by state law, Rehabilitation failed to take advantage of a resource that would increase the likelihood that its staff involved in the grant process will be fair and impartial.
|NAME OF GRANT AND YEAR AWARDED|
|RFA DEVELOPMENT AND SOLICITATION PROCESS||OIB
|Hold a kickoff meeting to ensure that management and staff are aware of their roles and responsibilities and that these expectations are documented||X||X||X||X|
|Ensure that management and staff receive ethics training, which includes conflict‑of‑interest training||X||X||X||X|
|Develop and train staff on confidentiality procedures||X||X||X||X|
|Ensure that management and staff sign the confidentiality and conflict‑of‑interest form specific to the grant||X||X||X||X|
|Solicit stakeholder feedback for RFA development||X||✓||♦||X|
|Develop RFA, including adequate:|
|Evaluation and award process||♦||♦||♦||♦|
|Create a single archive location for all documents developed during the grant process||X||X||X||X|
Source: California State Auditor’s review of four selected RFAs and supporting documentation from fiscal years 2014–15 through 2017–18, and Rehabilitation’s grant manual.
✓ = Followed
♦ = Partially followed
X = Did not follow
Rehabilitation also did not develop procedures or otherwise ensure that it informed staff at the beginning of the grant process of its expectations for handling confidential information or documentation, as shown in Table 2. According to the grant manual, confidentiality procedures should ensure that staff develop and store documents in a way that controls how, when, and to whom they are circulated; describe how staff identify, maintain, and dispose of draft documents containing confidential information; and designate specific communication channels for simultaneous disclosure of information to ensure that all interested persons have the same access. Such procedures are important to maintain the integrity of the process and to prevent accidental disclosure of documents before Rehabilitation issues the final award. Given the importance of this issue, we would expect Rehabilitation to have developed confidentiality procedures to help ensure that staff working on grants know how to handle documentation during the grant process. Rather, according to the former contracts chief, Rehabilitation typically provides verbal instruction to staff at the beginning of the grant process on how to handle confidential information.
For the four grants we reviewed, program managers and the deputy director of specialized services, who provided perspective on the 2014 OIB grant in the absence of the retired program manager, explained that they did not provide any written procedures to staff regarding how to handle confidential information. In one case, the program manager recalled verbally discussing it, and in other instances program managers indicated that they had not received instruction to develop confidentiality procedures. This informal approach is concerning because Rehabilitation lacks assurance that staff receive complete and consistent direction on how to handle confidential information at the beginning of each grant process, if they receive direction at all, and risks staff disclosure of confidential information. The former contracts chief acknowledged that developing standardized confidentiality procedures would be beneficial going forward.
Rehabilitation also did not ensure that staff certified that they were free from conflicts of interest or that they understood confidentiality expectations. Specifically, the grant manual states that all Rehabilitation staff involved in the grant process must certify that they have no conflicts of interest with any of the organizations that are competing for the grant and that they understand how to keep the grant process confidential. Staff are to certify this by signing a conflict‑of‑interest and confidentiality form. The grant manual also states that for staff to have the knowledge necessary to sign this form, Rehabilitation must first provide training so that they understand what they are signing, what is expected, and the penalties they could suffer if they breach confidentiality or have a conflict of interest. However, we found that only one of the 26 employees involved in the four grants we reviewed signed a conflict‑of‑interest and confidentiality form. In addition, none of these employees received training on conflict of interest and confidentiality specific to the individual grant, as shown in Table 2. The program managers for these grants generally stated that they did not provide staff involved in the grant process with training or have them sign conflict‑of‑interest and confidentiality forms because they were not aware that it is Rehabilitation’s practice to do so. By not having procedures for managers to follow to ensure that staff receive the required training, are free from conflicts of interest, and understand confidentiality procedures, Rehabilitation risks providing an unfair advantage to certain applicants if a staff member has a connection to an applicant or inadvertently releases confidential information before the award is made public.
Failure to Solicit Stakeholder Feedback
Rehabilitation did not always solicit stakeholder input and feedback to inform its development of RFAs, missing the opportunity to receive valuable insights from subject‑matter experts and the disability community. However, the policy of inclusion and full participation of individuals receiving public funds for specified rehabilitation services is articulated in federal law, while state law requires Rehabilitation both to be consistent with federal policy and to attempt to ensure the regular and meaningful involvement of the disability community in the policy development and implementation of independent living programs. Further, the grant manual states that Rehabilitation should reach out to stakeholders, including the potential pool of applicants, before and during the development of an RFA, in part to increase stakeholder and applicant satisfaction with the RFA and the evaluation process, as well as with the selection results.
However, as shown in Table 2, Rehabilitation did not adequately solicit stakeholder feedback in developing the RFAs for three of the four grants we reviewed. For example, the former chief of independent living explained that Rehabilitation did not solicit stakeholder feedback for the 2017 Systems Change grant because it had a short time frame in which to develop the RFA, as the State Council did not approve the operations plan for the State Plan until November 2016, due to new requirements and procedures from the Workforce Innovation and Opportunity Act. In addition, he stated that he relied on staff from Rehabilitation’s Contracts and Procurement Division for guidance, and they did not advise him to solicit stakeholder feedback. Further, because Rehabilitation had already issued the RFA for the Systems Change grant in past years, and the independent living community and stakeholders were already familiar with the grant, he did not think it necessary to solicit feedback on procedural components of the RFA. Although we understand the delay and shortened time frame to complete the RFA, we believe it would have been beneficial for Rehabilitation to seek feedback from the disability community and subject‑matter experts to inform the development of its RFAs and address any stakeholder concerns.
Rehabilitation sufficiently solicited and obtained stakeholder input and feedback for just one of the four grants we reviewed. Specifically, for the 2015 TBI grant, Rehabilitation published a survey on its website requesting stakeholder input and feedback on the number of grants it should award and asking them to rate the importance of scoring criteria. In contrast, Rehabilitation did not solicit stakeholder input and feedback when developing the RFA for the 2014 OIB grant. For the 2017 OIB grant, Rehabilitation chose to solicit input and feedback only from certain stakeholders. In particular, to notify stakeholders of its meeting regarding the RFA, Rehabilitation sent an email to the program’s mailing list that, according to the OIB program manager, included representatives from the grantees for the previous award. However, to ensure that it offered all interested stakeholders the opportunity to participate in the meeting, we would have expected Rehabilitation to at least post a notice about the meeting on its website. By not attempting to solicit stakeholder input and feedback from the greater disability community to inform the development of its RFAs, which are the foundation of the funding decisions for each grant, Rehabilitation risks that the final RFA could be missing information or could contain inaccurate information about the grant or the evaluation and award process.
Lack of Information in RFAs About the Evaluation, Award, and Appeals Process
None of the RFAs we reviewed included specific information about how evaluators would assign points to each of the scoring components when reviewing applications, which we found to be a best practice. To identify best practices for disclosing to applicants the process evaluators would use to score applications, we selected and reviewed two RFAs for grants of federal funds administered by two other state agencies. We found that both RFAs included detailed descriptions of how evaluators would score applications. For example, one agency described in its RFA the process evaluators would use to score applications, from 0 for inadequate to 4 for excellent or outstanding. It also explained the basis for each point assignment, such as that applications receiving a score of 4 demonstrate the applicant’s ability and intent to exceed the requirements, provide evidence of the applicant’s current ability to comply with the grant requirements, and propose detailed plans or methodologies that further describe how the applicant will exceed requirements. However, Rehabilitation did not include similar detailed descriptions in its RFAs. For example, out of a total of 10 points possible for a scoring component, we would expect the RFA to disclose that a well‑qualified score would be between 7 and 10 points, a qualified score would be between 4 and 6 points, an unqualified score would be between 1 and 3 points, and a component that was missing would receive a score of 0. Instead, for the four RFAs we reviewed, Rehabilitation disclosed only the total number of possible points for each component—such as 21 points for the Plan of Operation component. Without a description of the range of points for each level of quality, applicants may not fully understand the type of information and level of detail necessary to achieve high scores, the specific criteria that evaluators are using to assess their applications, or the significance of the scores their applications receive.
Further, Rehabilitation provided scoring documents to evaluators that lacked adequate guidance on how to score applications. The grant manual indicates that, in conjunction with the scoring criteria in its RFAs, Rehabilitation should develop one benchmark for each scoring component—such as Organizational Structure and Capacity, Core Services, and Accessibility and Populations Served. These benchmarks should provide evaluators with guidance on how to score applications by providing a detailed description of the content and level of quality needed for an applicant to receive a score of well‑qualified, qualified, or unqualified. For the 2017 Systems Change grant, we found that the benchmarks Rehabilitation provided to the third evaluation panel did not contain the numeric scoring range it expected for each level of quality, which may have given evaluators too much discretion in scoring applications.3 The former chief of independent living attributed the omission of numeric ranges from the scoring benchmarks to his general lack of experience in reviewing benchmarks.
Additionally, for the 2017 Systems Change grant and the 2017 OIB grant, Rehabilitation did not ensure that the scoring benchmarks it provided evaluators aligned with the evaluation criteria in the corresponding RFAs. For example, for the 2017 Systems Change grant, Rehabilitation provided the first evaluation panel with two sets of benchmarks, neither of which aligned with the scoring criteria in the RFA. Specifically, one set of benchmarks was incomplete because it did not contain portions of two scoring components, and the other included two scoring components that were not disclosed in the RFA, and excluded another four components that were disclosed in the RFA. Rehabilitation did not rectify these inconsistencies and provided each of the three evaluation panels inaccurate scoring benchmarks through subsequent rounds of scoring. Further, for the 2017 OIB grant, the scoring benchmarks indicated that applicants were required to submit letters of support, whereas the RFA did not specifically require these letters.
Similar to the issues with the benchmarks, we found that the score sheets Rehabilitation provided to evaluators did not align with the criteria in the RFA for two of the four grants we reviewed. In particular, the score sheets for the 2017 Systems Change grant included an evaluation component that was not listed in the RFA. For the 2015 TBI grant, the score sheets indicated that applicants were required to describe how they improved and developed needed services other than TBI, while the RFA required applicants to discuss only TBI services. When Rehabilitation does not ensure that its scoring materials align with the criteria in its RFAs, it risks that evaluators will perform unfair evaluations based on criteria of which applicants are unaware; use inappropriate criteria for scoring applications; and score applications inconsistently, which could result in scores that inaccurately reflect the quality of the applications.
Rehabilitation also did not fully disclose certain steps in its evaluation, award, or suggested appeals processes in the four RFAs we reviewed. For example, in each of the four grants we reviewed, we found that Rehabilitation did not adequately disclose important details regarding its evaluation process, such as the number of evaluators it planned to use, how it would select evaluators, and the type of oversight it would perform to ensure that evaluators conducted the evaluation as expected and fairly. Further, Rehabilitation did not disclose how it would announce the grant award decision in two of the four grants we reviewed. The grant manual also suggests that RFAs include detailed descriptions of the appeals process, such as how Rehabilitation will acknowledge an appeal, the evidence Rehabilitation will review, and the types of determinations it should reach based on its review of that information. However, the four RFAs we reviewed did not fully disclose the information regarding the appeals process that is set forth in regulations, such as that the review committee, which Rehabilitation's chief deputy appoints to review appeals, must deliver its decision within 30 days of the appeal date. Further, Rehabilitation did not disclose in the RFAs for the four grants we reviewed the additional steps suggested by its grant manual. By failing to include these descriptions in its RFAs, Rehabilitation has not adequately informed the public and potential applicants about regulatory requirements and its suggested appeals process, which could lead appellants to question the thoroughness or fairness of the process.
Rehabilitation also did not disclose, in three of the four RFAs we reviewed, how it would address tied scores between applicants. For the RFA for the 2015 TBI grant, Rehabilitation included a tiebreaker rule specifying that it would give greater priority to certain components in the event of a tie and would refer back to the highest scores of those components sequentially until it broke the tie. For this grant, there was a tie between two applicants, and Rehabilitation used the methodology disclosed in its RFA to break the tie. In contrast, we also found two ties between applicants for the 2014 OIB grant, and Rehabilitation did not explain a tiebreaker rule in its RFA for the grant. In this instance, Rehabilitation selected the applicants that were in closest proximity to the geographic area the grant would serve. Although we find this approach reasonable, we would expect Rehabilitation to disclose the methodology in the RFA. The grant manual discusses who is responsible for applying a tiebreaker rule; however, it does not prescribe a method for addressing applications that receive the same score, or require staff to include one in RFAs. Therefore, we also believe that Rehabilitation should consider establishing some standardized options for breaking ties that program staff can customize, such as using the highest‑priority components of each grant or the geographic location of the applicant and the county in which services would be provided. The former contracts chief agreed with our conclusion and stressed the importance of disclosing the chosen methodology in the RFA. By not disclosing a tiebreaker rule in its RFAs, Rehabilitation is missing the opportunity to increase transparency by informing applicants about how it will award grants in the event of a tie.
Failure to Enforce Application Filing Deadlines
Rehabilitation’s grant manual states that it should decide in advance of posting the RFA whether to apply a hard or soft deadline for applicants to submit their applications. The grant manual defines a hard deadline as one by which applicants must submit applications that are complete and ready for scoring, and any wrong or missing information, approvals, or certifications are grounds for rejection or a scoring penalty. The grant manual defines a soft deadline as one by which applicants must submit applications but can be given extra time to submit incomplete or incorrect information that is immaterial, as requested by the administrative review team. We found that Rehabilitation included hard deadlines in the RFAs for each of the four grants we reviewed by specifying that it would disqualify, or not consider for funding, any application packages received after the deadline or any that did not contain all required items by the specified deadline. However, as we describe here, Rehabilitation accepted some required items after the application deadlines for three of the four grants we reviewed.
Poor Document Management Practices and Responses to Requests for Public Records
For the four grants we reviewed, Rehabilitation could not demonstrate having responded completely or within the required time frame to 15 of the 29 public records requests it received from up to four applicants we selected for review from each grant. According to state law, within 10 days of receipt of the request for public records, Rehabilitation must respond in writing with an estimate of when it will provide copies of those records. Rehabilitation must also promptly make public records, with certain exceptions, available to any person that requests a copy of the record. State law also specifies that upon request, Rehabilitation must provide an exact copy of the requested record, unless it is impracticable to do so. However, for the 2017 Systems Change grant, Rehabilitation failed to fully respond to a request from SCRS by not providing it with a complete copy of an application that was requested because it omitted the application letter. In addition, across the four grants we reviewed, Rehabilitation did not fully respond to various other requests for documents, such as scoring criteria, evaluator selection notes, and evaluators’ score sheets.
Rehabilitation also failed to respond within the required or stated time frame to five requests, including three requests regarding the 2017 Systems Change grant, one for the 2015 TBI grant, and another for the 2017 OIB grant. For example, Rehabilitation took 13 days to respond to one of SCRS' requests and 31 days to respond to the request regarding the 2015 TBI grant. Further, for another request from SCRS related to the 2017 Systems Change grant, although Rehabilitation provided an initial response within 10 days, it delivered the remaining requested documents in multiple stages, starting three weeks after the date of the request and five days after it originally indicated it would provide the documents. Moreover, Rehabilitation provided the last of the documents nearly 40 days after the original request. Because of these delays, SCRS received relevant information past its deadline for submitting an appeal, and therefore filed an addendum to its appeal.
Rehabilitation provided differing reasons for not fully responding to the requests for public records or not responding in a timely manner. According to Rehabilitation's legislation and communications analyst, when Rehabilitation receives a request for public records, its Office of Legislation and Communications submits a document request and the corresponding request letter to relevant program staff. He and other Rehabilitation staff offered varying reasons for not fully responding or not responding in a timely manner, including that it was an oversight, that staff and management had to search their email for correspondence and save the documents to a centralized location, that program staff did not provide the requested records on time, and that the Office of Legislation and Communications inadvertently did not remind program staff to provide such records.
Rehabilitation’s poor records management and the fact that it did not ensure that staff complied with its records retention policy contributed to its failure to respond to public records requests completely or in a timely manner. The grant manual advises Rehabilitation to consider creating a single location to archive all documents developed during the grant process. Further, Rehabilitation’s records retention policy generally states that it must maintain all contract documents for seven years. According to the former contracts chief, this also includes agreements required by its federal agencies, such as those related to the grant process. However, Rehabilitation did not designate a centralized location to archive documents for any of the four grants we reviewed. Specifically, we found that some staff destroyed documentation; unnecessarily kept documents at their desks; did not save email correspondence in a centralized location; and deleted email correspondence, including key decisions and documents regarding the grant process. Staff, including one deputy director, attributed the failure to archive materials in a centralized location to a lack of instruction on how and where to store documents, particularly email correspondence. Staff's failure to retain all grant documents also indicates that staff were not familiar with Rehabilitation’s records retention policy. In addition, we found that the legislation and communications analyst could confirm the completeness of the public records act tracking log only beginning in September 2016, when he began working at Rehabilitation. Staff who created the log and tracked requests before that time have since left Rehabilitation. Thus, we cannot be certain that Rehabilitation provided us with all of the public records requests it received during our audit period. Rehabilitation’s failure to ensure that staff were familiar with its records retention policy, and to designate a centralized location for storing documents, sometimes rendered its responses to records requests related to grants incomplete and contributed to it not responding fully or in a timely manner to requests for public records.
To comply with federal and state requirements, and to ensure consistency and fairness in its grant process, Rehabilitation should do the following:
- Issue regulations describing its grant process from RFA development through appeals. It should submit its proposed regulations to the Office of Administrative Law no later than December 2018.
- Revise and formalize the policies and procedures in its grant manual to incorporate the rules adopted by regulation and to address the recommendations in this report. The grant manual should specify that any deviations from the required grant process must be for good cause and be documented.
To ensure that management and staff involved in the grant process are sufficiently informed about the process and their responsibilities, Rehabilitation should require these employees to attend a kickoff meeting before the development of each RFA in which participants discuss the key stages of the grant review process, each individual’s roles and responsibilities, and requirements surrounding conflicts of interest and confidentiality. Further, it should record these discussions in meeting minutes to ensure that expectations of employees are clearly defined and documented.
To comply with state laws and regulations and help ensure that staff involved in making governmental decisions during the grant process are impartial, Rehabilitation should ensure that they receive ethics training, which includes conflict‑of‑interest training, at least every two years.
To help ensure that staff involved in the grant process adequately protect confidential information, Rehabilitation should develop confidentiality procedures for each grant. Further, it should ensure that staff involved in the grant process sign the conflict‑of‑interest and confidentiality forms before the development of the RFA for each grant.
To ensure that it has received sufficient input and feedback from the disability community to inform the development of RFAs, Rehabilitation should solicit and document stakeholder input and feedback before and during the development of each RFA.
To increase transparency and ensure that applicants have the information necessary to understand the grant process, Rehabilitation should include in its RFAs clear scoring criteria and descriptions of the evaluation, award, and appeals processes, including the process it will use to address applications that receive tied scores.
To ensure that Rehabilitation maintains all relevant grant documentation and responds fully to requests for public records, it should immediately adhere to its records retention policy and save all grant‑related documents, including email correspondence and attachments, to a centralized location.
Significant Gaps in the Evaluation of Grants Raised Questions About the Adequacy and Fairness of the Process
- Administrative review teams accepted certain documents from some applicants after the application deadlines disclosed in the RFAs for three grants. In addition, by not publishing a solicitation for evaluators, as suggested by its grant manual, Rehabilitation limited its pool of prospective evaluators to select individuals and missed the opportunity to ensure that it obtained the most qualified evaluators possible from the larger disability community. Further, Rehabilitation did not adequately consider bias or the appearance of bias when selecting evaluators, or ensure that evaluators were free from conflicts of interest and were aware of confidentiality procedures before they scored applications.
- Rehabilitation could not demonstrate that it followed its grant manual by providing adequate training or written instructions to evaluators regarding the purpose of the grants, relevant regulatory requirements, or how to evaluate and score applications. Without adequate training, Rehabilitation’s evaluators did not have the knowledge they needed to score the applications appropriately, creating some delays in the grant process due to rescoring.
- The technical review teams, which are generally responsible for overseeing the evaluation process, did not always adequately review the evaluators’ scores and comments to ensure that evaluators followed the evaluation process and that the process was consistent and fair for the applicants for each of the grants we reviewed.
Acceptance of Grant Documents After the Filing Deadlines
Before Rehabilitation provides the grant applications to evaluators for scoring, the grant manual states that an administrative review team, which Rehabilitation assembles for each grant and which includes staff from the Contracts and Procurement Division, is responsible for reviewing all applications for completeness and ensuring that applicants submit all required documents. The grant manual also indicates that the administrative review team is responsible for determining whether applicants submit their applications by the deadline in the RFA. As we discussed previously, Rehabilitation included hard deadlines in each of the four RFAs we reviewed. Further, the grant manual states that in no case should Rehabilitation allow an applicant to change the content of an application narrative or other application component after the filing deadline. However, according to the former contracts chief, if the administrative review team identifies missing documents before the deadline, staff may ask the applicant to submit the documents, although Rehabilitation would accept only “immaterial” documents after the deadline. To explain what the administrative review team considers immaterial, the former contracts chief provided an example of an applicant submitting a required form without a necessary signature before the deadline, and explained that Rehabilitation would consider the missing signature immaterial and ask the applicant to sign and return the form after the deadline. Rehabilitation’s practice of accepting some documents after the submission deadline is similar to what the grant manual describes as a soft deadline.
The administrative review teams accepted certain documents after the hard deadline specified in the RFAs for three of the four grants we reviewed, as shown in Table 3, and thus may have provided some applicants with an unfair advantage. For example, for the 2015 TBI grant, we found that a member of the administrative review team requested additional documentation from the Independent Living Center of Southern California (ILCSC) after the submission deadline. One of the documents ILCSC submitted after the deadline fit the former contracts chief’s definition of immaterial documents; however, the other document was an entire revised section of ILCSC’s application narrative. The administrative review team accepted the required documents late and did not disqualify ILCSC, although under the terms of the RFA it should have done so. Subsequently, Rehabilitation granted one of the seven awards for the 2015 TBI grant to ILCSC, possibly preventing another qualified applicant from receiving grant funding. Rehabilitation also accepted late documents from some of the four applicants we selected to review for the 2014 OIB grant and for one of the four applicants we selected for review for the 2017 OIB grant. When we asked the chief deputy if the practice of accepting documents after the deadlines published in the RFAs was appropriate, she explained that she was unaware that the administrative review teams had accepted documents after the deadline, and that she did not believe this was an acceptable practice. Because Rehabilitation did not enforce its deadlines, it lacked a process control to ensure that the administrative review team did not accept portions of, or entire applications, after the deadlines.
Failure to Publish Solicitations for Evaluators and Ensure That Evaluators Are Bias Free
The grant manual states that Rehabilitation should publish a solicitation for evaluators on its website that includes the essential and desirable qualifications for evaluators. However, Rehabilitation could not demonstrate that it issued a solicitation for evaluators with a list of qualifications for three of the four grants we reviewed, as shown in Table 3. For the 2017 Systems Change grant, the chief of independent living at the time of the grant process stated that he did not publish a solicitation for evaluators for the first evaluation panel because he already had potential evaluators in mind, and for the second evaluation panel he was facing significant time constraints and wanted evaluators who would be seen as highly credible by the independent living community. As the grant manual indicates, the best evaluators are those with a breadth of knowledge that is relevant to the specific services provided under the respective grant. By not issuing a solicitation for evaluators, Rehabilitation limited its pool of prospective evaluators to selected individuals and missed the opportunity to ensure that it obtained the most qualified evaluators possible from the larger disability community.
|NAME OF GRANT AND YEAR AWARDED|
|ADMINISTRATIVE REVIEW AND EVALUATOR SELECTION PROCESS||OIB
|Administrative review team ensures that applicants met the deadline and included all required documentation per the RFA||X||X||X||✓|
|Issue a public solicitation for evaluators that includes essential evaluator qualifications||X||✓||X||X|
|Certify that evaluators are free from conflicts of interest before selecting evaluators||X||X||X||X|
Source: California State Auditor's review of Rehabilitation’s administrative review and evaluator selection processes for four grants awarded from fiscal years 2014–15 through 2017–18, and Rehabilitation's grant manual.
✓ = Followed
X = Did not follow
For one of the three grants for which Rehabilitation did not issue a solicitation for evaluators, it selected two evaluators with previous ties to one of the applicants, creating at least the appearance of potential bias. Consistent with state law, Rehabilitation’s grant manual states that evaluators must be free of financial interests in any of the applicant organizations. Although the grant manual states that evaluators must be free of personal relationships with any of the applicants’ principals or employees, state law is largely silent on prohibiting or disclosing personal bias in grant decision makers. We did not identify any evaluators with financial conflicts of interest; however, we found one grant in which Rehabilitation selected evaluators who had held leadership positions in an organization that had a known affiliation to the grant applicant that ultimately received the grant award.
Specifically, we found that the two evaluators who made up the second evaluation panel for the 2017 Systems Change grant were former executive directors of FREED Center for Independent Living (FREED), which is one of 21 member organizations of the awardee, the California Foundation for Independent Living Centers (CFILC). One of these evaluators had served as the executive director of FREED from 1994 through 2001, whereas the other served in this capacity from 2001 through 2007. These evaluators gave CFILC the higher score of the two applicants, and Rehabilitation subsequently awarded the grant to CFILC. The independent living deputy director indicated that Rehabilitation reviewed the evaluators only for financial conflicts of interest and did not discuss bias or the perception of bias when selecting them. However, by not carefully considering the effect its selection of evaluators could have on the integrity of the process, Rehabilitation created the potential for perceived bias when it selected evaluators with former leadership positions in an organization that was a member of one of the two applicants. The independent living deputy director explained that the independent living community is small, making it difficult to find evaluators who are subject‑matter experts without any connections to the applicants. Although this may be true, Rehabilitation further limited the pool of potential evaluators by choosing not to issue a solicitation for evaluators.
Rehabilitation also did not ensure that prospective evaluators were free of conflicts of interest before selecting them for any of the four grants we reviewed, as Table 3 shows. The grant manual indicates that after Rehabilitation provides evaluators with conflict‑of‑interest and confidentiality training, evaluators must sign a form certifying that they are free from any conflicts of interest with any of the organizations that are competing for the grant and that they understand how to keep the grant process confidential.4 For the 2014 and 2017 OIB grants, Rehabilitation had some evaluators sign conflict‑of‑interest and confidentiality forms on the day of the evaluation and could not demonstrate that it provided all evaluators with the list of applicants before they signed the form or scored the applications. By not following the procedure described in its grant manual, Rehabilitation risks facing delays in the grant process if it needs to find a replacement for an evaluator who identifies on the day of the evaluation, or after evaluations begin, that he or she has a conflict with an applicant.
For the 2017 Systems Change grant, Rehabilitation did not obtain conflict‑of‑interest and confidentiality forms until the day of the evaluation for some evaluators and, in one instance, could not demonstrate that evaluators signed such forms until months after they scored applications and after the grant was awarded. Specifically, several days after the evaluation concluded, an independent living manager sent the three evaluators on the first panel an email asking them to sign conflict‑of‑interest and confidentiality forms. More than three months later, she emailed two of the three evaluators, asking them to sign the conflict‑of‑interest and confidentiality forms again because she could not locate them. In another instance, Rehabilitation allowed an evaluator on the third evaluation panel for the 2017 Systems Change grant to submit her conflict‑of‑interest and confidentiality form along with her score sheet when she completed the evaluation—once again failing to ensure that an evaluator was free from conflicts of interest before scoring applications. The chief of independent living at the time stated that he was not concerned with the evaluator submitting the form late because he had a telephone conversation with her in which she declared that she did not have any conflicts of interest. However, we do not consider a verbal discussion between Rehabilitation staff and an evaluator sufficient to determine whether a conflict of interest exists.
In our review of another state agency’s conflict‑of‑interest policies for best practices in selecting evaluators, we found that they are more comprehensive than Rehabilitation’s. For example, the agency can also elect to use subject‑matter experts as evaluators; however, it requires evaluators to agree not to score an application from an agency for which they have worked, not to score an application on which they have consulted, and not to score an application from an agency with which they have had any connection that is, or might appear to be, a conflict. Further, the agency’s sample conflict‑of‑interest and confidentiality agreement requires evaluators to disqualify themselves if they have a personal or professional relationship with an applicant that could affect their objectivity. In contrast, Rehabilitation’s conflict‑of‑interest and confidentiality forms for three of the four grants we reviewed required evaluators to certify that they have no personal or financial interest, and no present or past employment or activity, that would be incompatible with their participation in the grant process. However, Rehabilitation’s forms did not define what it considers “incompatible” with the grant process. We also found that the conflict‑of‑interest and confidentiality forms used for the 2017 Systems Change grant process did not prohibit evaluators with past employment with applicants from participating in evaluations. To reduce potential bias or the perception of bias, we believe Rehabilitation should expand its disclosure on its conflict‑of‑interest and confidentiality forms to include past and current employment, past and current relationships, and activities that may constitute bias.
As described previously, Rehabilitation did not establish procedures or inform staff of how to keep information and documentation regarding its grant process confidential. Similarly, it did not provide adequate training to evaluators to ensure that they understood how to keep the process confidential before having them sign conflict‑of‑interest and confidentiality forms, as shown in Table 4. For the 2017 Systems Change grant, email correspondence indicates that Rehabilitation received notice of a possible confidentiality breach when an evaluator, who was not a Rehabilitation employee, requested a stipend from her superior for an additional two days to rescore applications due to irregularities with the grant review process. Although we did not consider this disclosure of information to be significant enough to adversely affect the outcome of the 2017 Systems Change grant, the independent living deputy director stated that it would have been irresponsible to disregard the confidentiality breach and explained that it was one of the reasons she convened a second evaluation panel. By not providing evaluators training on its confidentiality procedures, Rehabilitation exposes the grant process to confidentiality breaches that are significant. Such breaches could include providing an applicant with information that could give it an unfair advantage, or leaking the results of an evaluation before Rehabilitation publishes the award.
|NAME OF GRANT AND YEAR AWARDED|
|EVALUATOR TRAINING AND INSTRUCTIONS||OIB
|Provide evaluators training and written instructions on:|
|The purpose of the grant program||♦||X||X||♦|
|Regulatory requirements of the grant||X||X||X||X|
|How to evaluate applications||♦||♦||♦||X|
Source: California State Auditor's review of Rehabilitation’s evaluator training process for four grants awarded from fiscal years 2014–15 through 2017–18, and Rehabilitation's grant manual.
♦ = Partially followed
X = Did not follow
Inadequate Evaluator Training and Instructions
Rehabilitation could not demonstrate that it provided adequate training or written instructions to its evaluators to ensure that they understood the grant evaluation process, including how to score grant applications, for the four grants we reviewed. In its grant manual, Rehabilitation states that evaluators are to receive training and instructions on the purpose of the grant, relevant regulatory requirements, and the process for evaluating and scoring applications. Further, Rehabilitation acknowledges in the grant manual that evaluators may not have sufficient financial knowledge and experience to understand financial documents submitted by an applicant.
The most egregious and frequent errors that, in part, resulted from Rehabilitation’s failure to provide evaluators with adequate training and written instructions occurred in the 2017 Systems Change grant, causing the panel to have to rescore the applications. In the first round of scoring for the grant, the three evaluators provided identical scores and similar comments for each applicant. Although the grant manual states that evaluators should discuss their assessments of each scoring component among themselves, it also states that they do not need to reach a consensus on the scores they assign to the applicant. However, Rehabilitation cannot demonstrate that it provided this instruction to the evaluators. In addition, each evaluator provided SCRS, one of the two applicants, a score of 0 for one component, which was valued at up to 15 points per evaluator, for a potential total of 45 points. The scoring benchmarks stated that evaluators should assign a score of 0 when an applicant does not answer a question or does not address any of the components of a question on the application. After the evaluation, however, the chief of independent living at the time indicated that program staff found that SCRS’ application was responsive to the component that the evaluators had given a score of 0.
When we asked two of the three evaluators how the identical scores occurred, they explained that because they were not instructed that they should not have identical scores, the group read a section of an application together, discussed the section, and agreed on a score and corresponding comment. This approach indicates that the evaluators did not conduct individual assessments as specified by the grant manual. In fact, the two evaluators indicated that program staff did not clearly explain to them that they could award points to an applicant for a component that the applicant did not address in the section for that component but did address in another section of the application. The program manager stated that after noticing these errors, program staff provided evaluators with additional verbal instruction that they provide a score of 0 only if the applicant did not address the component in any section of an application. However, the program manager did not document what was included in the additional verbal instruction, and Rehabilitation still did not provide written instructions to the evaluators before asking them to rescore the applications a second time.
In its second round of scoring, the same evaluation panel scored some components more than 3 points apart, a large enough difference that Rehabilitation grew concerned that this indicated that the evaluators were not using the same criteria to review the applications. The new scores also resulted in a different grant recipient. Specifically, without adequate instruction on how Rehabilitation expected them to score the second round, for a component with a possible score of 15 points, one evaluator provided SCRS with a score of 14 points, another provided a score of 11 points, and the remaining evaluator scored it as 1 point. The evaluator who provided SCRS with 1 point explained that she believed SCRS had not addressed the question and that a low score for this component was entirely appropriate. The chief of independent living at the time emailed the evaluators to ask them how they would resolve the misalignment in the few areas where they had more than a 3‑point difference between the highest and lowest scores. This email indicates that Rehabilitation expected the evaluators to score each component within a 3‑point differential, although it had not documented this expectation in the grant manual or, based on available documentation, provided them with this instruction before the second round of scoring.
Upon identifying the differences among the scores, Rehabilitation asked the evaluation panel to score the applications a third time to resolve those areas where they had scored the applicants more than 3 points apart. The program manager stated that she provided the evaluators with additional verbal instruction that the component scores had to be within 3 points, but again did not provide them with written instructions before they rescored the applications a third time. Without providing evaluators with sufficient training and written instructions to reference during the evaluation process, Rehabilitation cannot ensure that evaluators understand how to score applications consistently and adequately.
Failure of the Technical Review Team to Fulfill All Responsibilities
According to the grant manual, Rehabilitation must appoint a technical review team for each grant consisting of subject‑matter experts on program policy, such as the program manager of the division responsible for awarding the grant, and on the RFA process, such as the program manager of contracts. The team is responsible for providing the evaluators with orientation information; meeting with evaluators to answer program or process questions; ensuring that evaluators follow instructions; applying the tiebreaker rule if necessary; and summarizing the evaluation process and results, including an awardee recommendation, in a memorandum to the director and chief deputy. Although Rehabilitation did not always appoint technical review teams to oversee the evaluation for each grant, as shown in Table 5, we found that staff sometimes fulfilled the responsibilities of these teams. However, the grant manual does not describe procedures sufficient to guide a technical review team’s assessment of the evaluation process to ensure a consistent and fair process for all grant applicants.
|NAME OF GRANT AND YEAR AWARDED|
|TECHNICAL REVIEWS OF THE EVALUATION PROCESS||OIB
|Appoint technical review team||X||✓||♦||♦|
|Technical review team ensures that evaluators follow the evaluation process||♦||♦||♦||♦|
|Technical review team summarizes the evaluation process, resulting in a memorandum to the director and chief deputy||♦||✓||♦||X|
Source: California State Auditor's review of Rehabilitation’s technical review team selection and review process for four grants awarded from fiscal years 2014–15 through 2017–18, and Rehabilitation's grant manual.
✓ = Followed
♦ = Partially followed
X = Did not follow
Rehabilitation’s technical review teams and staff failed to adequately ensure that evaluators followed the evaluation process for each grant we reviewed, as shown in Table 5. Technical review teams and staff generally stated that they reviewed evaluators’ scores and comments. However, because these teams and staff could not demonstrate that they provided adequate training and instructions to evaluators on how to evaluate applications, as described previously, and because Rehabilitation does not have procedures to guide assessments of the evaluation process, we found that the technical review teams and staff did not adequately ensure that evaluators followed the evaluation process. For example, the evaluators’ scoring sheets for two of the grants included checklists, and in both instances the evaluators either did not consistently use the checklists to support their scores or did not use the checklists at all. In particular, an evaluator for the 2015 TBI grant gave an applicant a score of 5 out of a total of 7 points for one component by checking all of the boxes on the checklist, thus apparently indicating that the applicant addressed all of the criteria for the component, which could have warranted a score of 7. However, this evaluator did not provide comments, and therefore it is unclear why the evaluator deducted points from this component. In another instance, the evaluators gave an applicant identical scores for a component but disagreed on how much of the component the applicant addressed in their checklists, with one evaluator indicating that the component was not addressed at all and another indicating that it was fully addressed. The program manager for this grant could not explain why there was a discrepancy in how evaluators used the checklist, and could only speculate as to why this was the case. In addition, Rehabilitation included a checklist on the score sheets program staff provided to the first and second evaluation panels for the 2017 Systems Change grant. However, none of the evaluators from the first evaluation panel used the checklist to support their scores. Moreover, evaluators’ comments for two of the four grants we reviewed did not always align with their scores or the scoring criteria in the RFA. For example, for the 2017 OIB grant, evaluators noted that an applicant did not include braille as a service it provides; however, the RFA scoring criterion for that component did not require applicants to specifically provide braille services. Had the technical review teams performed a more thorough review, they could have identified and rectified these issues.
In addition, although the grant manual indicates that evaluators should provide evidence, such as comments, to support their scores, Rehabilitation did not always ensure that they did so for the four grants we reviewed. Specifically, the first evaluation panel for the 2017 OIB grant generally did not include comments to justify their scores for three of the six applications we reviewed.5 The deputy director of the Administrative Services Division, who is primarily responsible for the grant manual, stated that she did not know why Rehabilitation does not require evaluators to include comments to support their scores. We question how the technical review teams can reasonably conclude that evaluators followed the evaluation process if evaluators do not include comments to justify their scores.
Further, according to the grant manual, the technical review team should submit a memorandum summarizing the evaluation process and results to the director and chief deputy and request approval for its grant selection recommendation. After the director and chief deputy have reviewed the memorandum, the grant manual states that they have two options: accept the recommended applicant and issue the award or reject the recommended applicant and restart the grant process. Three of the four grants we reviewed did not fully follow this procedure. For example, for the 2017 Systems Change grant, the technical review team did not issue a memorandum to the director and chief deputy. According to the chief of independent living at the time, who was also a member of the Systems Change technical review team, he did not submit a memorandum because he was not instructed to create a memorandum and he expected his supervisor, the independent living deputy director, to communicate the evaluation process and results to the director and chief deputy. Because of this failure to follow the process outlined in the grant manual, and because Rehabilitation staff did not identify any errors in the evaluation process, the independent living deputy director made the decision to award the grant to CFILC, based on the recommendations of the evaluation panel but without input from the director and chief deputy. Although the grant manual states that the director and chief deputy should consider and decide whether to accept the technical review team’s recommendations to award grants, the chief deputy stated that, when she assumed her position in 2016, it was Rehabilitation's practice to have program deputy directors award grants. Therefore, she explained that she did not consider modifying the practice. However, the chief deputy was unable to provide any documentation to support that the director or chief deputy had delegated the authority to award grants. The chief deputy ultimately agreed that Rehabilitation should clarify the roles of the deputy director in the grant manual. Therefore, if Rehabilitation determines that the director and chief deputy can delegate this authority to program deputy directors, we believe Rehabilitation’s policies and procedures should reflect this.
Questionable Rescoring of Applications
Although the grant manual does not specifically allow Rehabilitation to rescore applications, we found that when program staff identified scoring inconsistencies in the evaluation of applications, Rehabilitation sometimes convened new evaluation panels to rescore applications. In cases where an RFA has omissions or errors in its disclosed scoring criteria, it may be necessary for Rehabilitation to restart the grant process to remedy those issues. However, when issues that caused the scoring inconsistencies relate to internal processes, such as errors in the benchmarks or score sheets Rehabilitation provides to evaluators, we believe it may be reasonable for Rehabilitation to rescore applications without restarting the grant process, but only if it corrects the issues that caused those inconsistencies. For example, if Rehabilitation identifies a discrepancy between the scoring criteria outlined in the benchmarks and the RFA, Rehabilitation could amend its benchmarks to align with the RFA to remedy the discrepancy. In such a case, Rehabilitation could then ask evaluators to rescore applications with the correct benchmarks as guidance.
However, for two of the four grants we reviewed, Rehabilitation chose to rescore the applications before it issued an initial grant award, without sufficiently addressing the issues that caused concern. For example, for the 2017 Systems Change grant, Rehabilitation created three separate evaluation panels for a total of five rounds of scoring, four of which occurred before the grant award and one of which occurred after an appeal of its decision to award the grant to CFILC, as illustrated in Figure 2. As we discussed previously, Rehabilitation identified inconsistencies in the first two rounds of scoring, specifically related to inadequate and inaccurate instruction provided to evaluators. However, rather than developing clear written instructions for evaluators to follow, Rehabilitation asked the same evaluators to rescore the applications again, resulting in the third round of scoring. After the third round of scoring, in which the panel scored SCRS higher than CFILC, Rehabilitation chose to convene a second evaluation panel to rescore the applications for a fourth time because, according to the review committee’s decision regarding SCRS’ first appeal, the total scores had “changed dramatically” between the second and third rounds of reviews by the first evaluation panel.
Rehabilitation Convened New Evaluation Panels to Rescore Applications for the Systems Change Grant and Deviated From Its Intended Grant Process
Source: Rehabilitation’s grant manual, the California State Auditor’s review of the Systems Change RFA, SCRS’ appeals, review committee decisions, and documentation provided by and interviews with Rehabilitation staff.
→ Indicates that Rehabilitation deviated from its intended grant process.
Once Rehabilitation established a second evaluation panel, one of the evaluators informed Rehabilitation that the scoring benchmarks the evaluators received did not align with the score sheets. Despite learning of this discrepancy, Rehabilitation did not remedy it before the second evaluation panel completed its scoring. Further, Rehabilitation could not demonstrate that it provided evaluators with direction to address the difference in the scoring criteria. Because of this failure to resolve the discrepancies between the benchmarks and score sheets, a grant review committee invalidated the evaluation and ultimately recommended that a third panel evaluate the applications, as we discuss later in the report. If Rehabilitation had provided evaluators with additional guidance on how to resolve the discrepancies, it could have resolved any confusion about the evaluation process and potentially prevented the circumstances that led to a fifth round of scoring of the applications.
The other grant for which Rehabilitation chose to rescore the applications before issuing the initial award was the 2014 OIB grant. Before it awarded this grant, Rehabilitation convened a new evaluation panel to rescore the applications. The deputy director of specialized services explained that after the first evaluation panel finished scoring the applications, she became concerned with the panel’s scoring and understanding of the OIB program and its services. Specifically, she felt that the first panel did not fully understand how to properly evaluate the grant applications because, contrary to her expectations, a disproportionately large number of agencies that had not provided services in the past received higher scores than some previous grantees for the OIB program. Because of this issue, and her concern about the impact that different awardees could have on the quality of services provided to OIB consumers, she and the program manager selected a second panel of subject‑matter experts to rescore the applications. However, we question the necessity of a second evaluation panel. As we discussed previously, Rehabilitation did not issue a solicitation for evaluators to ensure that it had a broad pool of potential candidates, or provide adequate training and written instructions to evaluators on the evaluation process. Given that it is Rehabilitation’s responsibility to select qualified evaluators and provide them with the necessary information to adequately conduct their scoring of applications, we question why Rehabilitation did not initially take the necessary steps to select and adequately prepare evaluators. If it had done so, it may have been able to avoid the circumstances that led to it selecting a second evaluation panel.
Further, one of the evaluators Rehabilitation selected for the 2014 OIB grant’s second evaluation panel was a program analyst for the OIB program who had participated in the development of the RFA and had overseen the initial evaluation panel. Specifically, we found emails indicating that she assisted in the development of the RFA and compiled the scores from the first evaluation panel into a spreadsheet. Further, she also conducted the administrative review for some applications for this grant. According to the program manager for the grant, it was not Rehabilitation’s practice to have program staff act as evaluators for grant applications, and it is generally inappropriate for program staff to do so. In this case, however, he stated that he felt comfortable with the program analyst serving on the second evaluation panel because she was very familiar with applicants who provide OIB services and is an extremely objective person. Regardless of his belief of the analyst’s ability to be objective, Rehabilitation risked introducing bias into the evaluation when it selected an individual who participated in the development of the RFA and reviewed scores from the first evaluation panel. Because the grant manual indicates that program staff are responsible for developing RFAs, we believe it is inappropriate for them to participate as evaluators for grants their program administers.
In addition, this program analyst had already participated in conducting extensive analyses of several applications and made recommendations regarding which applicants Rehabilitation should award funding to before she joined the second evaluation panel. The analyses included a review of the types of services the applicants provide and the number of consumers applicants proposed to serve. In some cases, the analysis included recommendations that Rehabilitation award funds to applicants even if they did not score the highest in the evaluation, indicating, for instance, that the applicant that scored higher had no experience or staff to provide OIB services, or that the applicant did not provide services in the area at that time. We question why Rehabilitation chose to place this individual on the second evaluation panel, given that her perspective may have influenced the evaluation and awards to certain applicants.
We also found that the evaluators from this second panel did not complete individual score sheets as specified in the grant manual, and there was not a clear separation of duties. When we spoke to the program analyst and one of the other evaluators, both generally explained that since the program analyst was the only sighted member of the evaluation panel, she read the applications to the other evaluators and recorded the panel’s scores. They both stated that the panel submitted one combined score sheet because of time constraints. However, the evaluator elaborated that the panel scored the applications together and reached a consensus on the score or averaged the evaluators’ scores to give applicants a single score. Although we appreciate Rehabilitation’s attempt to accommodate both sighted and blind evaluators, the grant manual indicates that each evaluator should use an individual score sheet, and that the evaluation coordinator should subsequently enter the individual scores into a spreadsheet. However, in this instance, the program analyst served both as an evaluator and as the coordinator who recorded the scores. Further, neither the program manager nor one of the evaluators recalled anyone other than the program analyst reading the scores to the evaluation panel to verify their accuracy. We believe that keeping these roles separate is a critical process control to ensure that the scores accurately reflect each evaluator’s assessment. Furthermore, in this case, because two of the evaluators were blind, Rehabilitation lacks assurance that the sighted evaluator appropriately and accurately recorded their scores.
To avoid bias or the perception of bias, the Legislature should enact legislation that prohibits state agencies from selecting as an evaluator of grant applications a representative, former member, or former staff of any organization or person that is applying to receive grant funding from the state agency.
To ensure consistency and fairness in the evaluation process, Rehabilitation should make sure that it accepts only complete applications submitted before the deadline, unless otherwise specified in the RFA. If the RFA specifies a hard deadline and applicants submit incomplete applications, Rehabilitation should not accept any portions of applications submitted after the deadline and should assess the penalty for incomplete applications specified in the RFA.
To help ensure that evaluators adequately protect confidential information and that the evaluation process is fair, Rehabilitation should develop standardized evaluator training for confidentiality procedures and conflicts of interest, including a discussion of bias or the appearance of bias. Rehabilitation should also ensure that the candidates receive this training and sign conflict‑of‑interest and confidentiality forms before it selects evaluators. Further, it should prohibit program staff who participate in the development of an RFA from acting as evaluators for the applications Rehabilitation receives in response to that RFA.
To increase the transparency of its selection process and to ensure that it receives the most qualified evaluators possible, Rehabilitation should issue a public solicitation for evaluators for each grant that includes a description of essential and desirable qualifications.
To ensure that evaluators have the information necessary to sufficiently and fairly assess and score applications, Rehabilitation should develop training by December 2018 that can be tailored to each grant and includes at a minimum the following topics:
- The purpose and relevant regulatory requirements for the grant.
- Instructions on how to score applications, including an applicant’s financial information, and direction that they must provide comments to support their scores.
Rehabilitation should provide this training to evaluators before allowing them to score applications.
To ensure that it provides sufficient oversight of the grant process, Rehabilitation should ensure that the technical review teams it assigns to grants provide the director and chief deputy with a memorandum summarizing the evaluation process and the evaluators’ recommended grant awardees. Rehabilitation should also designate an individual responsible for reviewing and approving the memorandum and recommended awardees before it publishes its notice of intent to award.
If it finds errors in an evaluation that merit restarting the grant process, rescoring of applications, or convening a new evaluation panel, Rehabilitation should resolve any issues before it begins the rescoring process. It should also notify applicants to ensure that they are aware of any changes to the process due to the errors. Further, it should consider promulgating regulations and amending its grant manual to permit staff to request evaluators to rescore applications or convene a new panel when it finds issues with an evaluation.
Rehabilitation Did Not Always Follow Its Appeals Process as Suggested in Its Grant Manual, and Its Review Committees Did Not Always Ensure Fair Evaluations
- Although Rehabilitation followed the appeals process set forth in state regulations, it did not always follow the appeals process contemplated in its grant manual. For example, the chief deputy did not always notify the intended grantees of appeal requests that could affect their grant awards, a notification that the grant manual provides as an option but that we believe should be required. The chief deputy stated that she adhered to state regulations when addressing appeals but was unfamiliar with the additional steps provided in the grant manual.
- The review committee responsible for reviewing appeals and reaching a final determination on the outcome did not adequately review each appeal and missed key areas, such as potential evaluator prejudice and whether scores were supported by evidence. Further, for the 2017 Systems Change grant, the review committee made recommendations to Rehabilitation upon identifying deficiencies in the grant process; however, Rehabilitation chose not to implement these recommendations and allowed some errors to persist through the subsequent evaluation and award.
- Rehabilitation received nine appeals for the four grants we reviewed, and appellants cited various reasons for their appeals, including evaluator bias, that evaluators’ scores were not justified, and that the evaluation process lacked clear and appropriate scoring criteria.
For Most Grants, Rehabilitation Generally Followed Its Process When Receiving Appeals
Rehabilitation followed its appeals process as required by regulations, and generally followed the best practices identified in its grant manual, when receiving appeals and appointing review committees for the nine appeals it received, as shown in Table 6. State regulations provide minimal direction regarding how Rehabilitation should review and process appeals, while its grant manual describes optional procedures that are significantly more detailed. State regulations require applicants to submit appeal requests within 30 days of the date Rehabilitation notified the public of its intent to award the grant. We found that all appellants submitted their appeal requests within 30 days of the date of the award notice. The grant manual gives the chief deputy the option, upon receipt of an appeal, to acknowledge receiving the appeal in writing and to notify all intended awardees that could be affected by the decision for the appeal, a procedure we find to be a best practice and that we think should be required by Rehabilitation. Although Rehabilitation’s chief deputy acknowledged all appeal requests in writing, she generally did not notify all intended awardees of the appeals. However, we found in our review that other Rehabilitation staff, such as program managers, notified most intended awardees of the appeals.
|NAME OF GRANT AND YEAR AWARDED|
|PROCESS FOR RECEIVING APPEALS||OIB
|Number of appeals||1||1||5||2|
|Rehabilitation accepts appeal requests within 30 days of the notice of intent to award, as required by state regulations||✓||✓||✓||✓|
|Chief deputy acknowledges receipt of the appeal||✓||✓||✓||✓|
|Chief deputy notifies the intended awardee of the appeal||♦||♦||♦||♦|
|Chief deputy appoints a review committee, as required by state regulations||✓||✓||✓||✓|
|Chief deputy notifies appellant of the individuals appointed to serve on the review committee and their qualifications||✓||✓||X||X|
Source: California State Auditor’s review of the appeal process for selected grants awarded from fiscal years 2014–15 through 2017–18, state regulations, and Rehabilitation’s grant manual.
✓ = Followed
♦ = Partially followed
X = Did not follow
State regulations require the chief deputy to appoint a review committee of up to three Rehabilitation employees to review each appeal. In addition, the grant manual gives the chief deputy the option to notify the appellant regarding the members appointed to serve on the review committee and their qualifications. For seven of the nine appeals, the chief deputy provided the appellant with job titles of the members of the review committee, but she did not provide information regarding the members’ qualifications as suggested by the grant manual. The chief deputy explained that she followed state regulations when administering appeals and was unfamiliar with the additional guidance in the grant manual, since she had not reviewed it and Rehabilitation had not yet adopted it. We would expect her to be familiar with the guidance Rehabilitation has made available to its staff, and she acknowledged that she should have been familiar with the grant manual, even in its draft form.
Inadequate Assessment and Response to Appeals
For each appeal, the grant manual authorizes Rehabilitation to consider adding language to the RFA requiring the review committee to review records relevant to the evaluation and scoring of the appellant’s application, including the applications of intended grantees whose grant award the appeal could affect. It also suggests that the review committee should determine whether the evaluation panel followed the requirements for evaluation described in the RFA and, if it discovers any procedural errors or omissions, to determine if the error or omission had a substantial effect on the outcome of the overall scoring. Further, the grant manual allows the review committee to determine whether there is evidence that evaluator prejudice affected the scoring process, and whether evaluators supported their scoring with evidence from the relevant applications.
Because the grant manual suggests that the review committees should determine whether the potential for evaluator prejudice existed as part of their appeal reviews, we assessed whether they performed such a review. However, given the numerous issues we described previously regarding the evaluation of applications, including the potential for perceived evaluator bias, we believe that having the review committee determine whether these issues occurred after Rehabilitation awards the grant is too late in the process. Specifically, we believe that before Rehabilitation awards the grant it should designate staff, separate from those who are responsible for developing RFAs, scoring criteria, and selecting evaluators, to conduct such reviews. This oversight of the grant process will provide Rehabilitation with additional assurance that program staff and evaluators adhered to its grant process, and that it can demonstrate the process was followed as intended.
Nevertheless, we found that the review committee did not always review all relevant applications and generally did not conduct comprehensive reviews of the evaluation and scoring process to identify procedural errors, evaluator prejudice, or whether evaluators supported their scores with evidence in the relevant applications, as shown in Table 7. If the review committee identifies any of these issues, the grant manual allows it the option of rescoring the applications affected by the appeal and delivering a decision to the chief deputy within 30 days of the date of the appeal request. Although we identified procedural errors in all four grants we reviewed, the review committee identified errors in only two of the four grants. When the review committee identified these errors, it did not rescore the applications itself. Rather, for these two grants, the review committee required Rehabilitation to convene new evaluation panels to rescore the applications. Finally, state regulations require the review committee to notify the appellant in writing of its decision within 30 days of the date of the appeal request, and the grant manual suggests that the chief deputy notify all affected parties of the final decision as soon as practical. However, the review committee and the chief deputy did not always notify appellants or other applicants. As we discussed previously, the chief deputy was unfamiliar with the additional steps described in the grant manual for notifying applicants of appeals that may affect them.
|NAME OF GRANT AND YEAR AWARDED|
|PROCESS FOR REVIEWING AND RESPONDING TO APPEALS||OIB
|Number of appeals||1||1||5||2|
|Review committee reviews evaluation and scoring of the appellant's application and the applications of the intended awardees||X||♦||♦||♦|
|Review committee adequately determines whether:|
|Procedural errors or omissions occurred in the evaluation and scoring process||X||X||♦||♦|
|Evidence exists that evaluator prejudice affected the scoring process||X||✓||♦||X|
|Scoring is supported by evidence in the relevant applications||✓||X||♦||X|
|If the review committee identified errors or omissions, it shall rescore the applications||X||X||♦||X|
|Review committee notifies appellant of its decision within 30 days of the date of the request, as required by state regulations||X||✓||♦||✓|
|Chief deputy notifies all intended awardees of the review committee's final appeal decision||♦||X||♦||♦|
Source: California State Auditor’s review of the appeals process for selected grants awarded from fiscal years 2014–15 through 2017–18, state regulations, Rehabilitation’s grant manual, and the review committees’ decisions.
✓ = Followed
♦ = Partially followed
X = Did not follow
Further, although we found that the review committees sometimes contacted appellants or program staff for additional information regarding appeals, regulations do not specify a process for committees to do so. State regulations require appellants to clearly identify all issues in dispute, include in their appeal a full statement of their position with respect to each issue, and submit pertinent facts and reasons in support of the applicant’s position, but they do not provide a process for Rehabilitation to contact an appellant or program staff to obtain additional information that it deems necessary. Although we found that in some cases the review committee identified the need for additional information and allowed appellants to submit it, appellants may not always have the opportunity to clarify or further augment the assertions in their appeals if Rehabilitation does not specifically outline a process in regulation for the review committee to obtain additional information. Further, regulations also state that a review committee’s decision is final, so if it does not give an appellant an opportunity to clarify information submitted in its appeal, a review committee may make an irrevocable decision based on insufficient information. By outlining a process in regulations for appellants to submit additional information or documentation the review committee deems necessary, Rehabilitation can better ensure that its review committees thoroughly address appellants’ concerns. If Rehabilitation does not believe it can accomplish its secondary review of additional information that appellants may submit or that program staff may provide within 30 days, it could consider extending the regulatory time frame to issue its appeal decision from 30 days to 45 days.
As discussed throughout this report, Rehabilitation had patterns of errors and inconsistencies that occurred in the grant process, resulting in at least one applicant appealing its decision on each of the grants we reviewed. Although the appeals contained varying allegations, we identified some that were common to many. For example, five of the nine appeals cited discrepancies between the scoring criteria, application content, or evaluators’ scores and comments. Additionally, seven appeals alleged evaluator bias because evaluators had connections to the grant applicants, evaluation panels lacked representation from the disability community, or evaluators provided the appellant low scores. Finally, one appellant contended twice that Rehabilitation failed to follow its policies and procedures by not issuing public solicitations for evaluators. The following subsections describe specific details of the grant process and subsequent appeals for each of the four grants we reviewed.
2017 Systems Change Grant
During two appeals of the 2017 Systems Change grant, Rehabilitation did not consistently follow the appeals process as described in the grant manual, which we identified as a best practice, and it did not rectify procedural errors before it awarded the grant. Figure 2 illustrates the various evaluation panels, rounds of scoring, and appeals for the 2017 Systems Change grant. In May 2017, SCRS filed an appeal of Rehabilitation’s decision to award the grant to CFILC, alleging that Rehabilitation failed to provide appropriate guidance to the first evaluation panel, which resulted in identical comments, blank scores, and identical scores. It also alleged that Rehabilitation failed to award the grant to SCRS after the first evaluation panel identified SCRS as the grantee, that Rehabilitation provided CFILC with preferential treatment and selected biased evaluators for its second evaluation panel, and that Rehabilitation negligently and intentionally falsified scores in order to select evaluators who would score in favor of CFILC.
Although the chief deputy appointed a review committee upon receipt of the first appeal, she did not notify SCRS of the review committee members’ qualifications, as suggested by the grant manual. Instead, one of the review committee members informed SCRS only of the names of the appointed review committee members and their respective positions. We believe it is important for Rehabilitation to inform the appellant of the review committee members’ qualifications to familiarize the appellant with those who are responsible for conducting a thorough evaluation of the appeal and possibly rescoring applications.
As a result of SCRS’ first appeal, the 2017 Systems Change grant review committee made the following four recommendations to Rehabilitation for conducting the rescoring of the applications:
- “Rehabilitation should provide adequate and accurate written guidance to Rehabilitation staff in advance to manage and coordinate the evaluation review process (e.g., timelines, panel selection, panel instructions, scoring and rescoring criteria, and determination of award recipient). This is a recommendation because [Rehabilitation] asserted that the Grant Solicitation Manual was a draft manual and had not been adopted and was not applicable to the RFA.
- “Rehabilitation should provide adequate and accurate written guidance to the panel of evaluators regarding the scoring process in advance to ensure clear communication of expected scoring protocols (e.g., scoring is to be conducted independently, evaluators will be expected to rescore if there is more than a 3 point spread or range of scores, evaluators will be providing a raw score and the coordinator will weigh the scores in accordance with the Benchmark).
- “Rehabilitation should ensure consistency in the scoring [components] amongst the Reviewer (scoring) Sheets, the Benchmark, and the RFA.
- “Rehabilitation should ensure the content contained on the Reviewer (scoring) Sheets is accurately labeled to clearly differentiate different rounds of scores, as appropriate. Rehabilitation should also ensure the reviewers’ scores are discernable (e.g., free from black‑outs and strike‑outs).”
Source: The review committee’s June 2017 decision letter to SCRS in response to its appeal of the 2017 Systems Change grant award.
In reviewing the first appeal of the 2017 Systems Change grant, the review committee found procedural errors, identified that Rehabilitation needed to rescore the applications, and recommended that Rehabilitation address the errors before it conducted the rescoring. It also found that score sheets used by the first two evaluation panels did not always align with the scoring criteria in the RFA and the scoring benchmarks. The grant manual indicates that when Rehabilitation develops an RFA, it should also create scoring benchmarks and score sheets for evaluators to use when reviewing applications against the scoring criteria outlined in the RFA. In its decision, the review committee stated that based on Rehabilitation’s decision to have the first evaluation panel rescore the applications twice after the initial scoring, it was clear that Rehabilitation found inconsistencies in the evaluators’ scoring. Further, the review committee stated that an evaluator had pointed out the discrepancies in an email to Rehabilitation during the second panel’s evaluation. The review committee concluded that Rehabilitation evidently had not remedied the discrepancies in the scoring materials before giving them to the second evaluation panel. It stated that the score sheets, scoring benchmarks, and RFA form the framework for fair evaluation of the applications, and that the discrepancies between these documents invalidated the scoring for both the first and second evaluation panels. Further, the review committee compared the score sheets with the final scores for the first and second evaluation panels and, due to crossed‑out text and the lack of a date or other identifier, was unable to discern the applicable scores submitted by the first evaluation panel for its second round of scoring versus its third round of scoring. The review committee made four recommendations to Rehabilitation in advance of convening a third evaluation panel to rescore the applications, as shown in the text box.
Although the grant manual authorizes the review committee to determine whether there was evidence that evaluator prejudice affected the scoring process, and SCRS made such allegations in its appeal, the review committee did not make a determination for these allegations, stating that the scoring discrepancies it found were sufficient to set aside the award from both the first and second evaluation panels. However, in our review of the evaluation panels for the 2017 Systems Change grant, we identified that both evaluators appointed to the second evaluation panel had previous ties to CFILC, as we discussed previously. Therefore, because the review committee did not express an opinion as to whether evidence existed to indicate evaluator prejudice, it may not have completed the comprehensive level of review suggested in the grant manual, which we found to be reasonable and a best practice. We did note, however, that in its decision letter the review committee made a recommendation that Rehabilitation provide staff with written guidance in advance on how to manage and coordinate the evaluation review process, including guidance on panel selection. The committee stated that it made this recommendation because it believed the staff could benefit from such instructions.
The third evaluation panel scored the applications yet again, with the result that CFILC was again awarded the 2017 Systems Change grant, followed by a second appeal from SCRS in December 2017. SCRS’ second appeal alleged that Rehabilitation did not publish a solicitation for evaluators according to its grant manual, failed to select unbiased evaluators, and neglected to implement the recommendations of the review committee—allegations that we found had some merit. We noted that in response to the review committee’s decision and recommendations after the first SCRS appeal, Rehabilitation did not issue additional written guidance to staff or evaluators. We also found that Rehabilitation did not amend the scoring benchmarks and score sheets used by the third evaluation panel to fully include two of the eight scoring components described in the RFA, even though the review committee identified these errors in its decision. The benchmarks and score sheets for one of these two components—Organizational Experience—omitted the evaluation of whether the applicant demonstrated its expertise in and capacity to conduct an effective community organizing campaign to support issues of common interest to persons with disabilities that are related to community‑based living and implementation of the Olmstead Decision.6 The other scoring component—Assessment of Needs—failed to include an evaluation of the applicant’s understanding of what is necessary for the future direction of the grantee in providing guidance for the Systems Change—a program to provide statewide collaboration of organizations and individuals to solicit participation, develop leaders, and build coalitions in the disability and independent living communities.
Nevertheless, the review committee for the second appeal, made up of the same Rehabilitation employees who served on the review committee for the first appeal, not only upheld the decision to award the grant to CFILC, but also stated that Rehabilitation was not required to accept all of the review committee’s previous recommendations or adhere to the grant manual. We find this decision troubling because it contradicts the review committee’s own statement in the decision regarding SCRS’ first appeal that the RFA, scoring benchmarks, and score sheets form the framework for a fair evaluation of the applications, and its recommendation that Rehabilitation should ensure that these scoring materials are consistent.
We question why the review committee upheld the decision to award the grant to CFILC rather than invalidate the scores when Rehabilitation did not implement its recommendations. The review committee made these recommendations in response to its review of the issues SCRS provided in its first appeal. Further, given the issues the recommendations addressed—such as adequate and accurate written guidance for staff and evaluators about the grant process, and consistency in scoring components, score sheets, benchmarks, and the RFA—we would expect Rehabilitation to have addressed them before it convened a third evaluation panel. Because Rehabilitation did not rectify these issues, we would have expected the review committee to stand by its previous conclusions, absent evidence showing that the review committee’s previous conclusions were incorrect. Therefore, the review committee not only undermined its previous appeal decision, it also allowed the deficiencies it identified in the process in the first appeal to persist. Further, Rehabilitation’s failure to implement the review committee’s recommendations demonstrated its disregard for ensuring a fair and transparent process. We believe the review committee should have judged the grant process to be inadequate, once again invalidated the scores, and required Rehabilitation to restart the grant process.
The grant manual suggests that, if a review committee identifies procedural errors that had a substantial effect on the outcome of the overall scoring, it should rescore the applications itself. Instead, when the review committee for SCRS’ first appeal identified such errors, it required Rehabilitation to convene a third evaluation panel to rescore the applications and, as mentioned earlier, did so without fully resolving the errors the review committee identified regarding the scoring materials. According to the members of the review committee, they did not receive any written guidance on how to conduct an appeal review. They did explain that they referenced the grant manual for informational purposes but did not adhere to its guidance because the manual was a draft. When we asked the review committee members why they did not rescore the applications themselves, they stated that they believed that subject‑matter experts knowledgeable about the grant’s services would be better suited to conduct the rescoring. Although the grant manual does not currently provide an option to convene a new panel to rescore applications, we believe it is reasonable for Rehabilitation to do so when the review committee feels that subject‑matter expertise is necessary. Therefore, we recommend that Rehabilitation revise its process to provide this option. However, regardless of whether the review committee or a new evaluation panel rescores the applications, Rehabilitation must ensure that it remedies procedural errors and other shortcomings the review committee identifies in the grant process before any rescoring takes place.
In its final decision letter in January 2018, the review committee provided an overview of the four rounds of scoring completed by the first two evaluation panels. We believe this overview mischaracterized the events leading up to SCRS’ appeals. For example, it identified that CFILC received the highest score in the first round, but failed to mention that part of the reason it scored higher was that, as Rehabilitation had identified, evaluators had incorrectly assigned a score of 0 to SCRS for one component. This error accounted for up to 45 points out of a possible total of 300, or 15 percent of the overall score. Because CFILC scored only about 37 points higher than SCRS, if evaluators had scored SCRS for this component, SCRS could have received the highest score for this round. In its decision letter, the review committee also stated that Rehabilitation discovered that, for the second round of scoring, there was more than a 3‑point difference between the highest and lowest scores given by each evaluator in four scoring components. In addition, the changes in the second round of scores were significant in that they resulted in SCRS, not CFILC, being the recipient of the award. Although the second round of scoring resulted in a new awardee, we noted that the total points awarded to each applicant did not change significantly, even though the evaluators’ rescoring correctly gave SCRS points in a category for which it had previously received 0 points. Specifically, CFILC’s total score decreased by about 6 points and SCRS’ total score increased by roughly 40 points out of a possible 300 points—with 25 of those points attributable to the component evaluators originally scored as 0. Finally, the review committee stated in its final decision letter that, given the dramatic total score changes between the second and third rounds, Rehabilitation chose to select a second evaluation panel to perform a fourth round of scoring. However, we found that from the second to third rounds, CFILC’s total score had decreased by 15 points and SCRS’ score had increased by roughly 10 points out of 300, or just about 7 percent and 5 percent, respectively. Therefore, we believe the review committee overstated the changes in scores when it referred to them as dramatic, and we question the reasonableness of Rehabilitation’s decision to convene the second evaluation panel to rescore the applications.
Further, the review committee did not fully investigate SCRS’ allegations that the evaluators for the 2017 Systems Change grant were biased. The review committee’s response to the second appeal stated that SCRS did not provide convincing evidence of any evaluator’s bias or a close personal relationship with a person who applied for the grant. However, in support of its allegation of bias, SCRS provided screen captures of the social media contacts of one of the evaluators, including evidence that she was “friends” with the former and current executive directors of CFILC and “liked” CFILC’s webpage. Further, SCRS also provided evidence that the evaluator was not “friends” with SCRS management and had not “liked” SCRS’ webpage. When we followed up with the review committee regarding its decision related to SCRS’ allegations of bias, the review committee stated that it had considered SCRS’ allegations regarding the evaluator’s social media, but that it believed SCRS’ evidence regarding this matter was insufficient to prove bias. However, we question why the review committee did not investigate the allegation, as the evidence SCRS provided creates at least the perception of bias or evaluator prejudice in the evaluation process.
2014 OIB Grant
The 2014 OIB review committee did not identify a couple of key issues that occurred during the grant process, indicating that its review could have been more thorough. In particular, during the evaluation process for the grant, the second panel of evaluators used a single, combined score sheet, contrary to the grant manual’s direction that evaluators are to score applications individually. Given that this decision was a departure from the evaluation process specified in the grant manual, and that the review committee identified that the second panel of evaluators reached a consensus for each application component, we question why the review committee did not identify this as an error. Further, as we discussed previously, Rehabilitation assigned a program analyst to the second evaluation panel who had participated in the development of the RFA and helped administer the first evaluation of applications. When we attempted to follow up with the review committee to learn why it did not identify these concerns, Rehabilitation informed us that the only member of the committee was a retired annuitant that it no longer employs.
2015 TBI Grant
We also found an error in the grant process for the 2015 TBI grant that the review committee overlooked. In its decision, the review committee indicated that its one member reviewed the evaluation process, including scoring, and laws governing conflicts of interest and concluded that there were no defects in the evaluation process, nor was there evidence of a conflict of interest. However, as described previously, we found that Rehabilitation accepted application documents for the grant after the hard deadline in the RFA from ILCSC—an applicant whose total score tied that of the Betty Clooney Foundation, the appellant. Had Rehabilitation followed its grant manual and adhered to both the hard deadline and the penalty for missing that deadline specified in the RFA, it should have disqualified ILCSC. We also found that evaluators did not consistently use their checklists to support their scores, which we believe is a procedural error. Had the review committee adequately reviewed the evaluation process and identified the errors we found, we would have expected it to reconsider the original grant award, and instead award the grant to a qualified applicant that had met the deadline.
2017 OIB Grant
Rehabilitation received five appeals for the 2017 OIB grant, the most among the four grants we reviewed, as shown in Table 7. For this grant, Rehabilitation had received applications from 23 entities—some of which applied for funding in more than one county. The appeals included a variety of allegations, such as evaluator bias, whether evaluation panel members were knowledgeable and qualified, that the evaluation process lacked clear and appropriate scoring criteria, and that evaluators’ scores were sometimes not justified. For instance, one appellant stated that Rehabilitation’s scoring criteria lacked clear and appropriate benchmarks and that when evaluators did provide comments they were cryptic and vague. The appellant also alleged that the evaluation panel lacked members with backgrounds in independent living and that statewide scoring of independent living applicants seemed to show bias against having these types of entities operating OIB grants. Another appellant questioned the evaluators’ scores on some sections of its application.
The review committee found various shortcomings in the grant process and recommended that Rehabilitation convene a new evaluation panel to rescore certain components of the affected applications. Some of the issues that the review committee identified in the grant process included inconsistencies between the scoring criteria in the RFA and the benchmarks evaluators used to score the applications, and evaluators’ comments did not always support or align with their scores. As we stated earlier, while the grant manual does not currently provide an option to convene a new evaluation panel, we believe it is reasonable for Rehabilitation to do so in certain instances. In this instance, the review committee members did not believe they had sufficient time to rescore the applications and release their decision before the 30‑day deadline set forth in regulations. Therefore, they recommended that program staff convene a second evaluation panel to rescore the applications to ensure that they issued their decision within the required 30 days. As discussed previously, Rehabilitation could consider extending the time to issue its appeal decision from 30 days to 45 days, which would allow the additional time, if necessary, for the review committee to rescore applications, rather than issuing a decision to convene a new evaluation panel solely to meet the deadline to respond to the appeal.
To ensure that it consistently and thoroughly evaluates appeals, Rehabilitation should establish the following in state regulations and its grant manual:
- Staff at the appropriate level of authority are to acknowledge all appeal requests, notify intended awardees that could be affected by the appeals, and inform the appellant of the qualifications of the review committee members. Staff at the appropriate level of authority must also notify all affected parties of the review committee’s final decision within the time frame Rehabilitation establishes in regulations.
- A process for the review committees to request additional information from appellants or program staff. To allow time for an adequate review of any additional information, Rehabilitation should consider extending the time for review committees to issue their decisions on appeals from 30 days to 45 days.
- To be able to rescore applications when necessary, the review committee members should be subject‑matter experts or, if they are not subject‑matter experts, the review committee should have the authority to recommend a new evaluation panel instead of rescoring applications itself when it identifies a reason to invalidate previous evaluations.
To ensure that Rehabilitation has appropriate oversight of its grant process and can sufficiently demonstrate that it followed the process, it should designate staff, separate from those involved in the respective grant process, to conduct a review of each grant process for procedural errors, evaluator prejudice, and whether evaluators supported their scores with evidence from the relevant applications before it awards grants.
Scope and Methodology
The Audit Committee directed the California State Auditor to evaluate Rehabilitation’s application process for the grant solicitation program. Specifically, we were directed to review Rehabilitation’s policies and procedures for its grant application and review processes to determine whether they comply with relevant laws, rules, and regulations, as well as to review a selection of grants Rehabilitation awarded to determine whether it complied with its own policies and procedures. The Systems Change grant that Rehabilitation solicited in early 2017 was of particular concern to the requesters of the audit. The requesters’ key overall concern was whether Rehabilitation’s grant process is fair and transparent. Table 8 lists the objectives that the Audit Committee approved and summarizes the methods we used to address those objectives.
|1||Review and evaluate the laws, rules, and regulations significant to the audit objectives.||We reviewed relevant federal and state laws, rules, and regulations.|
|2||Review Rehabilitation’s policies and procedures for its grant application and review processes and determine whether they comply with relevant laws, rules, and regulations.||
|3||For a selection of grants awarded during the last three fiscal years, including the 2017 Systems Change grant, assess the following:
a. Whether Rehabilitation followed applicable policies and procedures during the grant application and review processes and ensured that evaluators reviewing the applications did not have any conflicts of interest.
b. Rehabilitation’s grant application and relevant documents to determine whether there were any inconsistencies in the review process.
|4||To the extent possible, assess Rehabilitation’s review of the applications for the 2017 Systems Change grant, including the following:
a. How Rehabilitation selected its panel of evaluators and whether this process was effective and appropriate.
b. The evaluators’ consistency of assessments and comments regarding each applicant.
c. Whether any of the evaluators had a conflict of interest.
d. Its methodology of scoring applicants in each round.
|We addressed this objective by following the audit procedures we describe under Objective 3.|
|5||Determine whether a more effective process exists for awarding grants, including using staff from another agency to select the panel of evaluators.||
|6||Review and assess any other issues that are significant to the audit.||
Source: Analysis of the Audit Committee’s audit request number 2017-129, planning documents, and information and documentation identified in the table column titled Method.
We conducted this audit under the authority vested in the California State Auditor by Section 8543 et seq. of the California Government Code and according to generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives specified in the Scope and Methodology section of the report. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.
ELAINE M. HOWLE, CPA
July 12, 2018
Laura G. Kearney, Audit Principal
Lauren A. Taylor, MPP
Mary Lundeen, Senior Staff Counsel
For questions regarding the contents of this report, please contact
Margarita Fernández, Chief of Public Affairs, at 916.445.0255.
1 Several months after we began our audit, Rehabilitation posted a revised version of its grant manual on its website. For the purpose of our review, we focused on the grant manual that Rehabilitation developed around 2015 and disseminated to certain staff. Go back to text
2 The administrative procedure act defines a regulation as a rule, regulation, order, or standard of general application or the amendment, supplement, or revision of any rules, regulations, orders, or standards adopted by any state agency to implement, interpret, or make specific the law enforced or administered by it, or to govern its procedure. Go back to text
4 Although similar, the discussion here refers to Rehabilitation not providing employees involved in the grant process with training on conflicts of interest and confidentiality. The discussion in this section specifically relates to evaluators. Go back to text
6 In the Olmstead v. L.C. decision of June 1999, the U.S. Supreme Court ruled that given certain conditions, states are required to provide community‑based treatment for persons with mental disabilities. Go back to text