Report 2019-113 Recommendation 7 Responses

Report 2019-113: The University of California: Qualified Students Face an Inconsistent and Unfair Admissions System That Has Been Improperly Influenced by Relationships and Monetary Donations (Release Date: September 2020)

Recommendation #7 To: University of California

To ensure that the university maintains a fair and unbiased admissions process, the Office of the President should require each campus to take the following actions:
- By March 2021, establish acceptable levels of application reader proficiency and maintain training and monitoring programs that ensure that its readers attain and sustain those levels. In addition, it should report annually to BOARS on those efforts and on reader consistency levels, including the frequency with which reader ratings align with campus guidelines for
rating applications.
- Beginning with the academic year 2021-22 admissions cycle, require each campus that does not admit all eligible transfer applicants to ensure that two readers review all transfer
applications.
- Beginning with the academic year 2021-22 admissions cycle, ensure that the second readers cannot see the ratings of first readers for both freshman and transfer applications.

Annual Follow-Up Agency Response From September 2023

From November 2022 through January 2023, the Office of the President communicated with the State Auditor's office by phone and email to obtain additional detail on the specific documentation for each campus that the State Auditor found insufficient to fully implement this recommendation. Based on this feedback, we followed up with the campuses to update their policies and procedures to address the State Auditor's concerns.

Evaluation for transfer admission is a highly technical process that requires professional staff trained to assess coursework for admission to the major at the junior level often from multiple institutions prior to transfer. With constraints on time and staff resources, campuses have implemented quality control measures when second reads are not feasible.

The June 2023 BOARS report includes a section on each campus with information on their reader quality control outcomes.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

As noted in previous updates, some of the campuses still do not have adequate procedures for ensuring that application reviewers achieve a sufficient level of accuracy when reviewing applications for admission. Some campuses have established accuracy standards and procedures for monitoring the accuracy of readers both during and after initial training. However, other campuses lack robust processes for monitoring application reviewer accuracy throughout the admissions cycle after their initial training has ended. As a result, those campuses lack adequate assurance that their application reviewers are consistent in their review of applications for admission. Additionally, the University has declined to conduct a second review of all transfer applications.

Annual Follow-Up Agency Response From October 2022

In March and April 2022, the Office of the President communicated with the State Auditor's office by phone and email to obtain additional detail on the specific documentation for each campus that the State Auditor found insufficient to fully implement this recommendation. Based on this feedback, we followed up with the campuses to update their policies and procedures to address the State Auditor's concerns.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

All campuses have now demonstrated that second readers can no longer see the first readers rating of an application. However, some of the campuses still do not have adequate procedures for ensuring that application reviewers achieve a sufficient level of accuracy when reviewing applications for admission. For example, UC Santa Cruz, UC Merced, and UC Davis each have established accuracy standards and procedures for monitoring the accuracy of readers both during and after initial training. However, other campuses lack robust processes for monitoring application reviewer accuracy throughout the admissions cycle after their initial training has ended. As a result, those campuses lack adequate assurance that their application reviewers are consistent in their review of applications for admission.

1-Year Agency Response

Each UC campus that employs application readers has successfully met this recommendation. UCOP worked with the campuses to ensure responsive policies and procedures articulate specific expectations for the proficiency that their readers must maintain and specifically describe processes to monitor their readers to ensure they maintain that level of proficiency. UCOP also collected documentation from the campuses to demonstrate that a second reader cannot view the first reader's score.

Over 20% of transfer applications go through a technical evaluation process where one campus reviews the application for completion of minimum junior transfer admission requirements. Data, including GPA, overall UC-transferable units, and completion of the 7-course pattern, as well as course-by-course evaluations, are then shared with the other campuses to which the student applied. Each campus then conducts its own comprehensive review for selection purposes. This may include identifying major pre-requisites, calculating a major GPA, and reviewing the personal insight questions.

As with any human review process, a small number of errors are expected, and when discovered by a reviewer, they are reported to UCOP as part of this review process. The annual error rate is extremely low and accounts for less than one-tenth of one percent of applications reviewed. Even with these errors, our records indicate that none of these students were unfairly denied admission to UC. Students who did not receive offers of admission were generally not competitive for the campuses and majors to which they applied.

In response to recommendation 7b related to transfer admissions, campuses who do not use an automated (non-human) process or do not have a two reader process are independently conducting an additional quality assurance measure by randomly sampling transfer applications to ensure they have been appropriately evaluated.

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

For UC campuses that employ application readers (UCR uses a scoring system), each location has successfully met requirements 7a and 7c, and have provided documentation outlining the reader training on proficiency and standards and how these programs are monitored.

For 7b, eight out of the nine campuses either have a two-reader process in place for transfer students or if a campus doesn't have a 2-reader process and anticipates not admitting all eligible transfers, will develop a random sampling process for a 2nd reader. The random sample is an effective statistical tool that allows campuses to ensure that transfer applications are appropriately evaluated.

California State Auditor's Assessment of 6-Month Status: Pending

Although the Office of the President acknowledges that the UC has not fully implemented the recommendation, it overstates the progress that campuses have made. Half of the campuses have not articulated specific expectations for the proficiency that their readers must maintain throughout the reading process. Additionally, none of the campuses have described with specificity the processes it will use to monitor their readers to ensure they maintain that level of proficiency. For other portions of its response, we cannot substantiate the progress because the Office of the President did not provide sufficient documentation. For example, we cannot validate that each campus's second reader cannot view the first reader's score because the Office of the President did not provide documentation related to this issue for each campus.

Further, the documentation the Office of the President did submit shows it does not plan to require campuses to ensure two reads of each transfer application as we recommended. Rather it plans to allow for a random sampling of applications to be read a second time. This approach does not fully implement the recommendation.

60-Day Agency Response

The University president has issued a letter to the campus chancellors to implement this recommendation immediately and provide documentation of follow-through no later than January 15, 2021.

For 7b, the University offers an alternative implementation as follows due to resource constraints: By the fall 2022 admissions cycle, campuses that anticipate not being able to admit all eligible transfer applicants and do not have a two reader review process must develop a process to have a second read of a random sample of transfer applicants, excluding applicants guaranteed admission.

California State Auditor's Assessment of 60-Day Status: Pending

The Office of the President provided the letter it sent to the campus chancellors. We will consider this recommendation fully implemented when the Office of the President adopts formal requirements that campuses take these actions. Because conducting a second read for only a sample of transfer applicants would not afford all transfer applicants two independent reads, which could affect the fairness of the evaluation of their application, we stand by our recommendation that campuses that do not admit all eligible transfer students conduct two reads of applications from all eligible transfer students.

All Recommendations in 2019-113

Agency responses received are posted verbatim.