Report 2012-108 All Recommendation Responses

Report 2012-108: School Safety and Nondiscrimination Laws: Most Local Educational Agencies Do Not Evaluate the Effectiveness of Their Programs, and the State Should Exercise Stronger Leadership (Release Date: August 2013)

Recommendation #1 To: Fresno Unified School District

To ensure that it is effectively preventing and addressing incidents of discrimination, harassment, intimidation, and bullying in its schools, Fresno Unified should continue its efforts to implement methods to measure the effectiveness of school safety programs at both the district and school site levels.

1-Year Agency Response

The Department of Prevention and Intervention in collaboration with the Research Evaluation and Assessment department has completed a full program evaluation of the Safe and Civil Schools program at 14 focus elementary school sites across the district. Additionally, an evaluation has been completed at a middle school site. Sites were prioritized after a district wide data review.

The evaluation included the following components:

-Site Self-Assessment Implementation Survey - December 2013- February 2014

-Staff Interviews and Site Visits - March - April 2014

-Synthesized archival data - June 2014

-Completed Safe & Civil implementation summary report - July 2014

Updates have been made to the Single Plan for Student Achievement(SPSA) document which includes a section within the document for site leaders to evaluate program effectiveness. In addition, the information can be found online at http://rea.fresnounified.org/spsa-completed.cfm. A Safe & Civil Schools utilization index score was piloted with a region of Principals to assist them in the process of evaluating program effectiveness and resource planning. The index compiled the annual survey result questions from the staff and student surveys in addition to longitudinal data on student behavior, attendance and academics. All District schools completed the SPSA process in April 2014. The SPSA document was modified to include sections which address pupil engagement, social emotional needs, and school climate (including safety and connectedness).

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

The Research Evaluation and Assessment department is in the process of presenting the recommended evaluation components and school sites to conduct the full program evaluation of the Safe and Civil Schools program across the district. Next steps include the following:

Survey result document reviews December - February 2013

Interviews and site visits March - April, 2014

Synthesize archival data June 2014

Complete evaluation report July 2014

Updates have been made to the Single Plan for Student Achievement (SPSA) document which includes a section within the document for site leaders to evaluate program effectiveness. Moreover, a Safe and Civil utilization index score was piloted to a region of principals to assist them in the process to evaluating program effectiveness. The utilization index compiles the annual survey result questions from the staff and student surveys in addition to longitudinal data on students behaviors, attendance and academics. The estimated SPSA completion date is April 2014.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

The Research Evaluation and Assessment department has begun the process of preparing for a full program evaluation of the Safe and Civil Schools program across the district. The program evaluation timeline includes the following milestone dates:

- Document reviews - October 2013 - December 2013

- Interviews and site visits - November 2013 - March, 2014

- Synthesize archival data - June 2014

- Complete evaluation report - July 2014

The Research Evaluation and Assessment department has also begun to work with the State and Federal Programs department to make updates to the Single Plan for Student Achievement (SPSA) document. An additional section which will be incorporated into the SPSA is called "program effectiveness as measured by improvements to school safety". This section of the SPSA will require school sites to analyze suspension and expulsion data along with parent, staff, and student survey data. The review will provide school sites with a mechanism to evaluate data results and connect results to program effectiveness. The estimated SPSA completion date is April 2014.

California State Auditor's Assessment of 60-Day Status: Pending

The district's milestones indicate that it has started to implement the recommendations from our audit report, but will not be complete this work until July 2014.

Recommendation #2 To: Fresno Unified School District

To ensure that it is effectively preventing and addressing incidents of discrimination, harassment, intimidation, and bullying in its schools, Fresno Unified should ensure that school sites follow the complaint procedures established in its policies.

1-Year Agency Response

Fresno Unified School Climate teams from every school site were invited to attend training sessions in Spring 2014. The training modules were designed to increase knowledge of the Safe Place to Learn Act, the School Safety and Nondiscrimination Laws Audit, and how the Universal Tier 1 school interventions support the audit findings. Attendees discussed "Seth's Law", reviewed applicable board policies and the procedures for investigating incidents of Discrimination, Harassment, Intimidation and Bullying (DHIB).

The District's DHIB Student Investigation Report template and Frequently Ask Questions (FAQ) documents were completed and distributed to sites (March 2014). In August 2014, the School Desk Guide document was completed. The documents provide a summary of key elements of the California Safe Place to Learn Act, a checklist, and a report template for schools sites; ensuring adherence to DHIB policies and procedures. The District contracted with a law firm to complete an online DHIB training for staff. In Spring 2014, the District invested in additional staff to support social emotional learning in the area of school climate and bullying prevention.

The District developed an internal DHIB compliance audit process which will be used to assess school site adherence to the established DHIB complaint/investigation procedures. The ongoing audit will occur twice a year. Enhancements were made to our District wide student information system, known as Atlas. In addition to the ability to monitor incidents of DHIB; recent Atlas improvements allow the District to monitor students acts of bullying by subtype (relational, cyber, physical, verbal, and social isolation) and on the basis of sex, disability, race, color or national origin. The District is able to review site level DHIB incidents in a variety of report formats. These reports will be utilized during the District's ongoing internal DHIB audit; ensuring school site procedures are being followed.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

Fresno Unified is currently in the process of creating original online discrimination, harassment, intimidation and bullying (DHIB) training for district staff. Fresno Unified is also in the process of creating school site desk guide procedures for DHIB matters. The initial draft for the school desk guide procedures was created December 2013. Adjustments to the draft procedures are currently being made and will be finalized by March 2014. The school desk guide procedures is a detailed step-by-step guide based on the board policies and administrative regulations.

Fresno Unified is developing a process to assess adherence to site level complaint procedures established in the policies. The estimated timing for the development of the process is July 2014.

Fresno Unified is currently in the process of developing reports related to DHIB which will assist the district in ensuring that school site procedures are followed. The estimated timing of when DHIB reports will be created is July 2014.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

Fresno Unified is currently in the process of assessing the viability of hiring a vendor to provide online discrimination, harassment, intimidation and bullying (DHIB) training to staff across the district. Fresno Unified is also in the process of creating school site desk guide procedures for DHIB matters which will be completed by December 2013. The school desk guide procedures is a detailed step by step guide based on the board policies and administrative regulations.

Fresno Unified is developing a process to assess adherence to site level complaint procedures established in the policies. The estimated timing for the development of the process is January 2014.

Fresno Unified is currently in the process of developing reports related to DHIB which will assist the district in ensuring that school site procedures are being followed. The estimated timing of when DHIB reports will be created is January 2014.

California State Auditor's Assessment of 60-Day Status: Pending

The district's response includes three dates for implementing the audit report recommendations, the first of which will not be completed until December 2013. Thus, its implementation of our recommendation is currently pending.

Recommendation #3 To: Los Angeles Unified School District

To ensure that it is effectively preventing and addressing incidents of discrimination, harassment, intimidation, and bullying in its schools, Los Angeles Unified should monitor school sites to ensure that they implement school safety programs.

Annual Follow-Up Agency Response From November 2014

- The District submits Exhibit 1, samples of completed biannual Rubric of Implementation, a monitoring tool to ensure that schools implement and assess school safety programs. The Rubric of Implementation was revised to identify the school administrator and school operations coordinators who review the plan for effectiveness.

- The District submits Exhibit 2, the School Operations iSTAR Evidence of Monitoring and Resolution report. This document memorializes the process by which iSTAR bullying reports are reviewed for school site compliance with the 30-day requirement for action and updates, and the 90-day requirement for resolution. The report is shared with the Operations Coordinators at the ESC levels who support the individual schools.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

1-Year Agency Response

- Will continue to use the biannual Rubric of Implementation, a monitoring tool to ensure that schools implement school safety programs. The ROI was revised to identify the school administrator and school operations coordinators who review the plan (Exhibit 1Ar).

- Will generate monthly iSTAR reports specific to bullying incidents for distribution to the ESC leadership beginning 2014/15 school year(Exhibit 1B).

- Is instating six (6) district-wide restorative justice coordinators and 25 school-site restorative justice advisers to support positive school climates. The restorative justice coordinators and advisers report to the central office.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

Los Angeles Unified did not respond to our requests for additional information to substantiate its status of "fully implemented."

6-Month Agency Response

The Rubric of Implementation (Exhibit 1A) is a guided assessment tool used to assess a schools level of implementation of the Discipline Foundation Policy. The Administrator of Operations assesses the school biannually and utilizes the Rubric of Implementation as a monitoring tool to determine the extent to which School Safety Plans are being implemented. Specifically, the school is scored on eight measures that give an in-depth analysis as to the extent to which the school safety is ensured. The Rubric of Implementation is an electronic document, allowing increased transparency and consistent data monitoring at the school, ESC and LEA levels.

Will continue to utilize the Rubric of Implementation as a monitoring tool to ensure that schools implement school safety programs.

Will generate quarterly iSTAR reports specific to bullying incidents and provide them to the ESC leadership. These reports will assist the ESC staff with determining the effectiveness of the schools efforts to prevent and/or respond appropriately to bullying incidents.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

Revise Safe School Plan Volume 1 to require schools to provide more detailed assessment of the schools' anti-bullying

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #4 To: Los Angeles Unified School District

To ensure that it is effectively preventing and addressing incidents of discrimination, harassment, intimidation, and bullying in its schools, Los Angeles Unified should measure the effectiveness of its school safety programs at both the district and school site levels.

Annual Follow-Up Agency Response From November 2014

- The District submits Exhibit 3, the revised Bullying and Hazing Policy, to denote 30-day requirements for updates and 90-day requirements for determination of resolution.

- The District submits Exhibit 4, the completed analysis of the bullying prompts on the 2013/14 School Experience Survey. The School Experience Survey queries staff, students, and parents on numerous measures associated with school satisfaction and safety. The 2013/14 version added nine prompts related to bullying. A cluster analysis identifies the prompts that, when grouped, provide a score for a given variable. The data can be disaggregated to the individual school site to inform site administrators, ESC and central office as to the stakeholders' rating of the efficacy of their school's bullying prevention and intervention practices.

- The District submits Exhibit 5, signed copies of Administrators Certification Form of the Safe School Plan for the 2014/15 school year. School administrators electronically certify that they analyze and monitor the efficacy of their Safe School Programs.

- The District submits Exhibit 6, samples of the Bullying Complaint Log. The log is completed by the school site's Bullying Complaint Manager to monitor critical information (dates of report, closure, monitoring, notification to complainant).

- The District submits Exhibit 7, sample iSTAR incident reports, illustrating school compliance with investigating, documenting and resolving incidents of bullying.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

1-Year Agency Response

- Has revised the Safe School Plan, Volume 1 for the 2014/15 school year to require that all reports of bullying are investigated and documented, and that appropriate interventions are implemented and monitored in a timely manner (Exhibits 2O and 2Ar).

- Has revised the Administrators Certification form of the Safe School Plan for the 2014/15 school year to require that school administrators certify that they analyze and monitor the efficacy of their Safe School Programs (Exhibit 2Br).

- Has drafted a revision of the Bullying and Hazing Policy to require that schools analyze and monitor the efficacy of their bullying prevention and intervention efforts (Exhibit 2Jr).

- The bullying section of the School Experience Survey of students, staff and parents was expanded to ten prompts (evidence provided at the 6-month submission). The analysis of the Survey will be released early Fall.

- Is continuing to use the bullying complaint log by the Bullying Complaint manager to maintain critical information (dates of report, closure, monitoring, notification to complainant) (provided at the 6-month submission).

- Has launched the MiSiS student data base which will provided greater detail about student-related behaviors. (Exhibit 2N).

California State Auditor's Assessment of 1-Year Status: Partially Implemented

Although Los Angeles Unified created tools to assist in measuring the effectiveness of its school safety programs at both the district and school site levels, it still needs to evaluate the information it obtains. Also, Los Angeles Unified did not respond to our request for additional information to substantiate its status of "fully implemented."

6-Month Agency Response

Is revising the Safe School Plan, Volume 1 for the 2014/15 school year to require a more comprehensive accounting of the schools efforts to monitor the effectiveness of their programs (Exhibit 2A).

Is revising the Administrators Certification form of the Safe School Plan for the 2014/15 school year to require that schools analyze and monitor the efficacy of their Safe School Programs (Exhibit 2B).

Has analyzed the 2012/13 School Experience Survey to gain baseline data for the effectiveness of bullying prevention and response programs (Exhibit 2C, School Experience Survey, School Climate Analysis 2012/13).

Has added seven (7) items to the 2013/14 School Experience Survey to gain a deeper understanding of how students, staff and parents view the extent of and their schools responsiveness to bullying. The survey is open for responses for approximately 6 weeks in the spring. We anticipate that this will give important assessment data to schools (Exhibit 2L, Revised Prompts for School Experience Survey, 2013/14).

Is revising the Bullying and Hazing Policy (Exhibit 2J) to

Provide specific guidance on monitoring and evaluation

Align with the revised Discipline Foundation Policys requirements for monitoring and evaluating the effectiveness of the school safety programs (Exhibit 2I)

Create a bullying complaint log for the use by the Bullying Complaint Manager to maintain critical information (dates of report, closure, monitoring, and notification to complainant) (Exhibit 2K)

Require that incidents of bullying be dually entered into the appropriate searchable data bases (iSTAR or LAUSDMAX) (Exhibit 2J Bullying Complaint Log and Exhibit 2L Bullying Complaint Form)

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

Analyze the 2012/13 School Experience Survey to gain a baseline data for the effectiveness of bullying prevention and response programs

Revise the Bullying Complaint protocol to require that incidents of bullying are entered into searchable data bases (e.g., Excel, iSTAR, LAUSDMAX). The data will be reviewed regularly to determine the effectiveness of school safety programs to prevent and address bullying.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #5 To: Los Angeles Unified School District

To ensure that it is effectively preventing and addressing incidents of discrimination, harassment, intimidation, and bullying in its schools, Los Angeles Unified should ensure that school sites evaluate the effectiveness of the programs they choose to implement.

Annual Follow-Up Agency Response From August 2015

5A. RESPONSE: The 30-day timeline for reviewing bullying complaints is memorialized into multiple documents:

- BUL-5212.2 Bullying and Hazing Policy (Exhibit 5a1)

o Section VIII, Responding to Bullying and Hazing (page 8)

o Section X, Resolution of Incidents of Bullying (page 10)

- Bullying Complaint Response Worksheet, Attachment C (Exhibit 5a2)

- School Safety Plan, Section 4.3.1 Bullying and Hazing (Exhibit 5a3)

- The Administrators Online Certification System directs the respondents to BUL-5212.2 timeline for review and resolution (Exhibit 5a4)

5B RESPONSE: The Rubric of Implementation (ROI) is a Discipline Foundation Policy monitoring tool. The operations coordinator, director and the principal review evidence and collaboratively rate the school's implementation of the School-Wide Positive Behavior on a Likert scale (Exhibit 5b1). The composite score classifies the school as Fully Implemented, Partially Implemented or Targeted. Targeted Schools receive additional support from the local district and the School Operations. At the Spring 2015 submission, 543 schools (69.3%) were "Fully Implemented," 241 schools (30.7%) were "Partially Implemented," and 0 schools were "Targeted." (see tabs "CHART" and "Spring 14-15," Exhibit 5b2).

5C RESPONSE: The Safe School Plan is a three-volume annual assessment in which principals rate their school's progress towards implementation of 63 overarching goals on a Likert scale. Two revisions were made to Goal 4.3.1, Bullying and Hazing for 2015/16 Safe School Plan pertinent to this audit: (Exhibit 5a2):

Safe School Plan, Volume 1, Section 4.3.1, Bullying & Hazing Policy— The school and all personnel shall take proactive measures to promote respect and acceptance among students, staff, and the school community.

- Administrators review the School Experience Survey data to inform bullying prevention efforts.

- Reported incidents must within 30 days; resolution must be achieved within 60 days.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From November 2014

- The District has two external protocols by which the schools' evaluative processes are monitored:

o The District submits Exhibit 1, samples of completed biannual Rubric of Implementation (ROI). The Rubric of Implementation is a tool by which the local ESC School Operations Coordinators evaluate the effectiveness of the school's safety programs on a 4-point Likert scale across eight key features.

o The District submits Exhibit 4, the completed analysis of the bullying prompts on the 2013/14 School Experience Survey. The School Experience Survey queries staff, students, and parents on measures associated with school satisfaction and safety. The 2013/14 version added nine prompts related to bullying. The data are disaggregated to the individual school site to inform site administrators, ESC and central office as to the stakeholders' rating of the efficacy of their school's bullying prevention and intervention practices. These data are used in conjunction with iSTAR reports and bullying complaints to determine the effectiveness of school safety efforts on the Rubric of Implementation.

- The District submits Exhibit 3, the revised Bullying and Hazing Policy which denotes 30-day requirements for updates and 90-day requirements for determination of resolution. The Bullying policy directs schools to measure the effectiveness of their programs.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Although Los Angeles Unified created tools to assist school sites to evaluate the effectiveness of the programs they choose to implement, it did not substantiate that schools are using those tools.

1-Year Agency Response

- Will continue to use the biannual Rubric of Implementation, a monitoring tool to ensure that schools implement school safety programs. The ROI was revised to identify the school administrator and school operations coordinator who review the plan for effectiveness (Exhibit 1Ar).

- Has drafted a revision of the Bullying and Hazing policy to require that schools monitor, evaluate and modify their bullying prevention and intervention efforts to promote positive school climates. The bulletin also provides metrics to consider in the evaluations (Exhibit 2J).

California State Auditor's Assessment of 1-Year Status: Partially Implemented

Los Angeles Unified has created tools to assist school sites evaluate the effectiveness of the programs they choose to implement. However, Los Angeles Unified has not provided support to show school sites are effectively using the tools.

6-Month Agency Response

Schools are expected to review all relevant forms of data, including school-based data (office referrals, incident reports, suspension rates, attendance rates, bullying complaints, graffiti, and other markers of school engagement) to determine the efficacy of their efforts. Schools describe their efforts of implementation on the annual Safe School Plan, which is filed centrally and reviewed at the ESC level.

Educational Service Center (ESC) Responsibilities

Each ESC Instructional Superintendent and Administrator of Operations is responsible for teaching, enforcing, advocating and modeling the Discipline Foundation Policy (Exhibit 2I) to ESC staff and school administrators. The Administrator of Operations will designate a team of ESC staff who will ensure the successful implementation of this policy through support and guidance. In addition to providing assistance, the ESC Operations staff, along with the Instructional Director, are responsible for:

Analyzing data, monitoring and supporting school practices in order to address situations where practices need to be strengthened.

Regularly reviewing the online Rubric of Implementation (Exhibit 1A) with the site administration and providing targeted assistance to schools as needed.

Supporting and assessing school practices in order to make suggestions to address situations where practices need to be modified. School support should include a regular review of the goals listed in the Safe School Plan.

Ensuring that all deans participate in mandatory District training on discipline and alternatives to suspension.

To comply with the California State Auditors Recommendation 3, the District:

Requires that the ESC Administrators of Operations work closely with their schools to ensure that schools design and implement practices that address the specific needs of that school community, and that those practices are regularly monitored, evaluated, and revised as necessary.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

Revise the complaint log to request critical information (dates of report, closure, monitoring).

Work with ITD to facilitate school site personnel's ability to generate reports on bullying incidents that were documented in iSTAR into a searchable data set, such as an Excel file. (Currently, only central office has the capacity to generate reports.)

-Require that schools use reports in searchable complaint databases (e.g. iSTAR, LAUSD MAX, Excel) to facilitate identification of patterns.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #6 To: Los Angeles Unified School District

To ensure that it is effectively preventing and addressing incidents of discrimination, harassment, intimidation, and bullying in its schools, Los Angeles Unified should resolve complaints within 60 calendar days regardless of the complaint process selected.

Annual Follow-Up Agency Response From August 2015

RECOMMENDATION #6 management disagrees that 90-days is an appropriate timeline. It is the opinion of the office at [sic] all complaints should be resolved in 60-calendar days. Also I looked at your UCP bulletin 5159.3, p. 9 and it still says 60 days it does not say 60 calendar days per regulations. Next year when you respond to this recommendation we hope to see the change to the UCP bulletin to confirm to regulations.

Recommendation 6 has two components:

a. Complaints of bullying should be resolved in 60 calendar days.

b. The UCP bulletin should define "day" as "calendar day."

RECOMMENDATION 6A. Complaints of bullying should be resolved in 60 days.

6A RESPONSE. The District has memorialized the 60-day timeline for resolving bullying complaints in multiple documents:

- BUL-5212.2 Bullying and Hazing Policy, November 26, 2014 (Exhibit 5a1)

o Section VIII K. Responding to Bullying and Hazing (page 8)

o Section X, Resolution of Incidents of Bullying (page 10)

- Bullying Complaint Response Worksheet Attachment C (Exhibit 5a2)

- BUL-5212.2 Bullying and Hazing Policy Attachment M, Title IX/Bullying Complaint Manager Log (Exhibit 6a1)

- School Safety Plan, Volume 1, Section 4.3.1 Bullying and Hazing (Exhibit 5a3)

- The Administrators Online Certification System directs the respondents to the timeline for reviewing and resolving incidents of bullying delineated in BUL-5212.2 Bullying and Hazing timeline and the Safe School Plan, Volume 1, Section 4.3.1 (Exhibit 5a4)

RECOMMENDATION 6B. The UCP bulletin should define "day" as "calendar day."

5B. RESPONSE: The Uniform Complaint Procedures bulletins define days as "Calendar days, unless designated otherwise" in Section III Related Definitions, G: Days (page 6):

- BUL-5159.3, May 15, 2014 (Exhibit 6b1)

- BUL-5159.4, November 14, 2014 (Exhibit 6b2)

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

LAUSD has clarified in both its UCP bulletin and Bullying and Hazing Policy bulletin that all complaints should be reviewed within 60 calendar days. Therefore, we have marked this recommendation as Fully Implemented.

Annual Follow-Up Agency Response From November 2014

- Auditor Rosa Reyes approved a 90-day timeline. The District submits Exhibit 3, the revised Bullying and Hazing Policy which denotes 30-day requirements for updates and 90-day requirements for determination of resolution.

- The District submits Exhibit 2, the School Operations iSTAR Evidence of Monitoring and Resolution report. This document memorializes the process by which iSTAR bullying reports are reviewed for school site compliance with the 30-day requirement for action and updates, and the 90-day requirement for resolution. The report is shared with the Operations Coordinators at the ESC levels who support the individual schools.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

We believe that 60 days, which was Los Angeles Unified's policy at the time of our audit, is a more appropriate timeline to resolve bullying complaints rather than 90 days. Further, the Uniform Compliant Process bulletin still states 60 days and does not clarify 60 calendar days, as required in regulations.

1-Year Agency Response

- Has launched the MiSiS system with an incident reminder function to follow up on critical incidents (Exhibit 2N).

- Has drafted a revision of the Bullying and Hazing policy to require that there be at least one update within 30 school-days of a reported bullying incident to document the status of the investigation, actions taken, and need for on-going monitoring (Exhibit 2Jr).

- Will generate monthly iSTAR reports of bullying incidents identifying the status of the incident, and the number of actions and updates since the initial filing beginning 2014/15 (Exhibit 1B)

California State Auditor's Assessment of 1-Year Status: Partially Implemented

Los Angeles Unified did not respond to our request for additional information to substantiate its status of "fully implemented."

6-Month Agency Response

Currently, the state does not mandate response times for bullying complaints, but does require investigations of UCP complaints within 60 calendar days, unless this time period is extended by written agreement of the complainant pursuant to 5 CCR 4631(a). With respect to UCP complaints, the LAUSD provided evidence of completion of the Auditors recommendations, dated October 18, 2013. That aspect of Recommendation 4 is considered satisfied.

Though bullying complaints do not fall under the same mandates as UCPs, the District has instituted guidelines to require timely responses to allegations of bullying. Schools are expected to take all reported cases of bullying seriously, investigate promptly and thoroughly, document the actions taken, and monitor and follow up to ensure that the bullying has stopped. The iSTAR system allows for ongoing updates to the status of an incident..

the District:

Has revised the 25-day notice on dormant iSTAR incidents to alert administrators to document the status of investigation, actions taken, and need for ongoing monitoring of bullying incidents (Exhibit 4A).

Is revising its Bullying and Hazing Policy to

o Instate a timeline for response that is reasonable and similar to those requested by state regulations for incidents of discrimination and harassment. The revised Bullying and Hazing Policy will require that there be at least one update within 30 school-days after an initial filing to document status of investigation, actions taken, and need for ongoing monitoring (Exhibit 2J), and

o To direct schools to ensure that the complainant be advised of the findings of an investigation of an alleged incident of bullying. A sample Letter of Findings has been provided as an attachment to the revised Bullying and Hazing Policy as a formal communication with the complainant. A copy of the letter will be provided to the Title IX Bullying Complaint Manager (Exhibit 2J, and Exhibit 4B, Sample Letter of Findings)

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

-Revise its policies on responding to bullying and instate a timeline for response that is reasonable and similar to those requested by state regulations for incidents of discrimination and harassment.

-Create an automated response for open cases of bullying that have been dormant, reminding them to update the iSTAR report.

With respect to UCP complaints, to comply with the California State Auditor's recommendations, the District has*:

-Reviewed its UCP Bulletin BUL-5159.2 and trained staff on the required 60-day timelines for responding to UCPs.

-Revised its UCP complaint logs to allow improved monitoring of timeline compliance.

*evidence sent electronically

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #7 To: Los Angeles Unified School District

To ensure that it is effectively preventing and addressing incidents of discrimination, harassment, intimidation, and bullying in its schools, Los Angeles Unified should ensure that school sites follow the complaint procedures established in its policies.

1-Year Agency Response

- Has drafted a revision of the Bullying and Hazing Policy to require that there be at least one update within 30 school-days after an initial filing to document status of investigation, actions taken, and need for ongoing monitoring. The revised Policy includes sample communication to complainants (Exhibit 2Jr).

- Will generate monthly iSTAR reports of bullying incidents identifying the status of the incident, and the number of actions and updates since the initial filing beginning 2014/15 (Exhibit 1B).

- Is continuing to use the 25-day notification of iSTAR incidents to alert administrators to document the status of investigations, actions taken, and need for ongoing monitoring of bullying incidents (evidence provided at 6-month submission).

California State Auditor's Assessment of 1-Year Status: Fully Implemented

Los Angeles Unified has created tools to monitor school sites to help ensure they are following complaint procedures in place per policy. Specifically, it provided support of the 25-day notification of iSTAR incidents to alert administrators of status of investigations, actions taken, and need for ongoing monitoring of discrimination, harassment, intimidation, and bullying incidents.

6-Month Agency Response

Incidents of bullying that culminate in suspension are reviewed monthly to determine whether additional supports are recommended (Exhibit 2I).

The ESC is charged with assisting schools with resolving complaints from all stakeholders regarding the implementation of School Wide Positive Behavior Support and Intervention (Exhibit 2I Discipline Foundation Policy). As such, the ESC closely monitors the schools responses to bullying.

The Title IX Bullying Complaint Log maintains critical information (dates of report, closure, monitoring, notification to complainant) that can indicate the schools response time to incidents (Exhibit 5A, Bullying Complaint Log). The revised Bullying Complaint Form allows cross-referencing of incidents that are also notated in iSTAR or LAUSDMAX (Exhibit 5B).

The iSTAR system generates an alert after 25 days of inactivity on an open incident to remind administrators to document the status of investigation, actions taken, and need for ongoing monitoring of bullying incidents (Exhibit 4A). School Operation administrators at the ESC and Central Office receive bi-weekly notifications of dormant incidents as a means of monitoring responsiveness.

To comply with the California State Auditors Recommendation 5, the District:

Is requiring that Operations Coordinators review the Bullying Complaint Log during their school site visits with the administrative team (Exhibit 5A, sample Bullying Complaint Log, Attachment to Revised Bullying and Hazing Bulletin), effective July 1, 2014.

Is revising the Bullying Policy to require that there be at least one update within 30 school-days to document status of investigation, actions taken, and need for ongoing monitoring (Exhibit 2J).

Has automated the iSTAR database systems to generate notifications to update the status of open cases of bullying that have been inactive for 25 days (Exhibit 4A).

Will generate and review quarterly iSTAR reports specific to bullying

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

-Thoroughly review the bullying complaint policy and protocol to add time guidelines for responding to and monitoring allegations of bullying and develop communiques to parents

-Automate the data base systems to generate notifications to update the status of open cases of bullying that have been inactive.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #8 To: Sacramento City Unified School District

To ensure that it is effectively preventing and addressing incidents of discrimination, harassment, intimidation, and bullying in its schools, Sacramento City Unified should ensure that school site staff complete the training required under its anti-bullying policy.

Annual Follow-Up Agency Response From October 2015

During the 14/15 school year, the District has continued its training of school administrators, district staff, after school program providers, and staff on all aspects of bullying prevention and intervention.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

1-Year Agency Response

The district has provided training to all site administrators, except the ones who have been newly hired. There was a site administrator training provided on 8/14/14 and 34 site administrators participated. The Board policy requires that site administrators "develop an annual process for discussing the school district policy on harassment and bullying with students and staff." We are working with site administrators to ensure that they complete this requirement. In the 2013/14 school year, 15 extensive staff trainings were provided. For the 14/15 school year, there have already been 10 school staff trainings scheduled in the upcoming months. The staff training power point has been made available to all site administrators who have been trained to train their school staff.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

The district office is providing training to all site administrators. About 99% have been trained. An additional training is scheduled in March for the remaining few. The Board policy requires that site administrators "develop an annual process for discussing the school district policy on harassment and bullying with students and staff." We are working with site administrators to ensure that they complete this requirement. In the 2013/14 school year, 14 extensive staff trainings have already been provided with several more scheduled in the upcoming months. The staff training power point has been made available to all site administrators who have been trained to train their school staff.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

The district office is providing training to all site administrators. The Board policy requires that site administrators "develop an annual process for discussing the school district policy on harassment and bullying with students and staff." We are working with site administrators to ensure that they complete this requirement.

California State Auditor's Assessment of 60-Day Status: Pending

The district stated that it is currently working with its site administrators and plans on implementing this recommendation by 5/30/2014.

Recommendation #9 To: Sacramento City Unified School District

To ensure that it is effectively preventing and addressing incidents of discrimination, harassment, intimidation, and bullying in its schools, Sacramento City Unified should continue its efforts to implement school safety programs at school sites.

60-Day Agency Response

SCUSD is continuing its efforts to implement school safety programs at school sites. Because this effort continues as stated in the recommendation, it may be considered as fully implemented. However, since this work requires ongoing implementation and adjusting, the work continues.

California State Auditor's Assessment of 60-Day Status: Fully Implemented

Recommendation #10 To: Sacramento City Unified School District

To ensure that it is effectively preventing and addressing incidents of discrimination, harassment, intimidation, and bullying in its schools, Sacramento City Unified should measure the effectiveness of its school safety programs at both the district and school site levels.

Annual Follow-Up Agency Response From November 2016

During 2015/16, the Bullying Prevention Specialist (BPS) tracked the number of reports of suspected bullying submitted (72 total), number of consultations that were provided to students, parents, and staff (132 total), number of mediations provided (29 total), and all the training provided (25 trainings), including evaluations.

The BPS worked collaboratively with the Social Emotional Learning (SEL) initiative, essentially integrating bullying prevention and SEL work together. A district wide survey was given to 5th grade students through our Social and Emotional Learning initiative (see SEL data). Students were surveyed regarding their safety, connectedness to school, and the six social and emotional skills (see attached SEL learning standards).

In partnership with Sacramento County Office of Education (SCOE), a school climate and bullying prevention survey is given each year to continue to track the progress of our original demonstration sites. The district will continue working with SCOE to utilize the data and integrate the assessment further into district practices.

The BPS partnered with the District Attorney's office to provide 45 - 60 minute #iSMART social media class presentations to a total of 438 Hiram Johnson 9th/10th graders and 313 Will C Wood 7th graders. The students completed a post-test that assessed their knowledge of definitions and what they learned. As an observer of several of these presentations and someone who read the post-surveys, the students were unaware of some of the definitions before the presentation; however, they were able to convey the definitions (ie. "sextortion") after the presentation.

The BPS also collaborates with the Sacramento Children's Home to provide an evidenced based 10 week "Stop and Think" curriculum to 3 - 5th graders and a 10-week "Too Good for Violence" curriculum for 6 - 12th graders at select schools. Both of these curricula come with a pre and post-test (sew eVIBE data report).

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From October 2015

During 2014-2015 school year two measures were taken. A district wide survey was given to all 5th grade students through our Social and Emotional Learning initiative. Students were surveyed regarding their safety, connectedness to school, and social and emotional skills. In partnership with Sacramento County Office of Education (SCOE) a school climate and bullying prevention survey was given at two schools sites. The district will continue working with SCOE to refine this assessment and integrate assessment further into district practices.

- 5th grade district wide climate and culture survey (inclusive of school safety and bullying) completed in 2013-2014 and 2014-2015. Results indicate an improvement in "Sense of Belonging-School Connectedness" and an improvement in social and emotional skills (see attached 5th Grade Climate/Culture/SEL) (B)

- Two school sites, Pacific Elementary and Bowling Green McCoy Elementary, participated in the SCOE student survey. Rating scale was 1-4, with 1 being low or negative and 4 being positive or high. Prevalence of Bullying mean score was a 2.3 and Perceived Safety of School was a mean score of 3.1. (see attached of scores).

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Beyond the two school site surveys mentioned in its response, Sacramento City Unified did not respond to our request for additional information regarding its efforts to implement our recommendation at all school sites within the district.

1-Year Agency Response

One way we will measure the effectiveness of school safety programs is to look at the number of reports of suspected bullying forms received and the number of parent and staff trainings that were provided and compare the data between the school years of 2012/13 and 2013/14.

In 2012/13, there were 67 reports of suspected bullying submitted; but in 13/14, there were only 53 reports received.

In the 2012/13 period, there were 13 parent and staff trainings, but in the 2013/14 period, there were 27 trainings provided.

In addition, select schools implemented a 10-week eVibe Stop and Think and Too Good for Violence anti-bullying curriculum. Included in this program was a pre and post test to measure the effectiveness of the program. We are waiting to receive that data.

California State Auditor's Assessment of 1-Year Status: Pending

Sacramento Unified provided examples of some metrics that were used to demonstrate the drop in bullying incidents; however, it did not provide any evidence to tie these results to any of its school safety programs. In addition, Sacramento Unified stated that it waiting to receive data to measure the effectiveness of one of its school safety programs. It expects to implement this recommendation in December 2014.

6-Month Agency Response

One way we will measure the effectiveness of school safety programs is to look at the number of reports of suspected bullying forms received and the number of parent and staff trainings that were provided and compare the data between September 2012 - February 2013 and September 2013 - February 2014.

From September 2012 - February 2013, there were 28 reports of suspected bullying submitted; but from September 2013 - February 2014, there were only 20 reports received.

In the 2012/13 period, there were 9 parent and staff trainings, but in the 2013/14 period, there were 18 trainings provided.

In addition, select schools have implemented a 10-week eVibe anti-bullying curriculum. Included in this program is a pre and post test to measure the effectiveness of the program.

California State Auditor's Assessment of 6-Month Status: No Action Taken

Sacramento City Unified's response provides data but does not indicate any analysis of that data to link its efforts to the level of incidents of discrimination, harassment, intimidation, and bullying that have occurred. As of August 20, 2014, Sacramento City Unified has not responded to our requests for the documentation that would support the analysis it asserts to have performed.

60-Day Agency Response

The most effective way to measure the effectiveness of school safety programs is to measure school climate at all SCUSD schools through the California Healthy Kids Survey. In order to make this useful though, participation rates need to be fairly high. We will be working this year to increase participation in this survey in order to better measure the effectiveness of our school safety programs.

California State Auditor's Assessment of 60-Day Status: Pending

The district stated that it is currently working to increase participation in its California Healthy Kids Survey to better measure the effectiveness of its school safety programs. It plans on implementing this recommendation by 5/30/2014.

Recommendation #11 To: Sacramento City Unified School District

To ensure that it is effectively preventing and addressing incidents of discrimination, harassment, intimidation, and bullying in its schools, Sacramento City Unified should ensure the impartial resolution of complaints by not assigning the investigation to site administrators or other staff specifically named in the complaint.

1-Year Agency Response

Complaints simultaneously filed against school staff AND school site administrator generates two complaint investigations instead of one. Specifically, the school administrator is the investigator of record for the school staff related complaint and the site administrator's supervisor, the Area Superintendent, is the investigator of the allegations against the school administrator.

Additionally, the district has modified its complaint log database to include target dates (30-day and 45-day) within the 60-day investigation period to contact or "check-in" with the designated investigation(s) responsible for resolving the case. These periodic check-in's are avenues to determine current status of complaint investigations and provides an opportunity to offer assistance to the investigator to ensure compliance established deadlines.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

The district failed to update its response in the recommendations database.

California State Auditor's Assessment of 6-Month Status: No Action Taken

Despite our requests, Sacramento City Unified failed to update its response in the recommendations database.

60-Day Agency Response

The District has implemented a changed policy as follows:

Complaints simultaneously filed against school site staff AND school site administrator generates two complaint investigations instead of one. Specifically, the school administrator is the investigator of record for the school staff complaint and the Area Superintendent assigned to the school is the investigator record for the allegations against the school administrator.

California State Auditor's Assessment of 60-Day Status: Pending

The district did not provide documentation to support its fully implemented status. It has agreed to provide these documents as part of its 6-month response.

Recommendation #12 To: Sacramento City Unified School District

To ensure that it is effectively preventing and addressing incidents of discrimination, harassment, intimidation, and bullying in its schools, Sacramento City Unified should notify all complainants of the right to appeal its decisions to Education.

60-Day Agency Response

The District's complaint acknowledgement letter has been modified to inform complainants of their right to appeal decisions to the Department of Education.

California State Auditor's Assessment of 60-Day Status: Fully Implemented

The district has fully implemented this recommendation by updating its acknowledgement letter template, which now contains the following paragraph:

"Additionally, please be advised that in the event that you are dissatisfied with the District's decision/outcome of your complaint, you may appeal in writing to the California Department of Education within 15 days of receiving the district's decision. For good cause, the Superintendent of Public Instruction may grant an extension for filing appeals. (CCR 4652) For your reference, the California Department of Education website is www.cde.ca.gov"

Recommendation #13 To: Sacramento City Unified School District

To ensure that it is effectively preventing and addressing incidents of discrimination, harassment, intimidation, and bullying in its schools, Sacramento City Unified should update its policies and procedures to calculate the state-mandated time limit for resolving complaints in accordance with state regulations.

Annual Follow-Up Agency Response From October 2015

Effective March 5, 2015, the District updated Board Policy/Administrative Regulation 1312.3 to reflect the state-mandated time limit for resolving complaints. See District website at:

www.scusd.edu/uniform-complaint-procedure.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

.

1-Year Agency Response

The District will update current board policies and administrative regulations to reflect the state-mandated time limits for resolving complaints in accordance with state regulations.

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

The District will modify current policies to ensure state-mandated time limits of for resolution are in accordance with state regulations.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

The District will review current policies and modify as needed to reflect complaint resolution in accordance with state regulations.

California State Auditor's Assessment of 60-Day Status: Pending

The district's response indicates that it is in the process of reviewing its current policies to implement our recommendation.

Recommendation #14 To: Sacramento City Unified School District

To ensure that it is effectively preventing and addressing incidents of discrimination, harassment, intimidation, and bullying in its schools, Sacramento City Unified should resolve complaints within 60 calendar days regardless of the complaint process selected.

1-Year Agency Response

The District has modified its general complaint acknowledgement letter to reflect the 60-day resolution guideline. The District has also modified its complaint log database to include target dates within the 60-day investigation period to contact or check in with the designated investigator of the case.

These periodic check-in's are avenues to determine current status of complaint investigations and provides an opportunity to offer assistance to the investigator to ensure compliance established deadlines as needed.

Additionally, the Uniform Complaint Procedure Extension form has been updated to include an extended deadline date to complete the investigation.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

Sacramento City Unified provided a copy of a recent complaint log which showed the calculated target dates (30-day, 45-day, and deadline) on which its staff would follow up with the assigned investigator to ensure timely resolution.

In addition, a previous version of SCUSD's complaint extension form did not specify the length of the extension that was being requested. As a result, it was not apparent that the previous version of this form would ensure the timely resolution of complaints. SCUSD has since revised the extension form to indicate the date on which the extension would end. Accordingly, SCUSD has improved its process to ensure a more timely and transparent resolution of its complaints.

6-Month Agency Response

The district failed to update its response in the recommendations database.

California State Auditor's Assessment of 6-Month Status: No Action Taken

Despite our requests, Sacramento City Unified failed to update its response in the recommendations database.

60-Day Agency Response

The District has modified its general complaint acknowledgement letter to reflect the 60-day complaint resolution guideline.

NOTE: Please see sample letter submitted with item 12.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

The district has updated its acknowledgement letter template to include the following language:

"If you do not receive a response within 60 days from «Date__Received», the Human Resource Services office will be contacting you directly to advise if an extension is needed. If you agree with the need to extend beyond the 60 days, please complete the attached complaint extension form and submit to our office."

However, the district has not provided documentation that would demonstrate its timely resolution of the complaints it has received.

Recommendation #15 To: Sacramento City Unified School District

To ensure that it is effectively preventing and addressing incidents of discrimination, harassment, intimidation, and bullying in its schools, Sacramento City Unified should ensure that school sites follow the complaint procedures established in its policies.

Annual Follow-Up Agency Response From October 2018

The District has trained all school site leadership regarding its responsibilities to investigate and document all complaints of discrimination and harassment. It has adopted a revised policy and complaint form for harassment. (see docs.) the District Legal Services and Title IX Officer have presented numerous "Student Harassment Training" for all administrators and required each site to designate a Title IX Administrator. This training included complaint investigation and documentation procedures- specifically including any discipline issued under EC 48900.2.

The District requires all complaints of harassment and investigations/responses to be forwarded to the Title IX Office for oversight pursuant to revised policy. The Title IX Office maintains copies of such complaints/resolutions and tracks all complaints and resolutions through its data base system.(see documents) The District is tracking the data to audit and provide necessary remedies, and additional training as required.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From November 2017

The District has continued its efforts to train all site leadership regarding its responsibilities to investigate all complaints of discrimination. During its 2017 annual required summer trainings, the District Legal Services and Student Hearing Departments presented a "Student Harassment Training" for all administrators. This training included complaint investigation and documentation procedures. The Human Resources Department has continued its annual fall "one-stop" visits to school sites and departments, where a review of complaint procedures and resolution strategies are reviewed.

For significant or formal complaints, the Student Hearing Department oversees student discrimination complaint investigations. This school year, the Student Hearing Department Director/Student Non-Discrimination Coordinator and In House Counsel will provide additional site based trainings to review anti-discrimination procedures, including the requirements of reporting and forwarding same to Student Hearing/Student Non-Discrimination Coordinator's Department for oversight pursuant to Administrative Regulation 5145.3, pg. 5, Investigations of Complaints, no. 11. The Student Hearing Department/Student Non-Discrimination Coordinator maintains copies of such complaints/resolutions and shall track all complaints and resolutions through its data base system. The District, through the Student Hearing Department, shall utilize the data tracking system to assess patterns that require site audits and provide necessary remedies, and additional trainings as required.

In addition, the District office has for many years used standardized processes to collect, monitor and review any and all Uniform Complaints (which all complainants of discrimination are offered pursuant to existing policy and law.)

California State Auditor's Assessment of Annual Follow-Up Status: Pending

During the audit we noted that the district provides an annual training to its school site personnel; however, our audit determined that the training did not effectively "ensure that school sites follow the complaint procedures established in its policies." Although the district provided evidence of its annual training and annual site visits, it did not provide sufficient evidence to demonstrate that these efforts adequately address the findings described in the audit report.

In a follow-up conversation, the district's in-house counsel noted that the district plans to institute annual audits of sites to help ensure ongoing compliance.

Annual Follow-Up Agency Response From November 2016

The district, through the Human Resources Department, has continued its annual fall "one-stop" visits to school sites and department, where a review of complaint procedures and resolution strategies are discussed along with staffing and certification analysis. Additionally, we work directly with school leadership to guide, assist in performance of investigation activities toward resolution, as needed. The attached documents show various methods that are used to schedule these meetings and the information shared with site administrators during the visit.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Sacramento City Unified provided its schedule of school site visits, which are supposed to cover "sexual harassment, discrimination, and prevention training" and the district's complaints and grievances process. However, it did not provide sufficient evidence to demonstrate that these school site visits adequately address the findings described in the audit report.

Annual Follow-Up Agency Response From October 2015

The district has continued its full implementation of this recommendation since the initiation in August 2014. The District conducted is annual harassment prevention training on August 2015 to site administrators, supervisors, and managers which included addressing complaints in accordance with district processes and procedures. The District, through the Human Resources Department (HR Directors), continues its annual fall "one-stop" visits to each school site and department where, review of complaint procedures and resolution strategies are discussed along with staffing and certification analysis. Additionally, we work directly with school leadership to guide, assist, in the performance of investigation activities toward resolution as needed.

Complaint instructions to Investigators (Attachment (1)

Time frame to file:

http://www.scusd.edu/sites/main/files/file-attachments/annual_notification_letter_of_ucp-2015-16_0.pdf

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

The district has continued to claim that it has fully implemented this recommendation; however, it has not provided sufficient evidence to demonstrate its efforts to ensure school sites' compliance with complaint procedures established in its policies.

During the audit we noted that the district provides an annual training to its school site personnel; however, our audit determined that the training did not effectively "ensure that school sites follow the complaint procedures established in its policies." Although the district's annual notification does well to describe the complaint process timelines, the notification does not demonstrate that the district has performed the "one-stop visits" it described in its response.

1-Year Agency Response

The district conducts annual harassment prevention training to site administrators, supervisors, and managers which includes addressing complaints in accordance with district processes and procedures. Staff from the Bullying Prevention Office, Student Behavior Office and the Human Resources Department work closely with school leadership to ensure adherence to related policies.

California State Auditor's Assessment of 1-Year Status: No Action Taken

The district did not provide sufficient detail or evidence to support that it has taken any corrective actions to implement the recommendation. During the audit we noted that the district provides an annual training to its school site personnel; however, our audit determined that the training did not effectively "ensure that school sites follow the complaint procedures established in its policies." In addition, the district failed to indicate how it works closely with school leadership to implement the recommendation.

On September 30, 2014, we notified Sara Noguchi, Sacramento Unified's chief accountability officer, that we assessed the district's implementation status as "No Action Taken" for the reason stated above.

6-Month Agency Response

The district failed to update its response in the recommendations database.

California State Auditor's Assessment of 6-Month Status: No Action Taken

Despite our requests, Sacramento City Unified failed to update its response in the recommendations database.

60-Day Agency Response

The District plans to introduce new software to monitor complaint activities including training for potential investigators of record.

California State Auditor's Assessment of 60-Day Status: Pending

The district's response indicates that it is currently in the process of implementing this recommendation.

Recommendation #16 To: Education, Department of

To provide stronger leadership with respect to school safety and nondiscrimination laws, Education, with direction from the superintendent of public instruction, should ensure that the EO office's program instrument is updated annually to include any new requirements in state law, and also ensure that the EO office performs monitoring visits as required and with sufficient rigor to evaluate LEAs' compliance with state law.

6-Month Agency Response

Education has reviewed the EO Offices program monitoring instrument; updates and revisions were included as necessary to ensure compliance with state law requirements. To date, 40 Education Equity reviews have been completed to evaluate LEAs compliance with state law.

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

Education is reviewing the EO Office's program monitoring instrument. Updates and revisions will be implemented as necessary to ensure compliance with state law requirements.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #17 To: Education, Department of

To provide stronger leadership with respect to school safety and nondiscrimination laws, Education, with direction from the superintendent of public instruction, should prioritize the review of parent, student, guardian, or interested party appeals to ensure that the EO office follows state regulations by processing appeals more promptly, notifying LEAs of when appeals are filed, and obtaining the investigation files and other documents when reviewing complaint appeals.

Annual Follow-Up Agency Response From November 2016

Fully implemented. As a result of the establishment of the Education Equity UCP Appeals Office (EEUCPAO), Education has processes in place that allow the prioritization of the review of parent, student, guardian, and interested party appeals, while ensuring that state regulations are followed and the processing of appeals are completed timely. In addition, Education notifies LEAs when appeals are filed, and obtains the investigation files and other documents when reviewing complaint appeals. For a summary of Education's processes for handling UCP appeals, please see Attachment 8.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From October 2015

Education continues to review the Uniform Complaint Procedures (UCP) and update/revise the process on an as-needed basis to remain in compliance with

state regulations. In 2014-15, Education Equity UCP Appeals Office (EEUCPAO) was established. The EEUCPAO consists of a Staff Services Manager I, an Associate Governmental Program Analyst, and a part-time retired-annuitant. The main purpose of the EEUCPAO is to review and process all appeals and complaints in accordance with all applicable regulations and timelines.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Although Education has established an Education Equity UCP Appeals Office to review UCP appeals, it has yet to create policies and procedures for this office. The creation of policies and procedures will help ensure Education follows state regulations on processing appeals more promptly, notifying LEAs of when appeals are filed, and obtaining the investigation files and other documents when reviewing complaint appeals.

Annual Follow-Up Agency Response From October 2014

Education continues to review the Uniform Complaint Procedures and update/revise the process on an as-needed basis to remain in compliance with state regulations. Currently, Education is in the process of securing funding to support three positions for the sole purpose of reviewing and processing the appeals, according to regulations, and in compliance with all federal timelines.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

Education continues to review the Uniform Complaint Procedures and update/revise the process on an as-needed basis to remain in compliance with state regulations. Currently, Education is in the process of securing funding to support three positions for the sole purpose of reviewing and processing the appeals, according to regulations, and in compliance with all federal timelines.

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

Education will continually revise Uniform Complaint Procedures to improve compliance with state regulations regarding: (1) the intake of appeals; (2) notification letters to parents and LEAs; (3) gathering and reviewing documentation from LEAs; and (4) communication with parents and LEAs.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

Education created a document that outlines the steps to take when reviewing UCP appeals. However, Education stated additional staff is needed to ensure timely review of UCP appeals, and it is considering how to obtain this additional staffing.

60-Day Agency Response

Education is revising Uniform Complaint Procedures to improve compliance with state regulations regarding: (1) the intake of appeals; (2) notification letters to parents and local educational agencies; (3) gathering and reviewing documentation from local educational agencies; and (4) communication with parents and local educational agencies.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation for Legislative Action

By spring 2014 the Legislature should require Education to report to the Senate and Assembly Budget subcommittees on what actions it has taken to improve its processing of appeals, so that the Legislature can consider redirecting existing resources through the annual budget process or taking other actions necessary to ensure that the review of appeals is prioritized.

Description of Legislative Action

The Legislature has not taken action to address this specific recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

Legislation has not been introduced to address this recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

Legislation has not been introduced to address this recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Recommendation #19 To: Education, Department of

To provide stronger leadership with respect to school safety and nondiscrimination laws, Education, with direction from the superintendent of public instruction, should use data from the kids survey and reported suspensions and expulsions to evaluate the levels of discrimination, harassment, intimidation, and bullying students encounter and to determine the effectiveness of its own and the LEAs' efforts, and report the results to the Legislature by August 1, 2014.

Annual Follow-Up Agency Response From November 2017

Education continues to not concur with this recommendation and no further updates will be provided.

California State Auditor's Assessment of Annual Follow-Up Status: Will Not Implement

We stand by our recommendation that Education should use data from the kids survey and reported suspensions and expulsions to evaluate the levels of discrimination, harassment, intimidation, and bullying students encounter and to determine the effectiveness of its own and the LEAs' efforts, and report the results to the Legislature. As we noted on page 40 of our report, by evaluating this data and sharing it with LEAs, Education could provide them with ways to better address the challenges they face in preventing and responding to incidents of discrimination, harassment, intimidation, and bullying.

Annual Follow-Up Agency Response From November 2016

Will not implement. Education did not report to the Legislature by August 1, 2014, as recommended by the CSA. The Healthy Kids Survey referenced in this recommendation is not required by law to be completed, and is not funded in the state budget act. Moreover, the Healthy Kids Survey and data collected regarding suspensions and expulsions was not intended to measure the effectiveness of Education's and LEAs' efforts to address the issues raised by the CSA.

However, notwithstanding the fact that Education cannot implement this recommendation, the Local Control Funding Formula, as enacted in 2013, requires all districts and charter schools to complete a Local Control and Accountability Plan which addresses state identified priorities, actions and goals, including pupil engagement, which by defintion, includes school climate as measured by suspension rates, expulsion rates, and other local measures.

California State Auditor's Assessment of Annual Follow-Up Status: Will Not Implement

Annual Follow-Up Agency Response From October 2015

The suspension and expulsion information collected through CALPADS is collected annually. The CALPADS data is located on Education's DataQuest Web site: http://dq.cde.ca.gov/dataquest/.

Education and WestEd have expanded the reports available to districts that elect to participate in the California Healthy Kids Survey (CHKS) — referred to as the "kids' survey" in the State Auditor's report. A participating district can now obtain information on the reason a student felt bullied; e.g. bias, disability, race/national origin, sexual orientation, et cetera. Reports aggregated by these subcategories are available on the CHKS Web site: https://chks.wested.org/indicators.

California State Auditor's Assessment of Annual Follow-Up Status: Will Not Implement

Education's response does not adequately address the recommendation. Education has informed us a report would not be provided to the Legislature as we recommended. Further, Education has stated it only conducts informal evaluations of the data available because of funding issues and that it encourages LEAs to review the data available themselves. Because Education is not making any effort to implement our recommendation, we have assessed this recommendation as will not implement.

Annual Follow-Up Agency Response From October 2014

The bullying-related data for the 2013-14 school year is collected through the California Longitudinal Pupil Achievement Data System (CALPADS). The window for submitting this data closes September 30, 2014. The 2012-13 bullying-related data was posted on Education's DataQuest Web site in February 2014, and was analyzed by Education staff.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

The bullying-related data for the 2013-14 school year is collected through the California Longitudinal Pupil Achievement Data System (CALPADS). The window for submitting this data closes September 30, 2014. The 2012-13 bullying-related data was posted on Education's DataQuest Web site in February 2014, and was analyzed by Education staff.

California State Auditor's Assessment of 1-Year Status: Pending

Education has yet to report the results of the data gathered and analyzed to the Legislature.

6-Month Agency Response

Education is in the process of analyzing 2012-13 suspension and expulsion data as reported from LEAs. To ensure data accuracy, Education contacted LEAs to confirm that any significant current year data variations from the prior year were correctly reported.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

1. Staff from WestEd, co-founder of the California Healthy Kids Survey (CHKS), or "kids survey" as referenced in the CSA's report, presented a professional development workshop to Education staff on October 3, 2013. The workshop featured discussions on how to utilize data from the CHKS, in addition to the data derived from the California School Climate Survey and the California School Parent Survey.

2. The Safe and Supportive Schools Grant program, a grant that supports statewide measurement of conditions for learning (school climate) and targeted programmatic interventions to improve those conditions, is hosting a conference on November 21-22, 2013. Education will present a breakout session on bullying prevention, resources, and how to use data to plan an effective bullying prevention program.

3. The 2012-13 suspension and expulsion data submission window closed September 30th; the data was recently posted and will be reviewed to assist in determining the effectiveness of ongoing efforts.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation for Legislative Action

The Legislature should consider amending state law to ensure that it aligns with the key components related to school safety that the U.S. DOE has identified. Specifically, the Legislature should consider amending the Education Code to address the concerns we raised in Table 4 on page 50. If the Legislature adds training requirements to the Education Code, it should consider modeling those requirements on the provisions in Massachusetts law.

Description of Legislative Action

The Legislature has not taken action to address this specific recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

Legislation has not been introduced to address this recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

Legislation has not been introduced to address this recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Recommendation #21 To: Education, Department of

To provide stronger leadership with respect to school safety and nondiscrimination laws, Education, with direction from the superintendent of public instruction, should within the next six months and annually thereafter, update and replace the resources on its Web site to provide more relevant information on best practices, such as preventing and responding to incidents related to a protected characteristic or that occur through cyberbullying, the U.S. DOE report on state bullying legislation, and best practices in other states, such as the Massachusetts law on LEA staff training requirements.

Annual Follow-Up Agency Response From October 2018

Education implemented the following items since the last update response. Education: (1) developed and posted a bullying/cyberbullying prevention training module on its Internet Web site in compliance with AB 1993 (Ch. 418, Statutes of 2014), (https://www.cde.ca.gov/ls/ss/se/bullyres.asp); (2) updated online bullying/cyberbullying resources (May and August, 2018); (3) sponsored 2 statewide train-the-trainer workshops on School Behavioral Threat Assessments presented by Readiness and Emergency Management for Schools Technical Assistance Center and the U.S. Department of Education (USDE), Office of Safe and Healthy Students (January 8 and 17, 2018); (4) presented, as a panelist, on the topic of School Safety and co-presented at a workshop on Comprehensive School Safety Planning with the CA Governor's Office of Emergency Services at the CA Association of African American Administrators and Superintendents Statewide Professional Development Summit on Educational Justice in Sacramento, CA, (March 14, 2018); (5) coordinated and implemented the preconference session at the 9th Annual School Safety Conference in Orange County, CA on Protecting Youth from Sexual Misconduct, during which the USDE, Office of Safe and Healthy Students, FBI, CA Interscholastic Federation, and San Mateo County Office of Education (COE) addressed bullying/cyberbullying and other safety issues (July 18, 2018); and (6) implemented a new state compliance audit guide requirement to ensure comprehensive school safety plans are updated and approved by March 1 of each year (beginning in the 2018-19 school year). In addition, Education contracts with the San Joaquin COE to provide objective, research-based criteria to identify high-quality instructional resources for posting on its Web site. Education is also receiving a $1 M grant to implement the Student, Teachers, and Officers Preventing Violence Act Program, in partnership with Sandy Hook Promise, to deliver school safety and mental health training.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Education has updated and replaced the resources on its website to provide more relevant information on best practices related to school safety and nondiscrimination laws, and it now has a policy of annually updating its website.

Annual Follow-Up Agency Response From November 2017

Education, with direction from the State Superintendent of Public Instruction (SSPI), annually updates and replaces resources on its Web site to provide best practices. Education updated bullying prevention Web site resources to include a link to the Massachusetts Model Bullying Prevention and Intervention Plan, participated in community-law enforcement training on Bridging the Racial Divide that focused on discrimination, harassment, bullying, cyberbullying, and racially motivated trends and events, has committed to sponsor two statewide workshops in the 2017-18 school year on School Behavioral Threat Assessments, which includes important federal guidance on school emergency management planning and will be presented by the USDE, Readiness and Emergency Management for Schools Technical Assistance Center, is continuing to convene the SSPI School and Community Safety Advisory Committee, presented a workshop at the 2017 Alliance for Boys and Men of Color Policy Summit, focusing on Education's programs and activities related to improving school climate, reducing bullying, harassment, and hate crimes, presented a pre-conference session at the 8th Annual School Safety Conference with the SSPI, the Director of the USDE, Office of Safe and Healthy Students, and school safety experts, on emergency preparedness, school climate, and school safety planning with federal, state, and local resources and components, provided technical assistance to parents, families, LEAs, and governmental agencies, is developing guidance on model procedures for approving and monitoring school safety plans, is developing guidance on building disaster plans, received approval to include school safety plans in the 2018-19 audit guide, is collaborating with the California District Attorneys' Association to increase compliance with reporting of child abuse and neglect; and endorsed and supported an educator-law enforcement partnership trauma-informed program titled, Focusing on Children Under Stress.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Education's response lists activities that it plans to implement in the future. As a result, we look forward to reviewing Education's next response and verifying that it has completed those activities.

Annual Follow-Up Agency Response From November 2016

Fully implemented. To provide stronger leadership with respect to school safety and nondiscrimination laws, Education has conducted a myriad of school safety and nondiscrimination efforts in regard to the CSA's recommendation, as further delineated in Education's written one-year corrective action update.

In addition, Education: (1) is developing bullying and cyberbullying technical assistance materials with research-based strategies and interventions, with input from state and national experts; (2) updated and is in the process of posting the Comprehensive School Safety Plan Self-Monitoring tool to include both bullying and cyberbullying as a requirement in school safety policies; and (3) is developing a plan and process for identifying model comprehensive school safety plans to be shared with LEAs at the 8th Annual Safe Schools Conference.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Although Education has provided some additional resources to LEAs, it has yet to update its website with relevant information on best practices such as preventing and responding to incidents related to a protected characteristic or that occur through cyberbullying. Further, as noted in Education's response, some of its actions are still in progress. Therefore, we have assessed this recommendation as partially implemented.

Annual Follow-Up Agency Response From October 2015

To provide stronger leadership with respect to school safety and nondiscrimination laws, Education has: (1) maintained the School Community Violence Prevention Training Grant Regional Safety Coordinator system; (2) provided training in safe school planning and bullying/cyberbullying prevention; (3) reviewed training materials to maintain quality and consistency of quarterly training; (4) presented bullying and cyberbullying breakout sessions at the California Parent Teacher Association Conference and the 6th Annual Safe Schools Conference; (5) provided technical assistance to parents and local educational agencies; (6) created guidance on Assembly Bill 1266, regarding transgender youth and equal access to facilities and programs; and (7) participated in the Legislative Women's Caucus Task Force on sexting.

In addition, Education is in the process of updating and replacing the resources on the Web site to provide more relevant information on best practices.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Although Education has provided some additional resources to LEAs, it has yet to update its website with relevant information on best practices such as preventing and responding to incidents related to a protected characteristic or that occur through cyberbullying. Therefore, we have assessed this recommendation as partially implemented.

Annual Follow-Up Agency Response From October 2014

1. Education staff virtually attended the 2014 Federal Bullying Prevention Summit, "Keeping Kids Safe: Opportunities and Challenges in Bullying Prevention," held August 15, 2014. Twenty-two speakers presented on topics ranging from the state of current bullying prevention research, multi-tiered behavioral frameworks, cyberbullying, and school discipline.

As a result of this training, Education is adding resources from the Bullying and Hate Motivated Behavior Web site that were shared at the 2014 Federal Bullying Prevention Summit to Education's Web site. Resources include:

- Educator's Guide

- 2000 Dear Colleague Letter (DCL) on Disability Harassment

- 2010 DCL on Harassment and Bullying

- 2013 DCL on Bullying of Students with Disabilities

- A Parent's Guide to Cyberbullying

- Campus Pride Fact Sheet

- Campus Pride Index Fact-Sheet

- Community Action Toolkit

- Educators and Community Leaders' Guide

- Facebook Resources for Teachers 2014

- Hazing in View High School Report

- Know Bullying App Postcard

- National College Fair-Program Fact-Sheet

- OSERS Dear Colleague Letter Bullying-8-20-13

- StopBullying.gov Infographic

- StopBullying.gov Training Module

- Supportive School Discipline Initiative Information Sheet

- SBSU videos for Federal Conference

2. Education is adding the following links to Speaker Presentations from the Federal Bullying Prevention Summit onto our Web site:

- Laws and Policies: Sarah Allen PowerPoint

- Media Guidelines: Media Guidelines PowerPoint

- Related High-Risk Behaviors: Darlene Johnson PowerPoint; Elizabeth Allan PowerPoint

3. Education is providing the link to Substance Abuse and Mental Health Services Administration's (SAMHSA) new App, "Know Bullying," revealed at the 2014 Federal Bullying Prevention Summit.

4. Education staff presented a workshop at the 5th Annual Safe Schools Conference, "Bullying 101," July 31, 2014.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

1. Education staff virtually attended the 2014 Federal Bullying Prevention Summit, "Keeping Kids Safe: Opportunities and Challenges in Bullying Prevention," held August 15, 2014. Twenty-two speakers presented on topics ranging from the state of current bullying prevention research,

multi-tiered behavioral frameworks, cyberbullying, and school discipline.

As a result of this training, Education is adding resources from the Bullying and Hate Motivated Behavior Web site that were shared at the 2014 Federal Bullying Prevention Summit to Education's Web site. Resources include:

- Educator's Guide

- 2000 Dear Colleague Letter (DCL) on Disability Harassment

- 2010 DCL on Harassment and Bullying

- 2013 DCL on Bullying of Students with Disabilities

- A Parent's Guide to Cyberbullying

- Campus Pride Fact Sheet

- Campus Pride Index Fact-Sheet

- Community Action Toolkit

- Educators and Community Leaders' Guide

- Facebook Resources for Teachers 2014

- Hazing in View High School Report

- Know Bullying App Postcard

- National College Fair-Program Fact-Sheet

- OSERS Dear Colleague Letter Bullying-8-20-13

- StopBullying.gov Infographic

- StopBullying.gov Training Module

- Supportive School Discipline Initiative Information Sheet

- SBSU videos for Federal Conference

2. Education is adding the following links to Speaker Presentations from the Federal Bullying Prevention Summit onto our Web site:

- Laws and Policies: Sarah Allen PowerPoint

- Media Guidelines: Media Guidelines PowerPoint

- Related High-Risk Behaviors: Darlene Johnson PowerPoint; Elizabeth Allan PowerPoint

3. Education is providing the link to Substance Abuse and Mental Health Services Administration's (SAMHSA) new App, "Know Bullying," revealed at the 2014 Federal Bullying Prevention Summit.

4. Education staff presented a workshop at the 5th Annual Safe Schools Conference, "Bullying 101," July 31, 2014.

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

Education updated Web site resources and encouraged the sharing of best practices as follows:

1. Education revised documents posted on the Safe School Planning Web site to include references to mental health as a crucial component to creating a safe learning environment, free of bullying and cyber bullying.

2. The State Superintendent of Public Instruction issued a press release calling attention to the revised Safe School Plan documents (Enclosure 1).

3. Education conducted a Safe and Supportive Schools conference with bullying and cyber bullying prevention breakout sessions on November 19-20, 2013. Best practices were shared amongst conference attendees.

4. Education provided training materials to the California School for the Deaf in Riverside on the topics of bullying, cyber bullying, and sexting.

5. Education staff are members of the Cyber Bullying Task Force convened by the Legislative Womens Caucus, under the direction of Vice-Chair Senator Hannah Beth Jackson.

California State Auditor's Assessment of 6-Month Status: No Action Taken

The actions taken by Education do not address the specific concern of our recommendations to "update and replace the resources on its Web site to provide more relevant information on best practices, such as preventing and responding to incidents related to a protected characteristic or that occur through cyberbullying." Education's updates to the Web site appear to be no more than a typical annual update of Web resources. To address our recommendation, Education needs to improve its leadership over school safety issues by making available more complete and robust resources to schools, rather than simply updating several links on its Web site.

60-Day Agency Response

1. Education is in the process of revising the resources listed on its "Bullying and Hate Motivated Behavior" Web pages and is seeking input from:

- San Diego County Office of Education

- Sacramento City and West Sacramento Unified School Districts

- School Community Violence Prevention Regional Safety Coordinators

- Dr. Justin Patchin, nationally recognized expert on cyberbullying and teens

- B.R.A.V.E. Society (Bullies Really Are Violating Everyone), a local nonprofit

2. On September 20, 2013, Education updated its Equal Opportunity & Access Web page to include information and resources available over gender equity through the requirements enacted in Title IX of the Education Amendments of 1972.

3. Education continues to work with the School Community Violence Prevention Training Grant's Regional School Safety Coordinators to provide statewide training in the areas of bullying/cyberbullying, safe school planning, crisis response, threat assessment and suicide prevention, classroom management, and legal strategies for information sharing. The 2013-14 schedule of trainings is posted on Education's Violence Prevention Web site at http://www.cde.ca.gov/ls/ss/vp/scvptraining.asp.

Additionally, Education and the California Resource Officers Association are co-sponsoring a conference on October 27-29, 2013: Bullying: Issues and Strategies. Education will moderate a panel discussion on cyberbullying and present two workshop sessions on bullying legislation and resources. Attendees include school, district, and county administrators, counselors, teachers, parents, practitioners, and students. (A copy of conference flyer is enclosed.)

California State Auditor's Assessment of 60-Day Status: Pending

All Recommendations in 2012-108

Agency responses received are posted verbatim.