Chapter 2

Departments Have Not Consistently Tested for Accessibility, and Statewide Web Accessibility Training Should Be Required

Key Points

» The four departments we reviewed conducted some accessibility testing, but most could improve the method and frequency of their tests. For example, Covered California has not regularly tested updates to its website, and the update testing that the California Department of Human Resources (CalHR) performed did not include automated testing techniques.

» State policy requires most departments to provide specific contact information so that website users can complain about accessibility issues. Although not specifically required to provide this information, Covered California and California Community Colleges (Community Colleges) do not provide all of this important information on their websites. This omission limits the number of ways for users who want to file an accessibility complaint to do so.

» CalHR lacks adequate processes for tracking the results of its accessibility testing and the complaints it receives about the accessibility of the Careers in California Government website (jobs site) and, as a result, cannot effectively monitor accessibility testing or the management of complaints.

» Shortcomings in department web accessibility testing demonstrate a need for a statewide formalized testing approach. In addition, we believe California would benefit from greater oversight of accessibility testing and the California Department of Technology (CalTech) is the logical entity to take on this responsibility.

Each Department Conducted Some Accessibility Testing, but Most Do Not Sufficiently or Consistently Test Updates to Their Websites

Guidance at both the state and federal levels indicates departments should test web products for accessibility, whether they are developed in-house or procured from outside vendors. The U.S. Department of Justice, which at the federal level is tasked with issuing recommendations regarding federal government compliance with Section 508 of the federal Rehabilitation Act of 1973, as amended (Section 508) standards, recommends that agencies develop procedures to regularly test their web products for accessibility, using both manual and automated methods of testing. Manual testing involves actual users attempting to use a website, sometimes with the help of assistive technologies such as screen readers. Automated testing can include the use of freely available tools online that scan web pages for accessibility problems. Also at the federal level, the U.S. General Services Administration advises departments to incorporate testing against the Section 508 standards into verification and validation of web products before they are deployed, including completing testing with assistive technologies.¹¹ The Statewide Information Management Manual (SIMM) says that accessibility testing should occur for web products and should include individuals who represent various types of disabilities.

The departments we reviewed all conducted some accessibility testing on their web-based services, as shown in Table 7. In general, testing occurred before the key release of the supporting web product to the public. Covered California’s vendor completed accessibility testing of the California Healthcare Eligibility, Enrollment, and Retention System (CalHEERS) website before issuing the first major release of the site to consumers in October 2013. This testing included both automated and manual review of the CalHEERS site. Similarly, Community Colleges’ vendor performed both manual and automated accessibility testing before the debut of its OpenCCCApply website (online application) in November 2012. According to the executive director of the Community Colleges Technology Center (technology center), Community Colleges also had a staff developer review some of the vendor’s work to ensure that the online application aligned with accessibility standards. CalHR’s supervisor over web development explained that when the current version of the jobs site, which was released in 2012, was being developed, developers conducted accessibility testing on portions of the site by using an online automated tool and that the California Department of Rehabilitation also reviewed some online job exams at the time using screen reader software.¹² However, there is no documentation of this testing or its results.

In contrast, the State of California Franchise Tax Board (Franchise Tax Board) did not conduct accessibility testing on its CalFile product until many years after it was developed. According to Franchise Tax Board’s accessibility lead analyst, Franchise Tax Board first conducted accessibility testing on CalFile in 2012. However, according to a systems software supervisor, Franchise Tax Board originally developed CalFile in 2003. We reviewed summary accessibility test reports and individual issues found during testing that show that since 2012, the start of the period we reviewed, Franchise Tax Board has regularly tested CalFile for accessibility and has used both automated and manual methods since 2013.

Table 7
Accessibility Testing Approaches by Four Departments
	Initial Testing		Update Testing
	Method		Method
Department	Automated	Manual	Automated	Manual	Frequency
Covered California	Yes	Yes	No	Yes^†	Not regular
California Community Colleges (Community Colleges)	Yes	Yes	No	Yes	Not regular
California Department of Human Resources (CalHR)	Yes*	Yes*	No	Yes	Not regular
State of California Franchise Tax Board (Franchise Tax Board)	No	No	Yes	Yes	Regular

Sources: California State Auditor’s interviews with staff and review of available testing documentation at Community Colleges, CalHR, Covered California, and Franchise Tax Board.

* Staff stated that testing was performed, but it was not documented.

^† This testing was limited to evaluating the Voter Registration page with a screen reader.

Although three of the four departments conducted some accessibility testing before the initial release of their web service, Franchise Tax Board is the only department that conducted regular update testing. Other departments we reviewed have not regularly tested subsequent updates to their websites. According to testing reports written by Covered California’s vendor, the vendor performed no dedicated accessibility testing after CalHEERS’ first major release to consumers in October 2013.¹³ The CalHEERS schedule shows that since that time, Covered California has issued updates to the CalHEERS website. Covered California’s chief of operational readiness confirmed that these updates included changes to content the public interacts with on the site but stated that most of the changes added functionality for Covered California staff. However, as we discussed in Chapter 1, Covered California recently updated a page on its website on which users begin an application for health insurance. That new page had a critical violation that blocked keyboard-only users from being able to begin an application. The assistant project manager for the CalHEERS project confirmed that Covered California was unaware of this accessibility violation until we brought it to its attention. More thorough testing of updates to the CalHEERS website would help Covered California guard against releasing inaccessible content. Table 7 summarizes the initial and update testing performed by the four departments we reviewed.

Community Colleges’ technology center does not have a formal plan to ensure that updates to its online application are tested for accessibility, and it acknowledged that the testing that does occur is not comprehensive. The technology center’s agreement with its vendor states that the vendor is responsible for testing the online application, but it does not specifically mention accessibility testing. The database that the technology center uses to track testing of its online application shows that its vendor conducted some manual testing of the website after the release of the website to the public. However, Community Colleges acknowledged that its vendor does not always test updates to its online application for accessibility. For example, the technology center’s chief technology officer acknowledged that the online application’s security time‑out feature was not tested for accessibility before being released in December 2014 and that because of this, the technology center’s vendor did not know about an error on its site wherein screen reader users would not know they were about to be timed out from the site. The technology center was unaware of the issue until a visually impaired user brought the issue to its attention.

Similarly, CalHR informally tests updates to its jobs site, and the testing does not follow best practice recommendations to include automated testing as well as manual. According to the supervisor over web development for its jobs site, he will occasionally ask for accessibility tests on new content. He explained that he usually decides to request testing when there have been changes to standard templates that CalHR uses for its online exams. The web manager confirmed that he conducts testing on new exams posted on the jobs site when it is requested, as there is no formal approach for regular testing. Further, the web manager stated that his testing did not include automated methods and involved using only screen reader software to manually test the exams. According to the W3C, automated accessibility evaluation tools can reduce the time and effort required to carry out testing, because they can determine websites’ compliance with standards that can be evaluated automatically, as well as assist reviewers in performing testing on some standards that need to be evaluated manually.

Three departments we reviewed track the results of their accessibility testing. Covered California and Community Colleges both use databases to track testing results from the time an issue is identified through resolution. Through 2013, according to its accessibility lead analyst, Franchise Tax Board used a database to track testing results. According to the accessibility lead analyst, in 2014 the department tracked unresolved accessibility issues through an annual accessibility report because it was transitioning to a new tracking database, which the department began using in 2015. Further, the accessibility lead analyst indicated that as Franchise Tax Board transitions to its new tracking database, the accessibility testers have exchanged emails with the CalFile development team about problems they identify with CalFile, and any issues not resolved by the end of each calendar year will be reflected in the annual accessibility report.

Despite using a database to track accessibility defects with its online application, Community Colleges has not provided adequate direction to its vendor about how to indicate that a defect has been resolved. Throughout the audit, we relied in part on Community Colleges’ database in order to determine whether it had addressed accessibility problems that it was made aware of through testing or because of a complaint from a user. However, in several cases, we found that Community Colleges’ vendor had closed the record of a defect without indicating that the defect had been resolved. Because of this, it was difficult for us and for the technology center’s chief technology officer to determine when or whether the accessibility problems underlying the database records had actually been resolved. According to the chief technology officer, the database was sometimes inadequate for determining whether a defect was fixed. He further indicated there would be value in changing how defects are tracked in the technology center’s database so that it is clear when they are resolved.

According to its web manager, CalHR does not track the results of its accessibility testing in a systematic manner that would allow it to easily determine whether the issues identified in testing were addressed. As previously described, CalHR staff reported that they perform accessibility testing when requested to do so after changes are made to existing website templates. According to CalHR’s web manager, who performs the testing, the only records of testing results are his email records. The web manager acknowledged that the emails he retains do not necessarily reflect all accessibility testing emails he sends or receives. We reviewed the web manager’s email records and found incomplete information about testing efforts. The level of information in the emails we reviewed would not allow CalHR to know consistently whether issues were resolved and, if so, how quickly. The web manager and the supervisor over development for the jobs site acknowledged that CalHR does not currently have a level of information about accessibility testing that would allow it to know what portions of the site had been tested, what issues were found, and whether those issues were resolved. According to the supervisor over development for the jobs site, since the audit began, CalHR has begun drafting new procedures for accessibility testing and tracking the results of that testing, but he did not have an estimate for when those procedures would be finalized.

Departments Generally Address Problems Found During Testing, but Not Always Before Releasing Their Websites to the Public

Three of the departments we reviewed did not always resolve issues found during accessibility testing before releasing their web service to the public or before issuing significant updates to the service. As discussed in objective 5(e) of the Scope and Methodology table on the Introduction page, we did not review a selection of testing results at CalHR due to the poor quality of the department’s record keeping related to accessibility testing. We explained in the previous section that the lack of comprehensive tracking at CalHR prevents it from effectively knowing what it has tested and whether it has addressed all issues it finds during testing. In the case of both Covered California and Community Colleges, we assessed whether issues were identified and resolved before the debut of the departments’ products to the public. For Franchise Tax Board, because the department did not begin testing CalFile for accessibility until after it was initially developed, we assessed whether defects that were identified after the product was released were resolved and how quickly.

Covered California did not resolve before the first major release of its CalHEERS site in October 2013 all of the defects that we reviewed that it had identified at that time. As shown in Table 8, of the 22 defects Covered California identified by October 2013 that we reviewed, 20 were not resolved before that date, despite Covered California knowing that the issues existed. In fact, Covered California took an average of 245 days from its release to resolve these defects. One of these unresolved defects was that alternative text, which screen reader users rely on to describe to them what images are found on a web page, was missing from most of the images on the site’s pages. Without these text alternatives, visually impaired users cannot understand what images are found on the page that they are navigating. According to similar types of violations that our web accessibility consultant (consultant) found at all four departments that we reviewed, an accessibility issue with alternative text would, at a minimum, be considered a serious violation that makes some content inaccessible.¹⁴ According to the CalHEERS assistant project director, the accessibility defects that were not resolved before the October 2013 release had to be prioritized along with all other CalHEERS defects or they were duplicates of other already existing defects. Further, she explained that the release date could not be pushed back due to the legally mandated date for the State’s first open enrollment to begin. She also stated that some of these defects were later rejected, meaning they were ultimately determined not to be true defects. However, CalHEERS staff did not know that these defect records did not represent actual accessibility problems or that they were duplicates of existing defects until they were reviewed and subsequently rejected. As of December 2014, all 20 of these defects were marked as resolved in the department’s tracking database. In addition, Covered California identified eight other accessibility defects that we reviewed after the first major release of CalHEERS and resolved these within an average of about 50 days of discovering the defects.

Table 8
Defects Found During Accessibility Testing at Three Departments
	California Community Colleges (Community Colleges)	Covered California	State of California Franchise Tax Board (Franchise Tax Board)
Defects we reviewed*	23	30	20
Defects department resolved	23	30	19
Resolved before release	12	2	NA^†
Resolved after release of product^‡
Identified before release	6	20	NA^†
Average days from release to resolution	219^§	245	NA
Identified after release	5	8	19
Average days to resolution	12	49	182

Sources: California State Auditor’s analysis of data from Community Colleges Technology Center’s JIRA database; Covered California’s Application Lifecycle Management database; Franchise Tax Board’s ClearQuest database; email records of Franchise Tax Board staff; and interviews with staff at Covered California, Community Colleges, and Franchise Tax Board.

NA = Not Applicable.

* Although we planned to review 30 defects at each department, the keyword search to identify defects at Community Colleges and Franchise Tax Board resulted in fewer than 30 accessibility-related defects.

^† Franchise Tax Board did not conduct accessibility testing on CalFile until 2012, so no defects were identified prior to its 2003 release.

^‡ In this table, release refers to the date on which the product supporting the online service we reviewed became available for public use at each department.

^§ This average includes two items that took 827 days and 398 days, respectively, to resolve which was notably longer than the other four items included in this average.

Similarly, at Community Colleges, the technology center and its vendor were aware of some issues prior to the release of the online application but did not resolve all of those issues prior to the release of the application. We reviewed 23 defects at this department, including 18 defects identified before Community Colleges released the online application. Community Colleges did not resolve six of the 18 defects despite the technology center and vendor knowing about them before the application’s release in November 2012. Four of these issues were resolved on average within about a month of the application’s release. However, the remaining two were not resolved for more than 300 days after the application’s release. One of these issues, in which error messages were not being read by screen readers, took 827 days to resolve. If visually impaired screen reader users do not have error messages read to them, the users will be unaware of the specific problem that prohibited them from continuing their application. The technology center’s chief technology officer was not in his current position in November 2012 when the online application was first released. However, he stated that the vendor who built the site told him that this problem was addressed shortly after the online application’s initial release. According to the chief technology officer, the vendor states that code change logs indicate the problem was addressed in December 2012. Nevertheless, the defect record in the technology center’s tracking database contains notes that show that the problem persisted after December 2012. Further, we were not able to locate any evidence in Community Colleges’ tracking database that demonstrates that it addressed this error message defect any earlier than February 2015.

In the other case, the technology center’s vendor closed the defect with a “Won’t Fix” status prior to the release of the application. However, the notes for this defect show that the technology center’s vendor knew that this defect meant a security measure that the online application used was not accessible to screen reader users because it relied on an image that did not have alternative text. The security feature that the online application featured was designed to help tell human users apart from malicious software. According to the technology center’s chief technology officer, the development team originally had to make a judgment call between security and accessibility and chose to focus on the security of the online application in this case. The technology center eventually addressed the accessibility of this security feature 398 days, or more than one year, after the site was released. According to the chief technology officer, this was possible because the security feature’s manufacturer released an accessible version of the feature, which the technology center adopted.

As discussed in the previous section, Franchise Tax Board did not begin accessibility testing on CalFile until 2012, after the product’s initial 2003 release. According to the manager overseeing CalFile, Franchise Tax Board releases an update to CalFile in January of each year at the start of each tax filing season. After that release, the department does not issue another update to CalFile until the following year, unless there are issues that would cause incorrect filings to occur. Of the 20 defects we reviewed, Franchise Tax Board addressed 15 before the next annual update of CalFile. However, four additional defects we reviewed were defects Franchise Tax Board was aware of prior to releasing the next update, but it did not release a fix for those defects until the following year’s update. Three of these defects were related to keyboard navigation errors in the CalFile application, and the fourth was related to screen reader use. For two of these defects, the accessibility lead analyst stated that due to constraints on time and resources to address problems, they were not addressed until the following year. For the third defect, Franchise Tax Board staff were unable to explain why the defect had not been addressed. According to a supervisor in Franchise Tax Board’s Internet and taxpayer folder section, the fourth defect was corrected before the next CalFile update, but the item was not closed in its database until after the update. However, he could not provide any documentation to that effect. Finally, out of 20 total defects we reviewed, we found one unresolved defect related to screen reader users being unable to review information they had entered in the CalFile application. We discussed this defect in greater detail in the Chapter 1 section on burdensome violations of web accessibility standards.

Some Departments Must Improve the Complaint Contact Information Presented on Their Websites

Key Elements Required on State Websites

“Contact Us” or “Accessibility” links
A statement affirming the department’s commitment to accessibility
Contact information, including telephone number, mailing address, email address, and teletypewriter (TTY) number
Instructions for reporting accessibility problems

Sources: Information Organization, Usability, Currency, and Accessibility Working Group recommendations from July 2006; California Department of Technology, Information Technology Policy Letter 10-10.

Before the start of this audit, most departments we reviewed did not provide important information to the public regarding how to complain about accessibility problems with websites. At the federal level, Section 508 states that individuals with disabilities can file complaints regarding failure to provide accessible electronic information technology, such as websites. State policies issued by the state chief information officer establish a variety of elements that most departments must include on their website, as shown in the text box. Some of the elements are designed to help members of the public make complaints about website accessibility issues. In addition, state guidance provides that complaints about website accessibility should be directed to a person or group with the responsibility and technical knowledge necessary to effectively respond to such complaints. Although Covered California is not required to comply with policies issued by the state chief information officer and it is unclear whether Community Colleges must follow those same policies, we assessed the complaint contact information on both departments’ websites against the guidance and best practices that are the basis of the requirements that other departments must follow. At the time of our review, Covered California did not provide all of the complaint contact information on its website that state policy requires other departments to provide and Community Colleges’ online application featured only a contact page for general help that lacked any reference to web accessibility or complete contact information. Also, at the time of our review, Franchise Tax Board lacked complete contact information on the portion of its website that discussed accessibility. Only CalHR’s website met all state criteria for providing information about how to complain about web accessibility problems.

As of November 2014 Covered California had updated the instructions on its website for submitting a complaint so that they are now more specific to website accessibility. This update was part of a website redesign initiated before this audit began. The previous version of the website had no accessibility page; instead, it had only a contact page that directed all requests for assistance to a service center. The site now features an accessibility page that includes a statement of commitment to accessibility. However, it provides only an email address as contact information for submitting a complaint and does not feature the remaining contact information that other state websites are required to provide. According to its marketing director, Covered California made an organizational decision to post only the email address because high turnover made it difficult to identify one designated person as the contact for accessibility issues. However, departments may create barriers to reporting web accessibility issues when they do not provide more than one type of contact information. To make itself more reachable by persons with disabilities, Covered California could also include a telephone number, teletypewriter number, and mailing address on its accessibility page as other departments are required to do.

Until the start of this audit, Franchise Tax Board’s website did not provide specific instructions for users wishing to file a complaint about website accessibility. Franchise Tax Board’s accessibility page outlined its Americans with Disabilities Act (ADA) grievance process, which it operates from its Equal Employment Opportunity (EEO) office and which is specific to ADA-related issues. The page did not mention web accessibility or Section 508. Additionally, the accessibility page did not include an email address to which complaints could be directed. A different area of Franchise Tax Board’s website provided a general complaint form for users who experience problems with the website but did not indicate whether it was the correct method for submitting an accessibility‑related complaint. A lack of clear information about how to report problems with website accessibility could result in individuals who experience problems not knowing where to direct a complaint. Franchise Tax Board’s EEO officer was not able to identify a reason that the department’s accessibility page did not include information specific to Section 508 web accessibility, and she stated that she had been unaware of the requirement regarding full contact information for web accessibility complaints until we informed her.

After we asked Franchise Tax Board about the content of its accessibility page, it modified its site to provide more clarity on how to report web accessibility problems. The website’s accessibility page now instructs users to contact Franchise Tax Board’s ADA coordinator if they have any difficulties accessing the site materials. Franchise Tax Board also updated the page to include the ADA coordinator’s email address. This change brings Franchise Tax Board’s accessibility page into full compliance with state requirements.

The Community Colleges online application lacks most of the information that state policies require other departments to include related to accessibility complaints. The site features a telephone number and email address for technical support but none of the other forms of contact information or a statement of commitment to accessibility. According to Community Colleges’ technology center executive director who oversees this site, he was unaware that there were state requirements for an accessibility page. As we indicate in the next section, some users followed the technical support contact information on the online application to complain about its accessibility. However, by not providing all forms of contact information that state guidance indicates should be included on a website, Community Colleges is limiting the number of ways in which persons with disabilities can complain if they encounter problems with the online application.

The Departments We Reviewed Appear to Receive Few Complaints About Web Accessibility

None of the departments we reviewed tracked web accessibility complaints separately before the start of our audit. When we began this audit, staff at both Covered California and Community Colleges indicated that both entities logged complaints about their web-based services into a database. However, neither department’s database included a separate flag that would allow the department to identify all complaints related to website accessibility. Both departments have since changed their approach to tracking accessibility complaints. After the start of our audit, the Community Colleges technology center added a flag to its database specifically to track complaints from persons with disabilities. Also, resulting from a website redesign initiated before this audit began, Covered California now directs all persons with web accessibility questions, suggestions, or complaints to submit them using a single email address. Finally, as discussed in the previous section, before the start of this audit, Franchise Tax Board did not provide clear instruction regarding where users should direct web accessibility complaints. As a result, users with accessibility problems may have reasonably used either Franchise Tax Board’s ADA grievance process or its general website complaint form, or they may not have complained at all. However, Franchise Tax Board now instructs website users to direct all questions about web accessibility to its ADA coordinator. By doing so, Franchise Tax Board has helped ensure that web accessibility questions and complaints will not be mixed among other website issues, which should assist it in accurately tracking any complaints about web accessibility.

Although Covered California now directs all accessibility complaints to one email address, there is no formal process in place for handling these emails. According to Covered California’s chief of information technology services, most of the issues users have emailed about are related to user accounts and choosing a health plan, which are issues better handled by Covered California’s service centers. However, when we asked about Covered California’s process for ensuring that accessibility‑related emails were addressed, the chief of information technology services confirmed that no written procedure exists for handling accessibility complaints. She stated that someone reviews the emails daily. She also stated that the department planned to develop procedures for processing the emails and explained what steps Covered California would take if it received a web accessibility complaint before finalizing these procedures. However, until it develops such procedures, Covered California is at a greater risk that a user’s accessibility complaint will go unnoticed or will not be adequately addressed.

CalHR provides clear direction on its website for how users can report problems with the accessibility of its site, but it does not track complaints that it receives. The contact information on CalHR’s accessibility web page connects complainants with its web manager for web accessibility issues. However, the web manager stated that there is no process for tracking user issues. The web manager stated that he receives few calls and emails regarding web accessibility and that he assists users who call to complain about the website. However, there is no procedure in place to ensure that if he discovers a problem with the website while assisting a user, the error is documented and resolved. For example, the web manager recalled helping a visually impaired complainant through a problem with web page navigation and in so doing realized that the “skip to content” option on CalHR’s jobs site was disabled, which contributed to the complainant’s issue with the site. However, when we asked whether this issue on the jobs site had been addressed, the web manager indicated that it had not. He acknowledged that CalHR could benefit from better tracking of the receipt and resolution of complaints.

We found no record of accessibility complaints at two of the departments that kept records of complaints related to their websites. At both Covered California and Franchise Tax Board, we searched available complaint records for keywords related to web accessibility, such as “screen reader,” “508,” “blind,” and “deaf.” At Covered California, we searched complaints received at its service centers from June 2013 through December 2014 and the record of emails it received at its new accessibility complaint email address since November 2014. At Franchise Tax Board, we reviewed the only complaint its EEO officer stated that it received through its ADA grievance process since 2010, along with emails submitted through its general website complaint form, which at the time of our review was a method website users could use to submit complaints. Our search of those records was limited to the emails that Franchise Tax Board retains. According to the staff member in charge of responding to general website complaints, Franchise Tax Board retains a record of those complaints for only 60 days after the issue has been resolved. As a result, our search was limited to the records that existed when we performed our search in March 2015. At both of these departments, we found no complaints related to web accessibility.

At Community Colleges, we identified several complaints related to web accessibility. We searched the records of complaints received about Community Colleges’ online application since it was first made public in November 2012 using keywords related to web accessibility. Through this process, we identified five web accessibility complaints. Most of these complaints concerned the interaction of the online application with screen reader software. One such complaint was already discussed in Chapter 1 and related to a problem with the time‑out feature of the online application not being accessible to those persons using a screen reader. The same complainant also identified several other problems he experienced while using his screen reader to complete the application, such as the screen reader not identifying a check mark symbol that shows if a section has been completed, which was also identified by our consultant and discussed in Chapter 1. This complaint was forwarded to Community Colleges’ development team so that it and other issues the user identified could be addressed. Records from Community Colleges’ tracking database indicate that the check mark issue was resolved in late February 2015, which was within one month of receiving the complaint about this issue.

The few complaints we observed across the departments we reviewed appear consistent with the few complaints reported at the federal level. In 2010 and 2011, the U.S. Department of Justice surveyed 89 federal agencies about their implementation of Section 508. In their responses to the survey, those agencies self‑reported that, from June 2001 through 2010, they had received a total of 140 administrative complaints about their web accessibility, or only about 15 complaints per year among all of the agencies. However, because our search for complaints at the departments we reviewed relied on keywords, it is possible that complaints exist that we did not identify. It is also possible that the few complaints we observed are related to the lack of clear complaint contact information provided by some departments, as we discussed in the previous section.

Levels of Training at Each Department Varied, and Statewide Training on Web Accessibility Is Not Required

We identified no requirement for statewide training specific to the development and maintenance of accessible websites. As discussed in the Introduction, the state law that requires departments to maintain their websites in accordance with the Section 508 standards does not establish a department or agency responsible for statewide enforcement or oversight of these standards. Additionally, no state law we reviewed as part of this audit requires web accessibility training for state employees. As part of its general activity as California’s lead agency for matters related to information technology, CalTech maintains a website dedicated to conveying information and resources to state webmasters about a variety of topics, including web accessibility.¹⁵ CalTech also facilitates a webmaster users group that meets periodically to discuss topics related to website management. According to the agendas for this users group going back to 2009, discussion at the meetings has included accessibility on several occasions, including as recently as February 2015, when the California Department of Rehabilitation (Rehabilitation) offered a refresher session on basic web accessibility requirements. According to the web services manager at CalTech, attendance at these sessions is optional.

At the departments we reviewed during this audit, the level of training offered by the departments or obtained by staff was inconsistent. CalHR’s web content manager stated that he leads training for department staff on how to ensure that documents uploaded to its site are accessible. Staff from Franchise Tax Board attended a training session provided by an accessibility consulting firm in 2013 about the accessibility of PDF documents. In contrast, staff at Covered California and the Community Colleges technology center reported that they did not receive training on web accessibility from their respective department or attend any such third-party training. Although both of these departments procured the websites associated with the web-based services we reviewed as part of this audit, neither department’s staff were trained on best practices for procuring an accessible website. Even at departments that procure their websites, we believe staff should receive some level of accessibility training to familiarize them with best practices for procuring an accessible website.

Best practice guidance regarding accessibility training advises that department personnel should attend such training. At the state level, CalTech’s SIMM recommends that agencies provide specific training to content creators and refers to federal and state resources for training in the areas of procurement, accessibility testing, and specific Section 508 standards. The U.S. Department of Justice, in a September 2012 report on Section 508 compliance among federal departments, recommended that agencies provide or facilitate the provision of more training to department Section 508 coordinators, agency personnel involved in the procurement process, and developers—including software, website, video, and multimedia developers.

One state department has offered training on the Section 508 standards and continues to offer training on accessible documents. Shortly after California adopted the recommendations of the Information Organization, Usability, Currency, and Accessibility Working Group (IOUCA) in July 2006, Rehabilitation offered several training sessions to state web professionals on Section 508 web accessibility requirements. It does not appear that attendance at these trainings was required, and Rehabilitation’s chief information officer stated that the classes were free to attend. Training materials from the classes show that the training covered the Section 508 standards and how to achieve or maintain compliance. The chief information officer also indicated that Rehabilitation continues to offer training to all departments at a small cost about how to ensure that documents posted to department websites are accessible to persons with disabilities. He stated that in 2013 and 2014, the department provided 62 such training classes for 697 attendees.

Given the level and frequency of training that we observed at each department we reviewed and the type of accessibility standards violations that we discussed in Chapter 1, we believe that California would benefit from a requirement that all relevant state personnel receive training on web accessibility on a regular basis. In doing so, California could adopt a model similar to the federal model for technical assistance with Section 508 standards. At the federal level, the U.S. General Services Administration is responsible for providing federal agencies with technical assistance regarding Section 508 requirements. If the Legislature required training for state employees involved in developing department websites and web-based services, it could assign CalTech the responsibility for coordinating and offering the training sessions. CalTech is in a unique position to coordinate and offer training, as it currently publishes guidance on how to incorporate Section 508 standards into the development cycle of an information technology project. Training could reinforce the importance of addressing accessibility requirements during key stages in the development of websites and web-based services and would help ensure that state employees receive training on current best practices in web accessibility. A web consulting supervisor at CalTech stated that the training that CalTech provides on web accessibility is limited to hosting Rehabilitation subject-matter experts, which CalTech does at some quarterly webmaster group meetings and on an ad hoc basis. Rehabilitation’s chief information officer indicated that his department would welcome the opportunity, with sufficient resources, to provide regular web accessibility training to state personnel. Although CalTech and Rehabilitation could partner to offer such training in the future, CalTech is California’s lead agency for information technology and a logical department to take the lead in coordinating and offering web accessibility training in consultation with other departments.

Further, although the website it maintains for state web professionals includes information on web accessibility, it has been almost five years since CalTech issued its policy letter concerning accessible websites in July 2010. The issues we observed at the four departments we reviewed indicate that all state departments would benefit from a reminder from CalTech about the State’s web accessibility requirements. Those issues are likely to exist at a number of other state departments, and thus it would be beneficial to remind all state departments of their responsibilities in this area. The web services manager at CalTech agreed that there is value in reminding departments about their obligations.

Some Departments Could Enhance Their Websites’ Accessibility by Listing Additional Usability Features

Popular web browsers offer standard features that enhance users’ ability to access and navigate websites. Generally, these features allow users to take actions such as magnifying a website’s content; adjusting the font, size, and color of a website’s text; and more quickly navigating within and among web pages using the keyboard. These specific features are in some cases related to, but distinct from, steps departments must take to ensure that their websites comply with California’s web accessibility standards; rather, they are built into web browser software and are not specific to individual websites. Users with disabilities may not always be aware of the availability of these features or how to use them.

Despite the value these features can have for users, only one of the departments we visited, Covered California, had taken steps to describe these features on its website before our review. Covered California’s website also provides links directing users to additional information that is specific to individual web browsers, such as Apple Safari, Google Chrome, Microsoft Internet Explorer, and Mozilla Firefox (Firefox), which are four very popular web browsers. Covered California posted these features after our audit began but as part of a planned update to its website that the department completed in November 2014.

At the time we began our audit in November 2014, none of the other three departments’ websites included information on standard accessibility features or links to more specific browser features. At that time, Franchise Tax Board and CalHR had accessibility pages on their websites stating that the departments intend to comply with accessibility requirements, but they did not list any additional accessibility features like those previously described. Community Colleges’ online application included a technical support page, but that page did not include any information specific to website accessibility. At the time, CalHR’s web manager and the executive director at the Community Colleges technology center both stated that their respective sites did not list browser usability features because those features are not unique to the sites. According to Franchise Tax Board’s accessibility lead analyst, these features are not specific to the Franchise Tax Board’s website, and the main focus has been on the site’s web content and online applications, given the limited resources available.

Adding information on these browser features to state websites is a simple and low-cost way to increase the accessibility of those sites. As a result of our conversations with the departments about this issue, CalHR and Franchise Tax Board have already updated their websites to include standard keyboard navigation features as well as links to browser-specific information on additional features. When we asked Franchise Tax Board about these features, the accessibility lead analyst stated that adding this information was something she could quickly implement. Community Colleges has not made these additions to its online application despite having already developed a replacement for its current help page that includes all of this information. According to the chief technology officer for Community Colleges’ technology center, the technology center prefers to package changes to the website into larger releases rather than making small and frequent changes. He stated that the new version of the website’s help page was scheduled to be released in late May 2015. Because providing this information could benefit users of all state websites, we believe CalTech could enhance the overall accessibility of California’s online services by posting information on these browser features on the website it operates as a resource for state web professionals and adopting a policy that requires all state departments to feature this information on their websites. CalTech’s chief deputy director of policy stated that CalTech could accomplish this easily and efficiently.

Oversight of Web Accessibility Testing Is Necessary to Help Ensure That California State Websites Are Accessible to Persons With Disabilities

California needs better guidance and oversight to ensure that departments adequately test their websites for accessibility and thereby maintain websites that are accessible to users with disabilities. As discussed earlier in this chapter, three of the four departments we reviewed did not always regularly test their websites to ensure that updates they made to those sites were accessible, and they also used inconsistent methods for testing. Moreover, in some cases we found critical accessibility errors on updated portions of departments’ websites that the departments did not thoroughly test before they released those updates to the public. These shortcomings in departments’ approaches to accessibility testing demonstrate a need for a statewide formalized testing approach. Although CalTech has issued general guidance to departments that outlines a broad approach to accessibility testing, it can go further in its direction to departments. Specifically, it could issue a policy that sets a minimum standard for the testing approach used for all state websites.

Adopting a policy to require a standard testing approach would align California’s approach to web accessibility with a recommendation made by the IOUCA and with the World Wide Web Consortium’s (W3C) suggested testing approach. When it issued its recommendations for improving web accessibility in California in June 2006, the IOUCA recommended that a centralized body explore the use of a single testing tool that could be used across departments. The IOUCA stated that such a tool would likely boost compliance as well as provide a centralized method of reviewing accessibility and holding departments accountable. Further, the W3C recommends that organizations establish an accessibility support baseline, or minimum combination of operating systems, browsers, and assistive technologies that is expected to work with their websites, when evaluating the accessibility of the websites. An effective accessibility support baseline would include various forms of assistive technology to ensure that testing would consider the needs of users with a range of disabilities.

Given that three of the four departments we reviewed did not regularly test their websites for accessibility, we believe California would benefit from greater oversight of the accessibility testing conducted by all state governmental entities, including those not currently under the direction of CalTech. As the lead technology agency for the State, CalTech is the logical state entity to take on this responsibility. To facilitate CalTech’s ability to conduct such oversight, under this approach, state governmental entities should be required to report to CalTech about their accessibility testing methods and the frequency of their testing. These reports to CalTech should include certifications of the testing signed by the department’s highest-ranking technology officer and supporting documentation to demonstrate that the testing occurred and what the department did or plans to do about any problems it identified during testing. Once it had received these certifications, CalTech could then make a determination about the sufficiency of each department’s testing approach as well as the department’s approach to fixing identified issues and publish the results of its review online. Any such determination should be made against the criteria found in the statewide accessibility testing policy discussed earlier. If CalTech publicized its assessment of each department’s accessibility testing approach, departments would be motivated to follow that policy. According to CalTech’s deputy director of policy, CalTech does not currently have adequate resources to provide web accessibility training to relevant staff throughout the State, develop a standard testing approach that all government entities should use when testing for web accessibility, and assess the sufficiency of accessibility testing approaches. However, we believe that increased and standardized web accessibility testing across all state departments in combination with required training on web accessibility would increase the likelihood that state websites would be accessible to persons with disabilities who attempt to access critical information and services through state websites.

Recommendations

Legislature

To ensure that state governmental entities have a clearly identified resource for web accessibility training, the Legislature should amend state law to name CalTech as the lead agency responsible for providing training to state governmental entities on web accessibility issues, in consultation with Rehabilitation and other state departments as it determines necessary.

To ensure that governmental entity personnel have the information and tools necessary to develop and maintain accessible websites, the Legislature should require governmental entities to provide or obtain web accessibility training at least once every three years for staff involved in the procurement or development of websites or web-based services.

To help ensure that all state governmental entities appropriately test their websites for accessibility, the Legislature should direct all state governmental entities to report every other year to CalTech regarding the frequency and method of their web accessibility testing and their efforts to resolve accessibility issues they identify. Such reporting should include signed certifications from the highest-ranking technology officer at the governmental entity and documentation that supports the claimed testing as well as the entity’s effort to fix identified issues. Further, the Legislature should direct CalTech to assess the sufficiency of each governmental entity’s testing and remediation approach and publicize the results of its review online.

Departments

To ensure that updates to their websites are tested for accessibility, by July 31, 2015, Covered California, Community Colleges, and CalHR should develop and follow written test approaches that describe how and when changes to their websites will be reviewed. These plans should describe how the departments will include both automated and manual forms of accessibility testing.

To ensure that it can accurately track whether accessibility issues found during testing have been resolved, Community Colleges should direct its vendor to more clearly and consistently document when the fix for an accessibility defect has been implemented in the live version of the online application.

To ensure that it can adequately track the results of its accessibility testing, by July 31, 2015, CalHR should develop tracking tools that will allow it to document its testing efforts. At a minimum, these tools should track what portions of its jobs site were tested, what errors were found, and whether and when those errors were addressed.

To ensure that individuals have a wider variety of contact information available to them for reporting problems with website accessibility, by July 31, 2015, Community Colleges and Covered California should update their accessibility pages to include all methods of communication that state requirements mandate for other departments.

To ensure that it appropriately addresses any complaints it receives related to web accessibility, Covered California should develop procedures to regularly review the complaints it receives at its accessibility email address and address any web accessibility complaints in a timely fashion.

So that complaints from the public do not go unaddressed, by July 31, 2015, CalHR should develop procedures for addressing complaints about the accessibility of its website and methods for tracking the complaints it receives and their resolution.

To ensure that all state departments are reminded about web accessibility requirements and best practices, by July 31, 2015, CalTech should issue an official reminder that directs state departments to key policy documents and the SIMM for additional information about how to meet their obligation to provide accessible websites.

To enhance the overall accessibility of its website, by July 31, 2015, Community Colleges should list general web browser usability features on its websites after verifying that its sites are compatible with those features. Additionally, Community Colleges should add links to its website directing users to browser-specific usability information for these four popular web browsers: Apple Safari, Google Chrome, Microsoft Internet Explorer, and Firefox.

To help state websites achieve a higher degree of overall accessibility, by July 31, 2015, CalTech should post standard browser usability features and links to further information on its resources website, and direct all state departments to include this information on their websites.

To standardize California’s approach to web accessibility testing, CalTech should issue a policy that specifies the method by which state departments should conduct web accessibility testing. This policy should include information about a minimum combination of operating systems, browsers, and assistive technologies that should be used during testing.

Compliance With Audit Standards and Staff Listing

We conducted this audit under the authority vested in the California State Auditor by Section 8543 et seq. of the California Government Code and according to generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives specified in the scope section of the report. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Respectfully submitted,

ELAINE M. HOWLE, CPA
State Auditor

Date: June 2, 2015

Staff:
John Billington, Audit Principal
Bob Harris, MPP
Michaela Kretzner, MPP
Mark Reinardy, MPP
Julie Nguyen

Web Accessibility Consultant: Deque Systems, Inc.

Legal Counsel: Stephanie Ramirez-Ridgeway, Sr. Legal Counsel

For questions regarding the contents of this report, please contact Margarita Fernández, Chief of Public Affairs, at 916.445.0255.

Footnotes

¹¹ Verification and validation is a process used to ensure that a system conforms to requirements and satisfies user needs.Go back to text

¹² According to the supervisor over web development at CalHR, the current version of the jobs site was released in January 2012, before CalHR existed. However, the same team that built the jobs site while employed at the California State Personnel Board continues to work on the site as employees of CalHR, although some staff have left CalHR since the development phase.Go back to text

¹³ Covered California did have a separate contractor perform manual testing in October 2014 with a screen reader on its Voter Registration page as part of an update to its website. However, this was an isolated review of a single page.Go back to text

¹⁴ For the purposes of this report, we use the term violation to mean a violation of an accessibility standard a department’s website was expected to conform to because of state law or a grant or contract agreement.Go back to text

¹⁵ This website is located at webtools.ca.gov.Go back to text