Supervisory Unit for Web Accessibility has published the source code of its testing software

The publicly-distributed Salvia software is largely the same as the version of Salvia used by the Regional State Administrative Agency as part of the accessibility monitoring. With regard to accessibility testing, the software versions are consistent, and the public version produces the same observations of the websites to be tested as those used by the supervisory authority. In other words, anyone can use Salvia to check the accessibility of the website or to verify the observations made by the Regional State Administrative Agency.

Read more: Salvia software as a solution for accessibility monitoring

A note on security and licensing

It is important to be aware that the now released version of Salvia software itself does not include user rights management or other security features. The program code and the available Docker images are intended first and foremost for trial use by software developers and for research in a safe operating environment. Although the software can be installed  in various cloud services or even on a user’s work machine, using the Docker tool, information security must be ensured separately and with sufficient expertise in ICT architecture.

Unlike other software libraries used by Salvia, using the MS SQL Server requires a license. Using the SQL Server for development and testing purposes is free, but production use is subject to a fee.

How Salvia works

The software uses the QualWeb core, developed by the University of Lisbon, as an evaluation engine to test accessibility. It tests each website with 47 ACT tests. These ACT Tests (“Accessibility Conformance Test”) are systematic descriptions of how compliance with accessibility requirements can be inspected. They were developed in jointly by W3C and software developers and are partly funded by the European Union. The QualWeb software currently impelements 72 different ACT tests, but Salvia uses only those related to the WCAG criteria’s A and AA conformance levels.

The ACT tests have headings to describe an individual accessibility requirement or claim, for example, “the button must have an accessible name”. Pass means that compliance with the requirement was verified and fail that the requirement was not met.

Some ACT tests are such that they can be run in an automated manner – for example, measuring text contrast or ensuring that the html element contains a lang attribute indicating the language of the text. At least for the time being, some of the tests require a separate expert assessment. Although the test may detect that a video contains a soundtrack or a subtitle track, a software is unable to assess whether the video’s subtitles correspond appropriately to the relevant information from the soundtrack.

When a website does not pass automated testing this is therefore an indication of issues in accessibility, but passing ACT tests successfully does not yet guarantee that the website could not have other accessibility issues.

Introduction of Salvia software

The source code for the Salvia software is available in the GitHub repository for accessibility supervisory unit at https://github.com/accessibility-fi/salvia. User can download the source code to their own computer by cloning the repository using the git version control or on a one-time-basis by selecting “code” and then “download ZIP” from the drop-down menu.

The Salvia repository contains the components required by the Docker tool. Docker is a software solution that virtualises the operating systems and other dependencies needed to run the software. This way, for example, Salvia, which originally uses Microsoft Azure products, can be installed to run on a standard computer as well as on other cloud services. However, when introducing the use of Salvia it is a good idea to take into account, such things as its memory requirements.

Instructions for deployment can be found in English in the application’s readme file under “Building Docker images and starting the containers and services”. Once the Docker containers have started, Salvia can be used in a browser at http://localhost:3000/.

Testing the accessibility of websites with Salvia

Salvia has two accessibility testing functionalities: Automatic and manual testing. The manual test can be used to check everything in the same way as the automatic test, but the manual test can also be used to test websites and pages that cannot be checked with the automatic test. However, manual testing is more time-consuming.

Testing of desktop and mobile viewports

The desktop (1920 px X 1080 px) and the mobile viewports (360 px X 800 px) can be inspected separately for each website.  When requesting a server on the website to be inspected, the Salvia web crawler appears on the server in accordance with the selected testing method, either as a desktop or mobile device browser. It will depend on the technical implementation of each  website, whether it has implemented the mobile viewport solely with responsive CSS or JavaScript solutions, or if it will return completely different content to your desktop and mobile browser requests.

Automatic Testing

Automatic testing allows users to test websites that do not require login and that have a fixed URL for each view that is being tested.

Testing is carried out in two phases. First, Salvia’s web crawler searches for all the website’s subpages. In the second phase, the user selects the ones to be inspected and starts the test.

Site selection

The user enters the website URL in the field, selects whether they want to test the desktop or mobile viewport, as well as the “depth” and “width” of the pages to be searched. When Salvia starts searching for pages, it starts from the home page returned by the given URL and picks links to all other subpages on the site. After this, the program searches for all new links on these pages, and then also links on these linked pages. “Depth” on the form sets how deep will the crawler go in its search for pages. The last field in the form “width” determines the maximum number of pages Salvia will return.

Selecting pages for testing

Pages can be selected for evaluation completely on the basis on the search performed by the automatic test, but if the purpose is to check only a predetermined number of pages, it may be more productive to go through the site itself and select pages for testing at your own discretion. At the bottom of the selection list is also a field from which you can add an optional page to be included in the test. This page can be one that Salvia could not find in the first phase search, or it can also exist in a completely different domain.

Finally, click the “Start” button. Once the test is complete, the test report can be found on the Completed Tests page.

Manual testing

Manual testing allows one to check sites that require login (provided that the person performing the test has access to the service) and so-called single-page app sites, where user’s view may change a lot, but the browser’s address bar contains the same URL for all views.  In addition, very ordinary sites may have, for example, menus, chat functions, or other functionalities that only emerge as the result of user interaction. A manual test model can be used to test these too.

Manual testing uses the QualWeb Chrome browser plugin, which allows one to inspect any website view that is currently visible to the browser. The technical report for a test completed with the plugin needs to be downloaded and then uploaded to Salvia including information which page and viewport the added report applies to. Once all reports are downloaded, Salvia will combine them into a single entity. The final PDF report available in Salvia will not differ much from the one done with automatic testing.

Creating a new test

A new test is created by adding the URL of the site to be evaluated. In addition, select whether the test will be performed on desktop or mobile viewports. In the manual test, this parameter is only informative and the tester must remember to use the corresponding emulator settings in the browser when performing the test.

Pre-setting the desktop and mobile viewport for the browser emulator

Chrome DevTools provide a feature for viewing a web page by simulating or emulating various user agents. The user can select a view size of “responsive” and adjust the dimensions and view its effects on the content layout. The browser also has pre-installed display sizes for a wide range of mobile phone models.

One can also add and save their own device and user agent settings. For manual testing, the user should add their own emulated devices for Salvia desktop testing and Salvia mobile tests. Go to Chrome DevTools Settings and select “Devices” and “Add new device”.

The dimensions of the Salvia desktop viewport should be set to 1920 x 1080, the user agent to “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:22.0) Gecko/20100101 Firefox/22.0”, “Desktop” should be set as type in the Select menu.

The dimensions of the Salvia mobile viewport  should be set to 360 x 800, the user agent to ” Mozilla/5.0 (Linux; U; Android 2.2; en-us; DROID2 GLOBAL Build/S273) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1” and ”Mobile” should be set as type in the Select menu.

In the future, when performing manual testing with Salvia, the user must start the developer tools and emulator and select either the Salvia desktop or mobile viewport from the emulator menu. It is important to use the browser’s emulator and these settings for testing so that testing is actually performed with the same settings that is claimed on the test report.

Activating the QualWeb plugin

The QualWeb plugin is available for installation to the Chrome browser from the Google App Store. Once installed, the add-on appears as a new tab in your browser’s developer tools.

To test a single page open the page, launch the emulator in browser developer tools, and select either Salvia desktop or mobile from the device options, depending on which viewport type you are testing. After this, any actions on the website, such as logging in to the service or the selection of user interface, are carried out. Open the QualWeb tab in the developer tools , select “ACT Rules” as the tests to be performed and click “evaluate”. After the tool completes the test, click the “export report” button in the top left corner to download the json report produced by the plugin.

To test the next view, click the “repeat evaluation” button in the upper left corner.

The results of the tests are uploaded to the Salvia using the “add a page” button. The URL of the evaluated page is entered on the form. If you want to test several views from the same URL, add an additional description, for example, what the user has done or how the view differs from the default view of the URL in question. However, it is worth remembering that, unlike the other content of the PDF report, this description is not translated into other languages.

Once all desired pages have been evaluated and their test reports have been added to Salvia, choose “create test”.

Browsing Inspection Reports

In this view, all tests are grouped under their domain. For each performed test one can see the time the test was run, whether it was tested in desktop or mobile viewport, and the number of pages tested.  Next to this, you can see how many test types were completed successfully and how many failed. In addition, you can see how many test results remained uncertain. It is worth examining the matter in more detail in the case of these uncertain results. Success and failure have been viewed on a page-by-page basis and the number of passed and failed tests on all tested pages has been summed up.

There are three buttons next to the numbers. Click the JSON button to download the original machine-readable material. The same material can be downloaded in PDF format  for example for archiving purposes by clicking the JSON (PDF) button. A human-readable summary of the report can be downloaded from the PDF button.

In the “Settings” view of the Salvia main menu, you can select the language in which the PDF report will be printed. One report can be printed in several different languages if needed.

All reports of completed tests can be found under the “completed tests” menu. If Salvia is used via Docker and you want to save the old test reports for later use, pay attention to the command you use to shut down the Docker containers. For more information, see the readme description and https://docs.docker.com/engine/reference/commandline/compose_down/.

PDF Reports

The PDF reports downloaded from Salvia have been developed for the use of the Regional State Administrative Agency’s supervisory work, and they are primarily intended for technical developers of services. It may be a good idea to include a separate general description of the matter and of made observations for recipients who are not familiar with web technologies.

The shortest reports are under 10 pages long, but if there are many issues in the service and they are recure on several pages of the site, the length of the reports may increase to hundreds or thousands of pages. This is one of the reasons why a PDF report only specifies the elements where clear accessibility errors have been observed. Uncertain cases are not specified. If Salvia is used by someone in a role other than that of a supervisory authority or for checking one’s own digital service, it may be useful to modify the PDF document printing in the source code.