Automated vs manual accessibility testing

An automated tool loads a page, reads its code and runs a list of rules. Each rule is something a computer can decide on its own: does this image have an alt attribute, does this input have a label, is the contrast between this text and its background at least 4.5 to 1. Most free tools use one of a few engines. axe-core, from Deque, is open source and also runs inside Lighthouse. WAVE comes from WebAIM.

You can run them in three ways:

• In your browser, with an extension such as axe DevTools or WAVE, on the page you are looking at. This also works on pages behind a login or a password page.
• With an online checker, where you paste a public address and get a report.
• In code, where a developer runs the checks every time the theme changes.

Most tools give three kinds of result: a clear failure, an item that “needs review” by a person, and a pass. The needs review list is easy to ignore and often holds the important problems.

You will see very different numbers quoted for this. They are measuring different things. Here are three that come from published studies and data, with what each one measures.

The GOV.UK team built a page with 143 known barriers and ran 10 tools on it, most of them free. The best single tool found about four in ten, and together the tools still missed 29%. Deque, which makes axe, looked at more than 2,000 of its own audits and found automated tests covered 57% of issues by volume. Both can be true. The most common problems, like low contrast and missing labels, are the easiest to detect, so counting every instance gives a higher number than counting kinds of barrier.

WebAIM's yearly scan of one million home pages shows how common the detectable problems are: 96% of the errors it found fell into six types (low contrast text, missing alt text, missing form labels, empty links, empty buttons and missing page language). WebAIM also warns that the absence of detected errors does not mean a page is accessible.

This table covers the WCAG 2.2 AA checks that matter most on a store. “Partly” means software can find some failures but a person has to confirm the rest.

For the full list of WCAG 2.2 AA criteria in plain words, see WCAG 2.2 AA explained.

The W3C puts it simply: evaluation tools “can not determine accessibility, they can only assist in doing so”. It also warns that tools sometimes give false or misleading results. The US Justice Department's web guidance says the same thing for businesses: automated checkers are helpful but need to be used carefully, a clean report does not mean a site is accessible, and pairing them with a manual check gives a better picture.

Scores have the same limit. The Lighthouse accessibility score is a weighted average of automated pass or fail checks, and its manual checks, such as tab order and focus traps, do not affect the score. Shopify asks Theme Store themes for an average Lighthouse accessibility score of at least 90 on the home, collection and product pages. A theme can meet that and still have a quick view pop-up that traps the keyboard. Our automated tests of free Shopify themes also list what to check by hand before you pick one.

Automation is best at catching the same mistakes over and over, cheaply. Run it:

• on every template after a theme update or a new theme
• after you install an app that adds something to the storefront, such as reviews or a pop-up
• after a big catalogue upload, to catch products with no alt text
• before a sale, on the banners and landing pages you built for it

If you or your developer work on the theme in code, the checks can run on every change. Playwright, a free browser testing tool, has an official guide for this. The test below opens a product page and fails if axe finds any WCAG A or AA problems it can detect.

import { test, expect } from "@playwright/test";
import AxeBuilder from "@axe-core/playwright";

test("product page has no detectable WCAG issues", async ({ page }) => {
  await page.goto("https://your-store.com/products/linen-shirt");
  const results = await new AxeBuilder({ page })
    .withTags(["wcag2a", "wcag2aa", "wcag21a", "wcag21aa", "wcag22aa"])
    .analyze();
  expect(results.violations).toEqual([]);
});

A passing test means only that axe found nothing it can detect. The rest of this page is still needed.

Manual testing sounds technical, but most of it is using your own store in a different way. These five tests cover the parts software misses. Do them on the home page, one collection, one product page with variants, the cart and the checkout. The ADA website checklist has the same tests as a list you can tick off.

The best manual test is watching a disabled shopper use your store. The W3C says involving people with disabilities throughout design and development gives better results. Even one session often shows problems a checklist missed.

These are the parts of a store where a page can look fine to a scanner and still block a shopper. Check them by hand first.

• Cart drawers. When the drawer opens, focus should move into it, and Esc should close it and return focus to the cart button. Most scanners test the page as it loads, with the drawer closed.
• Variant pickers and swatches. Custom swatches built from plain elements can look like buttons but cannot be reached with Tab, and may not say which option is selected.
• Quick view and pop-ups. Email sign-up and discount pop-ups from apps often trap the keyboard or have a close button with no name.
• Sticky headers. When you tab down the page, the focused link can slide behind the header. That fails WCAG 2.4.11 Focus Not Obscured.
• Text on images. A heading over a hero photo may pass a contrast check against the background colour behind the image, while the photo itself makes it hard to read. Use the contrast checker with the real colours.
• Added to cart messages. If nothing is read out after pressing Add to cart, a screen reader user does not know it worked.

If you need a formal report, our guide to accessibility audits covers what a full audit includes, what it costs and what to ask a vendor.