What Automated Accessibility Testing Can and Can't Catch

Applies to: all WCAG 2.2 success criteria · Automated checkers such as axe-core (which our monitor uses), WAVE and Lighthouse

Automated accessibility checkers are fast and repeatable, and they find real problems. They also miss a lot. The W3C puts it plainly: tools "cannot check all accessibility aspects automatically. Human judgement is required," and they "can not determine accessibility, they can only assist in doing so" (W3C WAI). This guide explains where the line falls, so you know what a clean scan does and doesn't tell you.

Two ways to measure "coverage"

People quote very different numbers for automated testing because they measure different things.

• By issue volume: Deque reports that in its own sample of first-time audits (13,000+ pages and page states, nearly 300,000 issues), its automated axe-core tests found 57.38% of issues by volume (Deque report). That is Deque's figure for its sample, not a detection rate for any one tool, ours included. It is high partly because a few very common issue types, such as low contrast, make up a large share of all issues.
• By success criteria: in the same report, Deque found automated issues for only 16 of the 50 WCAG 2.1 Level AA success criteria in its sample (Deque report).
• Our own rule set: of the 55 WCAG 2.2 Level A and AA success criteria, at least one rule we run partly checks 21, and no rule we run checks the other 34. The full list, generated from our axe-core version's rule data, is on What our automated checks cover.

Both views are true at once: automated tools can find a large share of the instances of common issues, but there are whole kinds of barrier they can't see. A large share of issues still needs manual testing.

What automated tools are good at

These are mostly yes/no questions about the code, which a rules engine can answer reliably:

• Missing text alternatives: an <img> with no alt attribute, an icon button with no accessible name (SC 1.1.1, SC 4.1.2). See missing alt text, empty links and empty buttons and missing button names.
• Text contrast where the text and background colours can be computed (SC 1.4.3). See colour contrast.
• Form fields with no programmatic label (SC 1.3.1, SC 4.1.2). See form labels.
• Missing page language (SC 3.1.1; see page language) and a missing page title (SC 2.4.2).
• Broken ARIA: invalid roles, required children missing, aria-hidden on focusable elements.
• Zoom blocked by a <meta name="viewport"> that disables scaling (SC 1.4.4).

What automated tools can't judge (or only partly)

These need a person, because the answer depends on meaning, behaviour or context:

• Whether alt text is accurate. A scanner can see that alt exists. It can't tell whether alt="IMG_0423.jpg" or an unreviewed AI suggestion actually describes the image (SC 1.1.1).
• Whether link and heading text makes sense. "Click here" has a name, so it passes a name check. Whether headings and labels describe their content is SC 2.4.6, which no rule we run checks. See heading structure.
• Keyboard use: whether every control can be reached and operated with a keyboard, whether focus gets trapped, and whether the focus order is logical (SC 2.1.1, SC 2.1.2, SC 2.4.3).
• Visible focus, and whether sticky headers or banners hide the focused element (SC 2.4.7, SC 2.4.11).
• Non-text contrast: input borders, focus indicators and meaningful icons against their surroundings (SC 1.4.11).
• Text over images or gradients, where the background colour can't be computed.
• Captions and audio description quality: a tool may detect that captions are missing on a <video>; it can't tell whether they're accurate (SC 1.2.2).
• Reflow and zoom: whether content works at 320 CSS pixels wide or 400% zoom without horizontal scrolling (SC 1.4.10).
• Error messages and status updates: whether form errors are identified in text and whether "added to cart"-type messages are announced (SC 3.3.1, SC 4.1.3).
• Logins and multi-step flows: CAPTCHAs, cognitive tests at sign-in, and asking for the same information twice (SC 3.3.8, SC 3.3.7).

Automated tools can also report false or misleading results (W3C WAI), so a reported issue is worth confirming before you change working code.

What a scan only sees

Even for the checks it can run, a scan only tests what it loads:

• One state of one page. Menus, modals, accordions and tabs that are closed during the scan aren't fully tested. Neither are hover and focus states.
• Public pages. Pages behind a login, checkout steps and account areas need their own testing.
• The page as served at scan time. Content added later by editors, A/B tests or third-party widgets can change the result. That's the case for monitoring rather than a one-off scan: it shows when something new breaks.

A sensible way to combine them

1. Run automated checks first and fix what they find. Deque reports that in its sample the top five issue categories made up over 78% of issues, and most of those were found by automated tests (Deque report).
2. Monitor key templates (home, product or article, contact, checkout) so regressions show up.
3. Do the manual checklist below on each template, especially after redesigns.
4. Test with assistive technology (a screen reader, keyboard only, browser zoom) on your most important user journeys.
5. Bring in an accessibility specialist for an audit when the stakes are high. Laws and contracts differ by country and sector; check with a lawyer about what applies to you.

What automated tools can and can't tell you: Automated checkers (including axe-core, which our monitor uses) can flag many instances of common issues, but they cannot judge everything. A human still has to check meaning and context. For scale: Deque reports that in its own sample of first-time audits, its automated tests found 57.38% of issues by volume (Deque report). That is Deque's figure for its sample, not a detection rate for any one tool, and it means a large share of issues still needs manual testing. A scan with no issues reported does not mean a page meets WCAG. Automated tests find only some accessibility issues; manual testing is still required. Not legal advice.

Manual checklist

• Tab through the whole page with the keyboard: every link, button and field is reachable and works, and focus never gets stuck
• The focus indicator is always visible and isn't hidden behind sticky headers or cookie banners
• Focus order follows the visual and reading order
• Alt text, link text, button text and headings make sense out of context
• Open every menu, modal, accordion and tab, then re-check them
• Zoom to 400% (or a 320px-wide window): content reflows without horizontal scrolling or lost content
• Submit forms with errors: each error is identified in text and explained
• Videos have accurate captions; audio has a transcript
• Information isn't conveyed by colour alone (SC 1.4.1)
• Try key journeys (sign up, contact, checkout) with a screen reader