WCAG Auditing Isn't a Checklist. It's a Research Method.

Two auditors evaluate the same interface against the same Success Criterion. One flags a failure. One passes it. Both followed WCAG. This isn't a hypothetical — it's a structural problem baked into how accessibility evaluation works, and it costs organizations real money, real credibility, and — most critically — real access for disabled people.

Adrian Roselli's "Evaluating a Success Criterion" (opens in new window) is a short piece that does something deceptively important: it describes a research methodology for WCAG evaluation, not just a reading list. The distinction matters. A checklist tells you what to look at. A methodology tells you how to think about what you're seeing. Most organizations have the first. Almost none have the second.

The Gap Between Normative Text and Real Interfaces

WCAG success criteria were written to be technology-agnostic. That was a deliberate design choice — the W3C (opens in new window) wanted guidance that would survive browser generations, framework churn, and interface paradigms that hadn't been invented yet. The tradeoff is that normative text is abstract almost by necessity.

The problem surfaces the moment you try to apply WCAG 2.2 (opens in new window) to something like a custom date picker built in a JavaScript framework, a drag-and-drop kanban board, or a multi-step modal workflow. The SC was written before these patterns existed. The normative text doesn't name them. The Understanding document might gesture toward them. The Techniques documents may not have caught up.

This is where auditor judgment fills the gap — and where inconsistency enters the picture. As our research on testing methodology shows, this gap is structural, not a matter of individual auditor skill. The methodology problem runs deeper than any single practitioner.

Roselli's Seven Steps: A Closer Look

The framework Roselli lays out is worth unpacking because each step represents a different type of evidence:

| Step | Source | Evidence Type | Caution | |------|--------|---------------|---------| | 1. Normative WCAG text | w3.org/TR/WCAG22 | Binding requirement | Read from w3.org only — not wcag.info or wcag.com | | 2. Understanding documents | W3C WAI | Interpretive guidance | Non-normative but authoritative | | 3. Techniques documents | W3C WAI | Known patterns | Some are outdated; a single technique may apply to multiple SCs | | 4. GitHub Issues | W3C WCAG repo | Working group decisions | Prioritize closed issues; watch for spam | | 5. GitHub Pull Requests | W3C WCAG repo | Editorial deliberation | Unmerged PRs often contain the most useful debate | | 6. GitHub Discussions | W3C WCAG repo | Community signals | Sparse; comment if your question is unanswered | | 7. Trusted expert blogs | Various | Practitioner interpretation | Evaluate for agenda and evidence quality |

The sequence isn't arbitrary. It moves from binding to interpretive to deliberative to community. Each layer adds context when the previous layer is insufficient. This is how legal interpretation works — statute first, then legislative history, then case law, then commentary. Roselli is essentially describing statutory interpretation applied to a technical standard.

One detail that deserves more attention: his warning about sources masquerading as W3C material. Sites like wcag.info and wcag.com are not W3C properties. They may contain useful information, but they can also embed interpretations that serve vendor or commercial interests. For organizations trying to establish defensible audit processes, sourcing matters. A finding grounded in w3.org documentation holds up differently than one grounded in a vendor's interpretation of a standard.

The Adversarial Conformance Problem

Roselli surfaces something the accessibility field doesn't discuss loudly enough: the difference between passing a Success Criterion and actually removing a barrier. He calls out "adversarial or malicious conformance" — the practice of finding loopholes or gaps in SC language to justify a technically passing implementation that still fails users.

This matters enormously for organizations that procure accessibility audits. An audit report showing "pass" against every tested SC can coexist with an interface that screen reader users genuinely cannot use. The Settlement Trap research documents exactly this pattern in post-litigation contexts: organizations achieve paper compliance without achieving functional access.

The corrective Roselli offers is straightforward but demanding: test everything anyway. Not because the research is wrong, but because new browser releases introduce regressions, assistive technology support changes, and user expectations shift. A pass documented six months ago may not hold today. This is why accessibility isn't a project with an end date — it's an operational discipline.

What This Means for Audit Procurement

For organizations buying accessibility audits — and most Title III businesses and Title II entities eventually do — Roselli's framework implies several concrete questions worth asking vendors:

On methodology: Does the auditor distinguish between normative requirements and interpretive guidance when documenting findings? A finding that cites only an Understanding document is epistemologically different from one grounded in normative SC text.

On currency: How does the auditor account for the gap between when a Technique was written and when the interface pattern being tested was developed? Techniques documents, as Roselli notes, can reference superseded technologies.

On judgment calls: When an auditor makes a judgment call on an ambiguous SC, what's their documentation trail? Can they show the GitHub issue thread, the working group discussion, the expert consensus they relied on?

On adversarial conformance: Does the audit methodology explicitly test for functional usability beyond technical pass/fail? Our research on automated versus manual testing shows that automated tools catch at most 37% of actual barriers — but even manual audits can miss functional failures if the auditor is optimizing for technical conformance rather than user experience.

The Standard Is a Means, Not an End

WCAG is a technical standard, but the reason it exists is to ensure that disabled people can access digital content and services on equal terms. The standards fragmentation research documents how organizations can become so absorbed in navigating overlapping requirements that the human access question recedes.

Roselli's methodology is partly a corrective to that drift. The final step — checking trusted expert blogs, understanding working group intent, being willing to break from the W3C position when a remediation creates a worse experience — is a reminder that the standard is a means, not an end. The goal is access. The SC is the tool.

For practitioners, this means building the research habit Roselli describes before you need it. The GitHub issues and PR threads are publicly available at github.com/w3c/wcag (opens in new window). The Understanding documents live at w3.org/WAI/WCAG22/Understanding/ (opens in new window). The normative spec is at w3.org/TR/WCAG22/ (opens in new window). None of this requires a paid subscription or a vendor relationship.

The auditors who produce defensible, useful findings aren't the ones who memorize the most SCs. They're the ones who know how to research the ones they haven't seen before — and who stay anchored to the question of whether a real person can actually use the thing they're evaluating.