Dialog Testing Starts With Disabled Users, Not QA Pipelines

Marcus makes a compelling case for better testing infrastructure around dialog dismissal. But the operational gap he identifies has a root cause that better tooling alone won't fix: disabled users aren't in the room when these decisions get made.

In their analysis of dialog testing failures, Marcus correctly diagnoses that automated tools catch only 30-40% of accessibility issues and that dialog components are particularly resistant to automated evaluation. The prescription that follows — better tooling, more sophisticated CI/CD integration, clearer internal protocols — is reasonable. It's also insufficient. Organizations that have genuinely closed the dialog accessibility gap haven't done it primarily through infrastructure investment. They've done it by structuring disabled user input into their development process at the point where dismissal behavior decisions are actually made.

This distinction matters because it changes where you invest first.

What Testing Infrastructure Actually Measures

The framing of dialog dismissal as a testing infrastructure problem implicitly positions disabled users as end-stage validators — people who confirm whether a decision made upstream was correct. That's a costly role for them to play and a structurally late intervention for the organization. By the time a dialog component reaches QA, the dismissal behavior has already been implemented, the component has been integrated into the design system, and changing it carries real engineering cost.

Research from the Nielsen Norman Group on inclusive design (opens in new window) consistently finds that accessibility issues caught during requirements and design phases cost a fraction of what they cost to remediate post-development. For dialog components specifically, where the accessibility question is fundamentally about user intent and consequence — what does it mean for this particular user if this dialog closes unexpectedly? — that upstream involvement isn't just more efficient. It's the only way to get the right answer.

Automated tools, as Marcus notes, cannot evaluate whether a dialog's dismissal behavior is appropriate for its content. Neither can a QA engineer who doesn't use a screen reader daily. Neither, for that matter, can a WCAG practitioner working from criteria alone. The contextual judgment that SC 3.2.2 (opens in new window) requires — determining what constitutes an unexpected change of context for a specific interaction — is a judgment that benefits enormously from people who navigate these interactions as a matter of daily life.

The Structural Problem With "User Testing" as Practiced

Most organizations that do involve disabled users in accessibility testing do so through periodic usability studies — quarterly sessions, pre-launch audits, or annual reviews. This is better than nothing. It is not community input in any meaningful sense.

Community input means disabled users are consulted during requirements gathering, not just testing. It means screen reader users have a voice in whether Escape key dismissal is the right default for a given dialog type before that default is built. It means users with motor disabilities are part of the conversation about whether click-outside-to-dismiss creates unacceptable error rates for them — a question that research on motor accessibility from the Web Accessibility Initiative (opens in new window) makes clear is real and significant.

The ADA National Network (opens in new window) has documented repeatedly that organizations with formal disability advisory structures — not just user research panels — produce more consistently accessible products. The difference isn't that advisory structures catch more bugs. It's that they shift the question from "did we implement the standard correctly?" to "does this work for the people who need it to work?"

Those are related questions. They are not the same question.

Where Marcus's Framework Is Right and Where It Stops Short

The original testing infrastructure analysis identifies something genuinely important: most organizations cannot reliably catch dialog accessibility failures before they ship because they lack the internal capacity to evaluate context-dependent behavior. That's accurate. The Section 508 program's guidance on testing methodologies (opens in new window) acknowledges this same gap — that manual testing by trained evaluators is necessary precisely because automated tools cannot assess contextual appropriateness.

But trained evaluators working from WCAG criteria are still working from an abstraction. They're applying a framework developed through a process that, while it included disability community input, produces generalized guidance rather than context-specific answers. When a practitioner is trying to determine whether a cookie consent dialog's Escape key behavior constitutes an unexpected change of context under SC 3.2.2, the most direct evidence available is whether users who navigate primarily by keyboard find it unexpected — and that evidence comes from those users, not from the criterion.

The DOJ's updated guidance on web accessibility under Title II (opens in new window) emphasizes effective communication as a core standard. Effective communication is a user-defined concept. Whether a dialog's dismissal behavior enables or disrupts effective communication for a screen reader user is something that screen reader user is best positioned to evaluate.

The Investment Sequence Question

Organizations with limited accessibility budgets face a real sequencing question. Marcus's argument implies that testing infrastructure — better tooling, more rigorous CI/CD integration, clearer internal protocols — is the primary investment. My argument is that community input structures should come first, because they change what you're testing for.

An organization that has structured disabled user input into its design process knows which dialog types its users find most problematic. It knows whether its particular user base navigates primarily by keyboard, switch access, or voice control — and therefore which dismissal behaviors create the highest error rates. It has qualitative data that makes its testing criteria more precise and its automated tool configuration more targeted.

That's not an argument against investing in testing infrastructure. It's an argument that testing infrastructure built without community input is optimized for the wrong things. You end up with rigorous processes for catching ARIA attribute failures while the contextual judgment questions — the ones Marcus correctly identifies as the hard ones — remain unaddressed.

Our approach at this publication, which you can read more about at our methodology page, has always been that accessibility coverage should center the people most affected by these decisions. That principle applies to coverage, but it applies equally to the organizational practices we're analyzing.

What Community Input Actually Looks Like

For dialog components specifically, meaningful community input doesn't require a large formal program. It requires:

Disabled users involved in design reviews when dialog patterns are being established — not tested after implementation, but consulted during the decision about what behavior to implement. Screen reader users, keyboard-only users, and users with motor disabilities have meaningfully different relationships with dialog dismissal, and those differences should inform pattern decisions before they're built.

Feedback channels that reach the people making component decisions, not just the people doing QA. This is an organizational structure question as much as a process question. If disabled user feedback goes to a customer support queue and never reaches the design system team, it doesn't function as community input.

Recognition that the disability community is not monolithic. A dismissal behavior that works well for screen reader users may create problems for users with motor disabilities. Building on this framework of context-dependent evaluation, the relevant context includes which user populations are most likely to encounter a given dialog type — and that's information that comes from community engagement, not from WCAG criteria alone.

The testing infrastructure Marcus describes is necessary. It is not sufficient. The organizations that have actually solved the dialog dismissal problem at scale have done it by making disabled users part of the decision, not just the evaluation.

Keisha covers accessibility policy and community practice. Her methodology and sourcing approach is detailed at her author page and the publication's editorial standards.