Your regression suite is green. The deployment passes. Then a customer reports that they cannot complete the one workflow your business depends on.
Nothing “broke” in the conventional sense. The service responded. The API returned the expected payload. The UI test reached the confirmation screen. Yet the user can't tell whether the transaction succeeded, the interface is unusable on their device, or the workflow has left the underlying data in an inconsistent state.
This is not a coverage problem. It's an assertion problem.
The assertion gap
Most automation suites validate a narrow slice of system behaviour: an element exists, a response code is correct, a value changes, or a workflow reaches a particular state. Those assertions are necessary, but they are not sufficient evidence that the system is fit for purpose.
Assertion debt is the accumulating gap between what a system must do and what its automated checks can actually observe.
A test can assert that a payment endpoint returns 200 OK, while missing that the confirmation state is visually absent, the error message is ambiguous, the control is inaccessible, or the user can submit the payment twice. The suite remains green; the customer experience fails.
Coverage is not assurance
Coverage measures execution. It doesn't measure whether the right outcomes were observed.
A high-coverage suite can still have weak failure-mode coverage. The more useful questions are:
- Which business-critical outcomes have an explicit assertion?
- Which failures would turn a test red?
- Which failures would only be discovered by a customer, support ticket or incident?
- Which assertions would survive a refactor, browser update or design-system change?
- Which failures are technically detectable but invisible to the current assertion layer?
A test that checks whether a button is clickable is not equivalent to a test that proves the user can recognise the next action, understand the result and recover from an error.
Assertions at the wrong boundary
The common mistake is not a lack of tests. It’s placing assertions at the wrong boundary.
| Assertion layer |
Useful for
|
Blind to
|
|---|---|---|
| Unit tests |
Business rules, calculations, isolated functions
|
Integration, rendering, user interaction
|
| API / contract tests |
Service contracts, schemas, status codes
|
Visual output, accessibility, end-to-end state
|
| DOM assertions |
Element presence, text, attributes, enabled state
|
Layout, overlap, contrast, responsive behaviour
|
| E2E workflow tests |
Task completion and navigation
|
Partial failure, timing, stale state, environment divergence
|
| Visual assertions |
Rendered output and layout regressions
|
Semantic intent, data correctness, cross-service consistency
|
| Observability |
Errors, latency, traces, production signals
|
Whether observed behaviour matches the intended user outcome
|
No layer is sufficient on its own. The engineering challenge is to decide which guarantee each layer is responsible for proving.
The UI is an integration boundary
The UI is often treated as the least interesting layer: a presentation concern beneath API and unit tests. That view is increasingly inadequate.
The interface is where frontend state, backend responses, permissions, platform rendering, accessibility layers and environmental configuration converge. It is also the only layer where many user-visible failures become observable.
A component may receive correct data and still render it incorrectly. A workflow may complete successfully at the service layer while leaving the user without a comprehensible confirmation. A legacy desktop application may expose no clean API surface, yet remain central to a business-critical process.
That is why GUI-level automation matters. A black-box, non-invasive approach operates at the graphical interface, interacts as a user would and uses image-based recognition to verify what is actually rendered on screen. This is particularly valuable where object identifiers are unstable, code access is restricted, or the system under test spans web, desktop, mobile and legacy environments.
Design assertions around outcomes
Strong automation does not ask, “Did the test execute?” It asks, “What evidence would convince us that the intended outcome occurred?”
For each critical journey, define:
- The user outcome: What must the user be able to accomplish?
- The UI evidence: What must be visible, selectable, readable or actionable?
- The system evidence: Which API response, database state, event or downstream process proves it?
- The failure signals: What should cause the test to fail?
- The environment matrix: Where must this hold: browser, operating system, device, locale, resolution or legacy client?
- The recovery path: What happens when a step fails, retries or leaves partial state?
A payment test, for example, should not merely assert that the payment API succeeded. It should validate that the user receives an unambiguous confirmation, the account or order state reflects the transaction, and a repeated submission cannot create a duplicate payment.
Stop asserting implementation
The most brittle tests tend to assert implementation details: CSS classes, internal element IDs, exact DOM structure or pixel-perfect snapshots. These checks often fail during harmless changes while missing meaningful regressions.
Better assertions focus on stable observable intent:
- Assert the outcome, not the mechanism: “the user can see confirmation”, not “the success banner has class alert-success”.
- Assert invariants, not snapshots: “a payment cannot be submitted twice”, not “this screenshot matches byte for byte”.
- Assert critical visual properties, not every pixel: readable text, visible controls, correct state, absence of overlap and expected layout.
- Assert negative paths: timeouts, permission failures, stale sessions, retries, rollback and partial completion.
- Preserve evidence: capture the screen state, application context and relevant logs when a test fails, so the result is diagnosable rather than merely repeatable.
Visual validation is most valuable when it asserts user-relevant rendered behaviour, rather than acting as an indiscriminate screenshot diff.
A practical test: remove the human
Here is the challenge worth putting to your team:
If every manual QA reviewer disappeared tomorrow, which production failures would your current assertions still fail to detect, and how would you know?
If the answer is “not many”, your automation is doing more than executing scripts. It is providing a credible assurance system.If the answer is “quite a few”, the problem is probably not test count. It is assertion debt: too many tests are checking that software runs, while too few are checking that users can successfully do what they came to do.
See what your users see
Assertion debt cannot be solved by adding more tests. It requires assertions that validate the outcomes users depend on, across the interfaces and environments where they work.
