Skip to main content

Error tracking

Error tracking is one row per distinct failure seen across automation runs, with a count of how often it has occurred — rather than one row per run, which is what the run list already gives you.

:::info Plan Error tracking is a Business plan feature. The link is hidden on lower plans, the route refuses, and the endpoint refuses again for itself. :::

How failures are grouped​

Two failures are the same error when their first error line matches — the first line of the stack that says what went wrong, and the message where there is no stack.

This is deliberately crude, and it is crude in the right direction. The rest of a trace varies per run: line numbers shift, timestamps and ids appear, paths differ between machines. Matching on the whole thing gives one row per run, which is the list you already had.

Two things refine it:

  • Playwright's own framing is skipped. A Playwright failure opens with the test's location ([chromium] › login.spec.ts:34:7 › …) and marks retries with Retry #1 ───; neither says what went wrong, so neither is matched on.
  • A bare assertion is placed by its line of code. expect(received).toBe(expected) is the same header for every assertion of its kind, so it is matched together with the failing line from the report's code frame — … — at expect(response.status()).toBe(200);. The same assertion failing in twenty tests is one error; two different assertions are two.

:::note Recently changed Playwright failures used to be matched on their location line, so every failing test was an error of its own, seen once. Errors recorded before October 2026 have been merged into the errors they really are: counts added together, the earliest first sighting and the latest last one kept, and any defect type a person chose carried over. :::

What a row tells you​

  • The failure's first line, which is the identity
  • Whether it is a product defect or a test defect (below)
  • How many times it has been seen
  • When it was last seen, and in which run
  • Which test it came from
  • The screenshots from its most recent occurrence, when that run sent them — open the row's details to see them

Product defects and test defects​

Every error is a product defect (the application did the wrong thing), a test defect (the automation broke), or unclassified.

Sekee decides when the failure says so plainly, and marks its choice auto with the reason as a tooltip:

  • Product defect — a value that differs from what the test expected (expect(received)…, Expected:/Received:, AssertionError), or an HTTP error the application answered with.
  • Test defect — a locator or selector that found nothing or timed out, code in the test that threw (TypeError, ReferenceError, a missing module), a test or hook that ran out of time, a page that closed under the test, or the application's rate limit (HTTP 429) — the suite asking too often.
  • Anything else — a network error, an element that never became visible with nothing else to go on — is left unclassified for a person.

A member or above changes it from the row's Defect chip, or from the failure on the run page; the choice is the error's, so every run that hit it shows the same. A choice made by a person is never overwritten. Let Sekee decide hands the error back to the automatic rule.

The toolbar filters by defect type, the Insights panel breaks the list down by it, and Analytics has three sources for boards: Errors by defect type, Product defects found and Test defects found.

Errors recorded before defect types existed are classified the next time they occur; an administrator can classify the rest at once with scripts/classify-errors.js (a dry run unless given --apply). The same script re-applies the automatic rule to errors it classified before, and never touches a choice a person made.

:::note Recently changed Errors used to carry no classification. Defect types and screenshots are new. :::

Where the rows come from​

Only ingest writes them. They are created as reports are parsed, guarded on results having actually been stored — so a re-sent report cannot inflate the counts. Apart from its defect type, an error cannot be edited or created by hand.

Old runs uploaded before error tracking existed can be backfilled by an administrator with scripts/backfill-automation-tests.js, which does the same work for reports already in the system.

Using it​

The value is in the count. A failure seen twice is noise; a failure seen sixty times across three weeks is a bug in the product or a genuinely broken test, and the count is what tells the two apart from a list of runs that all look equally red.

Sort by occurrences, take the top row, and fix that.