Skip to main content

Automation overview

Sekee ingests JUnit-style XML test reports from CI and turns them into three things: runs, tests and results. Together they give an automated test the same kind of history a manual one gets from bug hunts — and let CI move requirement coverage.

The shape of it​

automation run          one CI execution: a name, a source, tags, its report files
  └── automation test   one test with a history, identified by a fingerprint
        └── result      that test, in that run

An automation test is found or created by a fingerprint of "<suite> :: <name>" within a project, so the same test keeps its identity from run to run even as runs come and go.

:::info What counts as "a test" depends on the framework A Cypress group is a test — its steps are the entries beneath it. A Playwright entry is a test. Sekee normalises both on the way in, so the history is per-test in the sense the framework means. :::

The same test, run somewhere else​

One spec run against three tenants, in three browsers or against three environments is one test with three outcomes, not three tests.

Sekee reads that from a leading [marker] on the test — [tenant-a] Login, [chromium] signs in — which is where every reporter that can say so puts it. Playwright writes it for you with the JUnit reporter's includeProjectInTestName option, and teams running one suite per tenant usually write it into the describe by hand. Either way the marker names where the test ran, so it is kept beside the result and left out of the fingerprint. Where nothing is written into the title, Playwright's project name (the hostname attribute on the suite) is used instead.

It follows that:

  • the test is one row in the automation list and one entry under any test case it is linked to, however many tenants or browsers run it;
  • a run leaves one result per variant, so a run's dots come in threes and the history says which tenant each was;
  • the test's headline status is the worst of the newest run's — one tenant down is the test down;
  • linking it from any thread links the same test.

Only the first marker counts, and only at the front: [tenant-a] Login › [SEK-412] rejects a stale token runs as tenant-a and is called [SEK-412] rejects a stale token.

:::note Recently changed

Until this shipped, a marker was read as part of the test's name, so the same spec run against three tenants was three tests with a third of the history each — and a report carrying every variant in one file stored only the first, the rest being read as retries of it.

Rows already written are merged by an administrator running node scripts/merge-automation-test-variants.js once per environment; until that is run, an environment shows the old, split rows.

:::

Getting reports in​

Two ways, both authenticated with an API token sent as x-api-token:

  • sekee-cli — the supported path. One command for a simple run, or init / thread / complete for parallel shards.
  • The HTTP API directly, if you would rather not add a dependency to your pipeline.

Both derive the organization from the token. The token is what makes the request tenant-scoped; there is no other credential.

Finding a run​

Automation lists the runs of the project you have open, newest first, banded by when they arrived — Today, Yesterday, This week, and so on.

The toolbar narrows the list by status, source, tag and a date range, and the search box matches a run's name or its source. They combine, and the footer pages through whatever is left at the size you choose.

A run's status is its result: Failed in red when any test or step in it failed, Passed in green when none did, and Running while threads are still arriving — Failing if one of them has already come back red. The list refreshes itself while a run on it is still running, so its counts and colour move as each thread lands.

:::note Recently changed A run that finished used to read Passed whatever its tests did, because the status showed that the upload had finished rather than how it went. The Status filter's Passed and Failed now mean the result too. :::

:::note Recently changed

The filters, the search and the pages are all answered by the server now, so they cover every run the project has rather than the batch the page had already loaded. A search that used to come back empty on a long history now finds the run.

The count is also the project's. The list fetched the whole organization and narrowed it in the browser, so the total counted runs from projects the table was not showing — 65 runs reported above 24 rows. It reports what it lists.

The list used to render every run in one scroll and now has the same pager as the rest of the app.

:::

Reading a run​

Automation → a run. You get the parsed report: spec files, tests, durations, and the failure output for anything that failed. A test that has a row in Sekee is decorated with a link to it, and that is where you link it to a test case.

The page opens on All threads — every spec file the run reported, each marked with the thread it ran in. A thread's tab narrows the list to that thread. While the run is still receiving threads the header says Live and the page updates by itself as each one lands.

The header also says how long the run took, beside its date. For a run sent with init, thread and complete, that is the time from init to complete — shards that ran side by side count once. A run sent in one step with upload-reports has no start of its own, so it shows the time its report records instead; hover the time to see which. While a run is still receiving threads it shows the time so far, up to the last thread that arrived.

AI summary​

Under the thread tabs, above the filters, is the run's AI summary: what an AI assistant made of the run, written for whoever opens it next. Until one is written the box is a single line — Ask your AI assistant to fill in the AI summary for this run. Ask yours, through the AI connector, to summarise the run, and the summary appears there for everyone who opens it.

A long summary stops at about ten lines; Show all opens the rest. A run has one summary, and only an assistant can write it, never by hand. To change it, ask your assistant to update it: the new text replaces the old one, which is not kept, and the date in the box says when it last changed. An assistant that only asks to write a summary is refused when the run already has one, so nothing is replaced by accident, and a summary cannot be emptied. Each write and each update is recorded in Activity, marked AI generated.

Failures​

A failure is laid out in its parts: the error, where it happened, Expected against Received, and the line of code with the failing line marked. The stack and the reporter's raw output are a click away. If the report was sent with screenshots (see thread), they sit beside the failure — click one to see it full size. On plans with error tracking each failure also says whether it is a product defect or a test defect, and a member can change it from there.

Each failed test opens and closes from its own row. It starts open; click it (or press Enter on it) to close it once you have read it, and its error's first line stays in view so a column of closed failures still says what broke.

Insights, beside Comments, opens a panel with this run's numbers: how much of it passed — the chart with the count and share of failed, passed and skipped tests beside it — each thread side by side, failures by defect type, the most common failures and the slowest tests, each with its full name and the file it is in. It follows the thread tabs. Pin it to have it open on every run — the same switch is on Settings → Behaviour.

The test case's automation history marks each run that has screenshots with an image count you can open.

:::note Recently changed A run's AI summary can now be updated; it used to be written once and stay as it was. The AI summary and the run's total time are new. Failed tests used to stay open for good, the Insights chart showed no numbers beside it, and the slowest tests were a bar chart that cut their names short. Before that, the run page opened on its first thread, with three charts above it, and showed a failure as one block of text. :::

Linking a test to a test case​

Open the run, find the test, and link it. From then on:

  • The test case shows that automated test and its recent runs
  • The last CI result feeds the coverage of any requirement the case proves
  • The case's View all page lists every run of it as a table

This is the one manual step in the chain, and it is deliberate — Sekee will not guess that login.spec.ts › signs in proves the case you called "Sign in with a valid email".

Tags and sources​

A run carries a source (GitHub Actions, Local, Jenkins — free text) and tags. Both are for finding runs later and for scoping board widgets, and both are set by whoever uploads the report.

Watching automated tests​

You can watch a test and be emailed when it changes state — see Notifications. This is the automated equivalent of watching a bug hunt.

Errors​

Failures are also aggregated into error tracking, which is one row per distinct failure across every run rather than one row per run.