Analytics & flaky-test detection
Pass rates, trends and the flakiness score — how it is computed from the last ten runs, and why a test that always fails scores zero.
A suite's value is not the number of tests in it. It is whether a red run means something. The analytics views exist to answer that: what is passing, what is getting slower, and which tests have stopped telling you the truth.
What the dashboard shows
Summary — pass rate, run counts and totals for the project, so you can see at a glance whether the suite is healthy or whether someone has been ignoring it.
Trends — pass rate and duration over a window (default 30 days). The useful signal here is usually the slope, not the number: a pass rate drifting down over two weeks is a suite rotting, and a duration climbing steadily is a suite that will time out in CI a month from now.
Test health — a per-test breakdown, so a project-level pass rate of 94% can be resolved into "one test fails a third of the time" rather than "everything is slightly broken".
Flaky tests — see below.
How flakiness is scored
Vera looks at each test's last 10 runs and counts how many times the status changed between consecutive runs. The score is those alternations divided by the number of comparisons:
flakinessScore = alternations / (runs - 1)Tests with fewer than 2 runs are skipped — there is nothing to compare. Results are sorted worst-first.
What that means in practice:
| Recent history | Score | Reading |
|---|---|---|
PPPPPPPPPP | 0.00 | Stable. Passing. |
FFFFFFFFFF | 0.00 | Stable. Consistently failing — a real bug, not flake. |
PFPFPFPFPF | 1.00 | Maximally flaky. Tells you nothing. |
PPPPFPPPPP | 0.22 | Mostly stable, one blip. |
The important property: this measures instability, not failure. A test that fails every single time scores 0.00, because it is perfectly consistent. That is correct — it is a bug to fix, and it belongs in the failure list, not the flake list. A test scoring near 1.00 is the dangerous one: it is not reporting on your app, it is reporting on the weather, and its red runs will be ignored — including the one time it is right.
What to do about a flaky test
In rough order of how often it is the actual cause:
- A missing wait. The test asserts on something that has not rendered yet. Wait for the end state rather than sleeping for a guess.
- A brittle selector. Matching by text or by position finds a different element depending on load order. Move it to a test id or a role — see Recording.
- Shared state between tests. A test that passes alone and fails in a batch is usually reading data another test wrote. Seed its own fixture.
- Real intermittency in the app. A race condition, a slow query, a retry that sometimes gives up. The test is right and it is telling you something.
Cause 4 is rarer than the others but it is the one worth finding, which is the argument for fixing flakes rather than muting them.
Vera can also quarantine a persistently flaky test so it stops blocking merges while remaining visible — a holding pen, not a delete. The failure mode to avoid is a quarantine list that only grows.
Retention
Analytics are computed from run history, so the window you can analyse is bounded by how long runs are kept. Retention is a per-plan limit, and the effective value is the tighter of the platform setting and the plan's — so a shorter global policy is respected. When runs age out, their artifacts (video, screenshots, diffs) are swept with them.
API
The same figures are available over the API, all scoped to a project:
GET /api/projects/:projectId/analytics/summary
GET /api/projects/:projectId/analytics/trends?days=30
GET /api/projects/:projectId/analytics/flakyThese require the roi-analytics feature on the org's plan.
See also
- Suites & the run matrix — batch runs, which produce most history
- Scheduled runs — a steady cadence is what makes trends readable
vera check— coverage, the other half of "is this suite any good"- CI integration — gating on results