Analytics & flaky-test detection

Pass rates, trends and the flakiness score — how it is computed from the last ten runs, and why a test that always fails scores zero.

A suite's value is not the number of tests in it. It is whether a red run means something. The analytics views exist to answer that: what is passing, what is getting slower, and which tests have stopped telling you the truth.

What the dashboard shows

Summary — pass rate, total tests, total runs and average duration for the project, so you can see at a glance whether the suite is healthy or whether someone has been ignoring it. It also carries the ROI block (AI-written tests, their pass rate, healed selectors, estimated minutes saved) for plans that include it; on the plans that do not, the rest of the summary still works and the ROI block is simply absent.

Trends — daily run counts and pass rate over a window (default 30 days). The useful signal is usually the slope, not the number: a pass rate drifting down over two weeks is a suite rotting.

Test health — how many of the project's tests are currently passing, currently failing, and have never run. Three numbers, not a per-test table; the per-test view is the test explorer, which filters on latest status and on flakiness directly.

Failure clusters — recent failures grouped by the error they produced, so fifteen red runs that are one broken selector read as one problem.

Flaky tests — see below.

Control runs are not in any of these numbers

Vera runs some tests expecting them to fail. A negative control deliberately breaks a test to prove it was wired to your app at all, and the role matrix observes refusals. Those runs are marked, and every figure on this page filters them out.

Without that, proving a test was honest would lower your pass rate and raise its flake score — the measurement would damage the thing it measures. Accounting reads, metering included, still count them: they burned real browser seconds.

How flakiness is scored

Vera looks at each test's last 10 runs and counts how many times the status changed between consecutive runs. The score is those alternations divided by the number of comparisons:

flakinessScore = alternations / (runs - 1)

Tests with fewer than 2 runs are skipped — there is nothing to compare. Results are sorted worst-first.

There is a second flakiness number, and it is not this one. Each test also carries a stored flakeScore on a 0–100 scale over its last 20 terminal runs, recomputed after every run. It counts one thing this board does not: a run that only went green on a retry is an instability even though its final status never alternated. That stored score is what the test explorer's "flakiness" filter and the suites' flaky membership filter read. So a test can look calm on this board and still be filtered in as flaky — the two are measuring slightly different things on purpose.

What that means in practice:

Recent history	Score	Reading
`PPPPPPPPPP`	`0.00`	Stable. Passing.
`FFFFFFFFFF`	`0.00`	Stable. Consistently failing — a real bug, not flake.
`PFPFPFPFPF`	`1.00`	Maximally flaky. Tells you nothing.
`PPPPFPPPPP`	`0.22`	Mostly stable, one blip.

The important property: this measures instability, not failure. A test that fails every single time scores 0.00, because it is perfectly consistent. That is correct — it is a bug to fix, and it belongs in the failure list, not the flake list. A test scoring near 1.00 is the dangerous one: it is not reporting on your app, it is reporting on the weather, and its red runs will be ignored — including the one time it is right.

What to do about a flaky test

In rough order of how often it is the actual cause:

A missing wait. The test asserts on something that has not rendered yet. Wait for the end state rather than sleeping for a guess.
A brittle selector. Matching by text or by position finds a different element depending on load order. Move it to a test id or a role — see Recording.
Shared state between tests. A test that passes alone and fails in a batch is usually reading data another test wrote. Seed its own fixture.
Real intermittency in the app. A race condition, a slow query, a retry that sometimes gives up. The test is right and it is telling you something.

Cause 4 is rarer than the others but it is the one worth finding, which is the argument for fixing flakes rather than muting them.

Quarantine

You can quarantine a persistently flaky test so it stops blocking merges while remaining visible — a holding pen, not a delete.

Three things about it are worth stating plainly:

It is a decision you make, never automatic. Nothing quarantines a test because its score crossed a line, and there is no threshold that does it for you. It is a Pro-and-above feature; un-quarantining is not gated at all, so you can always get back out.
The test keeps running. Results, artifacts and its flake score are all still recorded. What changes is one thing: its run stops counting towards pass/fail gating in CI and in suites.
That suppression covers every non-pass, not just a failure. A quarantined test that is cancelled or errors also stops blocking. One predicate decides it, so the API, the suite roll-up and the CI gate cannot disagree.

Quarantined tests surface in the Needs Attention queue and carry a badge in the test explorer. The failure mode to avoid is a quarantine list that only grows.

Retention

Analytics are computed from run history, so the window you can analyse is bounded by how long runs are kept. The effective value is the tighter of the platform setting and the plan's, so a shorter global policy is respected. When runs age out, their artifacts (video, screenshots, diffs, logs) are swept with them.

Read "retention is off by default" carefully: the platform setting is off until an operator turns it on, but a plan's retention limit prunes regardless. On a hosted plan your history has a finite window whether or not anybody configured one.

One thing deliberately survives the sweep. Before a run is deleted, the definition it executed — the steps as they stood that day — is copied into an archive, deduplicated so every run of an unedited test collapses to a single row, and capped per test. The run, its step results, its timings and its artifacts still age out on schedule; what the run was asked to do is kept. There is no read surface for that archive yet; it exists so the history is there when one is built.

A few other clocks are not this one: AI interaction history and radar history each have their own retention setting, load runs and security scans ride the same window as runs (though their findings are deliberately kept), and product-usage events age out at 90 days on their own.

API

The same figures are available over the API, all scoped to a project:

GET /api/projects/:projectId/analytics/summary
GET /api/projects/:projectId/analytics/trends?days=30
GET /api/projects/:projectId/analytics/flaky
GET /api/projects/:projectId/analytics/failure-clusters?days=30

The last three require the roi-analytics feature on the org's plan. Summary does not — it answers on every plan; only its ROI block is withheld, and it is withheld as an absent block rather than an error, so the rest of the response is still usable.

What these numbers are not: cost

Nothing on this page is money. Two separate surfaces answer that, and both have a trap worth knowing.

AI spend lives in the AI cost panel, and its honesty rule is that a hole is reported as a hole. A call that reported no usage at all (a failed generation still spends tokens and returns none) makes every total a lower bound, and it says so. Separately, a call on a model Vera has no published price for records tokens but no cost — which would otherwise render as $0.00, i.e. as free. Those calls are counted and called out as unpriced, because unpriced is not free.

Browser-seconds and storage bytes are metered per workspace per UTC day and deliberately outlive the runs retention deletes — otherwise a free workspace's cost history would have a one-week memory. That record is read over an API by a platform administrator; there is no dashboard view of it yet, and on a hosted plan it is not yours to see. Two things it is not: it gates nothing and bills nothing — recording is the prerequisite for a decision, not the decision — and it is not the same as the platform's engagement telemetry, which answers "which parts of Vera do people come back to" rather than "what did this workspace burn".

Cancelled and control runs are both metered, and correctly so: they used real browser seconds. That is exactly why the pass-rate and flake numbers above exclude them and the meter does not — one measures your app, the other measures cost.