Suites & the run matrix

Group tests into suites and run them across a device × browser matrix, so one suite covers every combination you care about in a single run.

A suite is an ordered set of a project's tests that run together and report as one result. It is the unit you schedule, the unit CI gates on, and the unit that answers "is the app working?" rather than "does this one test pass?".

Creating a suite

A suite belongs to a project, has a name, and holds an ordered list of that project's tests. A test can be in several suites — suites are views over your tests, not folders that own them.

The groupings that earn their keep are usually about when you want to run them, not about what area of the app they touch:

Smoke — the handful of flows that must never break. Fast enough to run on every push.
Critical path — signup, checkout, the thing you get paid for. Run on every merge to main.
Full regression — everything. Run nightly, on a schedule.
Per-area — billing, admin, onboarding. Useful when one team owns one area.

A suite that takes forty minutes will not be run on every push no matter how good it is, so the split by duration is the one that changes behaviour.

Membership can also be a filter

Beyond the list you curate, a suite can carry filters — by tag, by "failed in the last N days", by "currently flaky" — and those are resolved fresh on every run, not stored. The two combine: the static list first, then whatever the filters add. A "recently broken" suite is a suite with no members and one filter.

Two things to know about the dynamic half. Resolution is capped at 200 tests, and a run that hit the cap says so rather than quietly running a prefix. And the "recently failed" window is refreshed on a sweep rather than continuously, so it is accurate to within about a quarter of an hour — fine for a nightly, not a mechanism to build a race on.

Running a suite

Two ways to run a batch:

Run a suite — its tests, in order.
Run all — every test in the project, as an ad-hoc suite. Handy before a release when you do not care about the grouping.

Both produce one suite run that aggregates the results: total, passed, failed, duration, and the individual runs it contains. Both stream progress live, and both share the same worker pool, so the concurrency limit applies to the whole batch either way.

Not every member necessarily runs

A batch runs only the tests that are runnable, and that gate is applied by every executor — the in-process one, the queue worker, the agent fan-out, the scheduler, the CI API and the MCP run_suite tool alike. Two states are held back:

draft or archived lifecycle.
pending or rejected review — an AI-written test waiting for a human, on a workspace that requires sign-off. See AI planning.

Everything else runs, and two exclusions people expect are deliberately absent: a deprecated test still runs, and a quarantined test still runs — it is excluded from gating a merge, not from executing.

If some members are gated the batch runs the rest. If every member is gated the run is refused with the count and the reason, rather than reporting a green suite of nothing.

One honest wrinkle: the membership preview — the list the suite page shows and the suite-resolved frame described below — does not apply the review gate, so a preview can name a test that dispatch then drops. The executed count is the true one.

Stopping a suite run

The suite run history carries a Stop on any batch still in flight. It is one action with three effects: the suite row is written cancelled, members already executing are asked to stop at their next step, and members that had not started never start.

Rows that already finished keep their verdicts — those are evidence. Members that never started leave no run row at all, so a stopped suite's passed + failed is smaller than its total, and that gap is what was stopped. A stopped batch notifies nobody.

Stopping is never plan-gated. An entitlement decides whether you may start work, not whether you may stop it. See Stop or cancel a run for what the answer codes mean.

Parallel execution

A batch runs its tests through a worker pool with a configurable concurrency, between 1 and 5. At 1 the tests run strictly one after another; above that, several browsers run at once and the batch finishes proportionally faster.

Three ceilings apply, and it is worth knowing which one you are hitting:

The batch's own cap of 5. Ask for 20 and you get 5, whatever your plan says.
Your plan's concurrency limit clamps it further. Asking for 5 on a plan that allows 2 gives you 2 — the request is clamped, not rejected, so a suite already running survives a downgrade instead of dying mid-batch.
The process-wide browser pool caps how many browsers can exist at once across everything the server is doing, including recordings and AI runs.

Separately from the clamp, a batch is refused if your workspace already has its plan's number of runs in flight — that one is a plan error naming the limit, not a silent queue.

More concurrency is not free. Tests that share state — the same fixture user, the same seeded record — start interfering when they overlap, and the symptom is a test that passes alone and fails in a batch. If that happens, the fix is to give each test its own fixture rather than to turn concurrency back down; a suite that only passes serially is a suite with hidden coupling.

Watching a run

A batch streams four kinds of event as it goes:

Event	Meaning
`suite-resolved`	The test list is settled — which members were listed, which a filter pulled in, and whether the cap truncated it.
`test-start`	A test began.
`test-complete`	A test finished, with its result.
`suite-complete`	The batch finished, with the totals — and `cancelled` as its status if somebody stopped it.

suite-resolved arriving first is what lets the UI show "3 of 12" instead of a spinner — the count is known before the work starts, including after a matrix expansion. Run all has no membership to announce, so it is the one batch that does not send the frame.

There is no cancel frame. A stop shows up as suite-complete with a cancelled status, so a client that already handles the ending handles this one.

Suite run history

Every batch is recorded with its status — running, queued, passed, failed or cancelled — plus totals, duration and the runs it produced. That history is what makes the analytics views meaningful: a single red run is noise, and twenty suite runs is a trend.

Device × browser matrix

A suite run can expand each test across a matrix of devices and browsers, producing one run per cell, all grouped under a single suite run. It is how one suite covers a device spread without maintaining a copy of each test per device.

{
  "concurrency": 2,
  "matrix": {
    "devices": ["iPhone 15", "Pixel 7"],
    "browsers": ["chromium", "firefox", "webkit"]
  }
}

Expansion rules

Cells are the cartesian product of devices × browsers.
An empty or absent dimension is not varied. Both absent means the suite runs exactly once per test — matrix-free suites are unaffected.
Ordering is test-major: every cell of test 1, then test 2, so a test's cells stay together in the results.
Each cell is its own run, labelled with the device and browser it used.

A 3-test suite with devices: ["iPhone 15", "Pixel 7"] produces 6 runs in one suite run. Multiply carefully: three devices × three browsers × twenty tests is 180 runs, and at concurrency 5 that is a long batch.

Devices and browsers do not combine

Device emulation is a Playwright feature, and a device profile sets its own browser — iPhone 15 implies WebKit. So the browsers dimension only varies cells that have no device.

In practice you pick either a set of devices or a set of browsers, not both. A device cell records the device it ran; it does not also record a browser, because the device chose it. See Browser & device configuration.

Where the matrix comes from

Highest wins:

The per-run override in the request body.
The suite's saved matrix.
Neither — one run per test.

"Run all" is an ad-hoc suite with nothing saved, so only a per-run override applies there.

Two limits

The WebDriver engine has no device registry, so device cells do nothing there. Vary browsers instead, or set the engine per environment. See Test engines.
Matrix runs are cloud-only for now. A suite whose environment targets a local agent is rejected with a clear error when the effective matrix varies a dimension, rather than silently running every cell with the same config.

Suites in CI and on a schedule

A suite is the natural target for both:

CI gates a merge on the smoke or critical-path suite. See CI integration.
A schedule runs the full regression suite nightly, so every morning starts with a current answer. See Scheduled runs.

A schedule with no suite selected runs every test in the project.