Suites & the run matrix

Group tests into suites and run them across a device × browser matrix, so one suite covers every combination you care about in a single run.

A suite is an ordered set of a project's tests that run together and report as one result. It is the unit you schedule, the unit CI gates on, and the unit that answers "is the app working?" rather than "does this one test pass?".

Creating a suite

A suite belongs to a project, has a name, and holds an ordered list of that project's tests. A test can be in several suites — suites are views over your tests, not folders that own them.

The groupings that earn their keep are usually about when you want to run them, not about what area of the app they touch:

Smoke — the handful of flows that must never break. Fast enough to run on every push.
Critical path — signup, checkout, the thing you get paid for. Run on every merge to main.
Full regression — everything. Run nightly, on a schedule.
Per-area — billing, admin, onboarding. Useful when one team owns one area.

A suite that takes forty minutes will not be run on every push no matter how good it is, so the split by duration is the one that changes behaviour.

Running a suite

Two ways to run a batch:

Run a suite — its tests, in order.
Run all — every test in the project, as an ad-hoc suite. Handy before a release when you do not care about the grouping.

Both produce one suite run that aggregates the results: total, passed, failed, duration, and the individual runs it contains. Both stream progress live, and both share the same worker pool, so the concurrency limit applies to the whole batch either way.

Parallel execution

A batch runs its tests through a worker pool with a configurable concurrency. At 1 the tests run strictly one after another; above that, several browsers run at once and the batch finishes proportionally faster.

Two ceilings apply, and it is worth knowing which one you are hitting:

Your plan's concurrency limit clamps the requested value. Asking for 10 on a plan that allows 5 gives you 5 — the request is clamped, not rejected.
The process-wide browser pool caps how many browsers can exist at once across everything the server is doing, including recordings and AI runs.

More concurrency is not free. Tests that share state — the same fixture user, the same seeded record — start interfering when they overlap, and the symptom is a test that passes alone and fails in a batch. If that happens, the fix is to give each test its own fixture rather than to turn concurrency back down; a suite that only passes serially is a suite with hidden coupling.

Watching a run

A batch streams four kinds of event as it goes:

Event	Meaning
`suite-resolved`	The test list is settled — you now know how many runs to expect.
`test-start`	A test began.
`test-complete`	A test finished, with its result.
`suite-complete`	The batch finished, with the totals.

suite-resolved arriving first is what lets the UI show "3 of 12" instead of a spinner — the count is known before the work starts, including after a matrix expansion.

Suite run history

Every batch is recorded with its status, totals, duration and the runs it produced. That history is what makes the analytics views meaningful: a single red run is noise, and twenty suite runs is a trend.

Device × browser matrix

A suite run can expand each test across a matrix of devices and browsers, producing one run per cell, all grouped under a single suite run. It is how one suite covers a device spread without maintaining a copy of each test per device.

{
  "concurrency": 2,
  "matrix": {
    "devices": ["iPhone 15", "Pixel 7"],
    "browsers": ["chromium", "firefox", "webkit"]
  }
}

Expansion rules

Cells are the cartesian product of devices × browsers.
An empty or absent dimension is not varied. Both absent means the suite runs exactly once per test — matrix-free suites are unaffected.
Ordering is test-major: every cell of test 1, then test 2, so a test's cells stay together in the results.
Each cell is its own run, labelled with the device and browser it used.

A 3-test suite with devices: ["iPhone 15", "Pixel 7"] produces 6 runs in one suite run. Multiply carefully: three devices × three browsers × twenty tests is 180 runs, and at concurrency 5 that is a long batch.

Devices and browsers do not combine

Device emulation is a Playwright feature, and a device profile sets its own browser — iPhone 15 implies WebKit. So the browsers dimension only varies cells that have no device.

In practice you pick either a set of devices or a set of browsers, not both. A device cell records the device it ran; it does not also record a browser, because the device chose it. See Browser & device configuration.

Where the matrix comes from

Highest wins:

The per-run override in the request body.
The suite's saved matrix.
Neither — one run per test.

"Run all" is an ad-hoc suite with nothing saved, so only a per-run override applies there.

Two limits

The WebDriver engine has no device registry, so device cells do nothing there. Vary browsers instead, or set the engine per environment. See Test engines.
Matrix runs are cloud-only for now. A suite whose environment targets a local agent is rejected with a clear error when the effective matrix varies a dimension, rather than silently running every cell with the same config.

Suites in CI and on a schedule

A suite is the natural target for both:

CI gates a merge on the smoke or critical-path suite. See CI integration.
A schedule runs the full regression suite nightly, so every morning starts with a current answer. See Scheduled runs.

A schedule with no suite selected runs every test in the project.