AI planning: plan, generate, prove

"Generate tests for my app" is a bad instruction, because the interesting decision is which tests, and that decision is the one you should be making. Vera splits it: the model proposes a plan you edit, and only then writes the tests.

The whole loop, end to end:

plan → generate the batch → edge cases as datasets → check sensitivity → read the coverage board

The first two are AI. The third is a fixed matrix with one AI call for the values. The fourth and fifth are ordinary test runs and arithmetic. That ordering is deliberate — the further you go, the less of it is a model's opinion.

Plan

Plan tests on the Tests page. One optional sentence of goal, and Vera reads a page of your app, your existing test names, your approved rules and business context, your reusable flows, the module names you already use, and the pages the radar has accepted as baseline.

What comes back is a checklist. Per proposed test:

Tick off the ones you do not want. The plan is never persisted — it exists between the proposal and your pressing Generate, and that is the whole design.

Two deterministic guards run on the model's output before you see it:

• A rule id the model cited that is not an approved rule is dropped. A hallucinated id would show up as coverage for a requirement that does not exist.
• An entry URL that leaves your app's origin is refused.

Above the list sit the warnings that matter: how much of the page was not examined, which rules were left out of the prompt, and a cost estimate as a p50–p90 range with the sample size it came from. With too little history it says so rather than showing a list price.

if and loop are deliberately absent from the generator's vocabulary. Where branching is needed it is assembled deterministically — see Edge cases, as a dataset.

Generate the batch

Generate (N) runs the plan. Each test is generated, then run once with self-heal on so you get a recording. Progress streams; a failure on one test is a warning and the rest continue.

Three separate things bound how many tests one plan may produce:

Asking for more than your plan allows is a refusal, not a silent trim.

Second, if your workspace is running on the free AI grant rather than its own key, a batch is capped at three tests — and the cap is announced: "3 of 10 generated — your free AI allowance covers 3 tests per plan."

Third, a runaway guard: a hard ceiling on model calls and thirty minutes of wall clock for the whole batch. It is a guard, not a quota — a normal batch never reaches it — and when it stops one, the tests already written are kept and the stop tells you what remains.

There is no monthly AI quota in the plan matrix. Tenants bring their own key; the only platform-funded spend is the one-time onboarding grant.

Nothing a model wrote runs until somebody says so

This is the gate worth understanding properly, because it is off by default.

Settings → Workspace → "Require review of AI-written tests." While it is on, every test a model writes is saved as a draft marked pending, run once to produce a recording, and then held.

A test carries two columns and they answer different questions. lifecycle is is this in service?; review_state is did anyone agree it should be?

not_required is the default and the truthful value for history: every test that existed before this gate was written or accepted by a person, and the question was never asked. It is emphatically not the same as approved.

What "held" means. One accessor decides which tests a machine may select, and every surface that picks a set asks it: the scheduler, suite and project batch runs, dynamic suite filters, chain segments, the /api/v1 CI run, MCP run_suite, the impact map behind vera check --run, agent watch mode, the radar's at-risk scan, the repair gardener's staging pass, and the coverage board. A run that would have picked up held tests says so — "3 tests skipped (2 awaiting review, 1 draft or archived)" — rather than quietly running fewer.

The notifier is in on it too: a pending test's own first run does not email anybody or fire a webhook. That run is evidence, not news.

What it deliberately does not cover is a person pointing at one row. The Run button, MCP run_test, the generator's own auto-run, and a reviewer re-running a pending test all still work — the last of those is the entire point of the queue.

Two honest gaps: cloning a test and importing one are not stamped, so an AI-converted import goes straight into service.

Clearing the queue

A pending count appears in the project header and as a chip in the test explorer; opening it replaces the list. Each item shows the recording on the left and the generated steps on the right, with three buttons: Approve, Reject, Open in builder.

Four states are rendered as first-class, not as an error:

• No recording was kept — judge it on the steps.
• It was never run — some paths write tests without an auto-run; the steps are the whole material.
• The generator's run did not pass — normal for a first run, and not by itself a reason to reject. It is what you are judging.
• This workspace does not require review — an empty queue with the gate off is not "all clear", and the queue says so.

Approving puts the test into service. Rejecting archives it — the record that a model proposed it and a person said no survives, along with its run and video.

There is bulk approve (with a caution naming what it is about to switch on) and deliberately no bulk reject, and no "modify" verdict: edit it in the builder, then approve. Each verdict carries a hash of the steps you looked at, so a test that changed underneath you is refused rather than approved blind.

Reviewing is not plan-gated on any tier. Charging for the brakes would be an odd product.

Edge cases, as a dataset

Twelve edge cases are not twelve tests. Ask a model for twelve tests and you get twelve near-duplicates that drift apart the first time the form changes.

So from an approved rule, Vera builds one parameterized test and N rows. The flask button on an approved rule card does it.

The slots are a fixed matrix, not a model's imagination:

The model fills in the values for your domain. The steps are assembled deterministically — action, then if expected == reject / else / end-if — which is where the branching the generator is not allowed to write comes from.

Other rule kinds say so instead of guessing: a permission rule belongs in the role matrix, a state-transition rule needs a per-case setup request, and the button is not offered at all rather than returning an error.

Two details that look like plumbing and are not:

• A row that cannot fail is discarded. A "reject" case with no expected message would pass whenever the app renders anything at all, so it is dropped and reported rather than counted as coverage.
• If your project already has a variable called value or expected, the dataset renames its column. Without that, the project variable would win at substitution, every row would run identical input, and all twelve would pass.

Row counts are capped per plan: 25 / 100 / 1,000 / 10,000.

This path writes the test and the dataset and does not run them — running once for evidence would mean running N times. On a gated workspace that means a queue item with no recording, which the queue renders honestly.

Sensitivity is not correctness

A test that passes may be passing because it is wired to your app, or because it asserts nothing that can fail. The negative control tells the two apart.

Check sensitivity in a test's row menu. It:

1. Runs the test unfalsified, right now, as a baseline. Not a passing run from history — same app, same data, same moment, one field changed.
2. Builds up to three in-memory copies with one expectation deliberately falsified — the expected value replaced by a sentinel, or an expected status set to one nothing returns. Only the expectation moves; the selector and the action are untouched, because a mutated locator would fail on "element not found" and prove nothing.
3. Runs them.

Neither your app nor the stored test is modified. The runs it spends are marked as control runs and are excluded from health, pass rates, trends, flake scores, digests and the repair gardener — while still being metered, because they cost browser seconds like anything else.

There is a fifth state, and it is the common one: no verdict at all. The check is an explicit action that costs two to four real browser runs, so it is never automatic, and on a real project most tests have never had one. Not checked is not a verdict and is never painted as one.

And sensitivity is not correctness. It proves a test is connected to the app. It says nothing about whether the expectation was right. A test generated from a rule that was read out of your implementation will be perfectly sensitive to the bug that implementation has.

Presence-only assertions (is it visible?) are never falsified — there is nothing there to falsify. If a test's verdict is unknown, that is usually why, and it is a finding about the test.

The coverage board

The Rules tab shows, per approved rule, whether anything actually covers it:

Insensitive is worse than uncovered, and it has its own bucket for that reason: an uncovered rule looks uncovered, while an insensitive one looks protected.

Only approved rules are on the board — a proposal is not the specification yet, and counting it would inflate the denominator. And the tests counted are read through the run-selection gate, so a test still awaiting review is not coverage you have. That matters because vera check reads this board: an unreviewed test counting as coverage would turn a gap into a green gate.

A test with no sensitivity verdict yields partial, never covered. Reading "not insensitive" as "fine" is exactly the mistake this board exists to prevent.

The board is a Team feature (coverage-map). Without it the rules page simply has no board.

Where a rule came from decides what it can prove

Since everything above generates from rules, the provenance matters as much as the text. A rule's source falls into three tiers:

If your code ships a 10% discount where the requirement said 20%, an inferred rule says 10%, every test generated from it asserts 10%, and the negative control will happily verdict that test sensitive. Sensitivity confirms the wiring, not the requirement.

Everything discovered or imported lands proposed, and approval is human-only — one function returns approved rules and it is the only place the gate exists. The exception is a rule you type yourself, which lands approved because there is nobody else to review it. A rule with no evidence is dropped at discovery rather than proposed without one; review that cannot see where a rule came from is rubber-stamping.

What has no dashboard control

• Cross-project planning — an org-wide planner that fills coverage gaps across projects and proposes chains exists on the API, Team-gated on the same chains entitlement. Nothing in the dashboard calls it.
• The planner's proposed flows. It names prerequisites that ought to become reusable flows, and nothing acts on them.
• Plan, edge cases, sensitivity and rule approval have no MCP tools. An agent can generate tests (generate_tests) and read coverage (check_coverage), and tests it writes pass through the same review gate.

See also

• Business context & the rule book — writing and approving the rules this all reads
• The test explorer & taxonomy — where the modules a plan proposes end up
• AI cost, budgets and estimates — what a batch costs before you spend it
• vera check — the gate that reads the coverage board
• Staged repairs & the drift gardener — the other machine writer, and its own review path