Three kinds of green, and the switch that is inert on purpose

A merge gate gives you one bit. Green or red. Everything a test run actually learned has to be squeezed through that bit, and most tools squeeze by throwing information away.

Vera's CI surfaces — the GitHub Action, and the vera-gate.sh script that the GitLab and Bitbucket templates embed verbatim — report one blocking outcome and three non-blocking greens. They are different things, they are printed on different lines, and only one of them stops a merge by default.

The four outcomes

A real failure blocks. A test ran, an assertion did not hold, the job exits 1 with a link to the report. This is the only outcome that is red out of the box.

A quarantined failure warns. A test you have explicitly marked as known-broken still runs and still reports, but its failure does not gate. One test nobody has had time to fix should not hold an entire team's merges hostage — and deleting it to get green is how a suite quietly stops covering the thing it was written for.

Passed on retry warns. The test failed, was retried, and passed. That is flakiness, not breakage. It is reported on its own line and gated only if you ask, with fail-on-flaky in the Action or VERA_FAIL_ON_FLAKY=true in the script.

Passed on heal warns. This is the interesting one.

Why a healed pass is not a failure, and not an ordinary pass either

When a selector-bearing step fails, Vera can snapshot the page, ask a model for a corrected selector, and retry the step. If the run then goes green, something true happened: the assertion still had to hold. Healing rewrites the locator, never the expectation. Nothing was weakened to get the pass.

But the step may no longer be driving the control it was written for. button.submit-btn and [data-testid=checkout-submit] can both exist on the page. A pass/fail column cannot show you which one the step ended up clicking, and folding a healed pass into the plain pass count destroys the only signal that would have told you to look.

So healed is reported beside status on every run response and is never folded into it. Two independent opt-ins, fail-on-flaky and fail-on-healed, because they are answers to different questions.

The switch that does nothing

Set fail-on-healed: true today and your job will not fail. Not because the flag is broken — because a run started from CI has no selector healer attached at all, so healed comes back zero and there is nothing for the gate to fire on.

That is a deliberate product decision, not an omission. Self-heal is an explicit repair action: the Repair button on a failed run, or the opt-in auto-repair gardener that a project owner has switched on. It is not an ambient property of every run. Two reasons:

• Cost. Healing is an unbounded number of extra model calls per run. An unattended nightly suite that healed by default would be an unbounded bill nobody approved.
• Ambiguity. If every run could silently rewrite its own selectors, every green would carry an invisible asterisk. The whole point of reporting healed separately is that somebody chose it.

We kept the flag anyway, documented it as inert in the Action's own input description and at the top of the shell script, and left it covered by the parity suite — so a team that sets it now gets the behaviour the day a CI-started run can heal, instead of discovering the control is missing in the week they need it.

Writing "this input does nothing yet" into the input's own help text is uncomfortable. It is still much better than the alternative, which is a team believing they have a gate they do not have.

Keeping three implementations honest

There are three places this policy is implemented: the GitHub Action, the shell script, and the templates that embed the script. If they ever disagreed about an exit code, the disagreement would show up as a merge that went through on GitLab and was blocked on GitHub — the kind of bug that costs trust rather than time.

So it is not documented and hoped for. A unit test executes the shell script and the Action side by side against the same canned API responses and fails on any exit-code divergence. Adding a fifth outcome means adding rows there, not prose. That test runs in the same suite as everything else, with no browser and no database, which is the only reason a check like it stays run rather than skipped.

The general shape

The pattern generalises past CI, and it is most of what we mean by "business-aware":

1. Do not collapse distinct outcomes into one column just because the column has room for one value.
2. Make the extra outcomes non-blocking by default, and opt-in to gate.
3. When a control cannot work yet, say so where somebody would reach for it.
4. Put the agreement between surfaces under a test, because a prose rule rots silently and an executed one does not.

If you want to see the three greens on your own suite, the CI integration guide has the Action, the script and both templates.