CI integration

Trigger Vera tests from CI with an API token and fail the build when a test fails — while quarantined tests, retry-flakes and healed passes surface as warnings instead of breaking the merge. (Healed passes are described here for completeness: a CI-started run does not heal today, so that outcome does not yet occur — details.)

This page runs a fixed set: a test, a suite, or the whole project. To run only the tests a pull request's diff actually touches, see §8 and change impact.

1. Mint a token

In the dashboard, open Settings → CI Tokens (or POST /api/tokens with { "name": "ci" }). The vera_… token is shown once — store it as a CI secret (e.g. VERA_TOKEN). Also note your project id and your Vera base URL (VERA_URL) — e.g. the hosted deployment https://vera-agent.com.

2. GitHub Actions — the vera-run action (recommended)

name: E2E (Vera)
on: [pull_request]

jobs:
  vera:
    runs-on: ubuntu-latest
    steps:
      # ... deploy your PR preview and capture its URL ...
      - uses: mahmoodnasr/vera-run@v1 # once published; or vendor the dir and use `./.github/actions/vera-run`
        with:
          api-url: ${{ secrets.VERA_URL }}
          api-token: ${{ secrets.VERA_TOKEN }}
          project: ${{ vars.VERA_PROJECT_ID }}
          # optional:
          # suite: suite_xyz              # or `test: test_xyz`
          # environment: staging          # named Vera environment profile
          # preview-url: ${{ steps.deploy.outputs.url }}
          # fail-on-flaky: 'true'
          # fail-on-healed: 'true'

The action fails the job only on blocking failures: a failed quarantined test or a test that passed on retry produces a ::warning:: annotation and a job-summary table (with a report link) instead. When preview-url is omitted it falls back to VERCEL_URL / DEPLOY_PRIME_URL / DEPLOY_URL — so the same suite runs against every PR's preview deploy. See actions/vera-run for all inputs/outputs.

3. GitLab CI

Copy ci/gitlab/.gitlab-ci.yml into your project's .gitlab-ci.yml (or include: it), then:

1. Settings → CI/CD → Variables: add VERA_TOKEN = your vera_… token, Masked (and Protected if your MR branches are protected).
2. Set VERA_URL and VERA_PROJECT in the job's variables: block.

vera-e2e:
  stage: test
  image:
    name: alpine/curl:latest
    entrypoint: ['']
  variables:
    VERA_URL: 'https://vera.example.com'
    VERA_PROJECT: 'proj_abc123'
    VERA_PREVIEW_URL: '$CI_ENVIRONMENT_URL' # review-app URL, when your deploy job sets `environment:`
  script:
    - | # the embedded gate script — see the template for the full body
  artifacts:
    when: always
    reports:
      junit: vera-junit/*.xml # per-step results in the MR test widget
      dotenv: .vera/result.env # VERA_STATUS/VERA_PASSED/... for later jobs

The job needs only curl and a POSIX shell — no Node, no jq, no install step. It deliberately does not set allow_failure: true: the script decides what is blocking, and marking the whole job non-blocking would hide real failures too.

4. Bitbucket Pipelines

Copy ci/bitbucket/bitbucket-pipelines.yml into bitbucket-pipelines.yml (or merge its step into yours), then:

1. Repository settings → Repository variables: add VERA_TOKEN = your vera_… token, Secured.
2. Edit the export VERA_URL= / export VERA_PROJECT= lines at the top of the step (Bitbucket has no per-step variables: block).

The step writes JUnit XML to test-results/, which Bitbucket picks up automatically and renders in the build's Tests tab. Bitbucket's default image already has curl.

5. Gating parity across CI providers

All three surfaces gate identically. This is a product promise, not a coincidence: a quarantined test that reddened your GitLab pipeline while the same test only warned on GitHub would make quarantine worthless.

The two healed rows describe what every surface would do, and the parity suite proves they agree — but a CI-started run cannot reach that outcome today. See Healed passes are not reachable from CI yet.

Opt into failing on retry-flakes with fail-on-flaky: 'true' (GitHub) or VERA_FAIL_ON_FLAKY: 'true' (GitLab/Bitbucket). Quarantined failures stay non-gating either way — quarantine is the escape hatch, and an escape hatch that can still fail the build is not one.

The two opt-ins are independent: fail-on-flaky never gates a healed pass, and fail-on-healed never gates a flake. Both are exercised as their own rows in the parity suite.

Green is not one thing

A run that reports passed got there in one of three ways, and Vera tells you which:

Healing only ever rewrites the locator, never the expectation — the assertion still had to hold against your app, which is why a healed pass is not a failure. What it can hide is a semantic change: if a Delete customer button disappears and the healer binds the step to a neighbouring control, a broad assertion can still pass. The interaction the test was written to exercise is gone, and a pass/fail column cannot show you that.

So healed is reported beside the status rather than folded into it, and the run detail names every rewritten selector (before → after). Teams gating a release branch typically set fail-on-healed: 'true' there and leave it off for feature branches: a heal is then a merge-blocking prompt for a human to confirm the test still means what it meant.

Healed passes are not reachable from CI yet

PASS ON HEAL is a real outcome, but not one a CI-started run produces today, and you should know that before you rely on the opt-in.

A run started through POST /api/v1/…/run — which is what all three surfaces above call — executes cloud-side, in the Vera server process or on its worker, whatever the project's runTarget says, and neither attaches a selector healer. Healing is an unbounded number of extra AI calls per run, and a CI run that silently rewrote its own selectors would make every green ambiguous — which is exactly the ambiguity this section exists to remove.

So on the CI path healed is false / 0, and fail-on-healed / VERA_FAIL_ON_HEALED never fires. The flag is kept, documented and executed by the gating-parity suite so that a team can set it once and have it take effect the day a CI-started run can heal, rather than finding the control missing at the moment they need it.

Where a run can heal. Only where someone opted in, and none of it is a run these surfaces start:

• vera-agent runs, when an org admin has turned on settings.agent.selfHeal (PUT /api/org/settings, off by default). That covers every run the agent executes: vera-agent start draining the queue, vera run, vera verify, and vera check --run --local / vera watch --affected. A vera check --run --local step inside a CI job can therefore heal. It reports each healed pass with the rewritten selectors, and gates on it with its own --fail-on-healed flag or the same VERA_FAIL_ON_HEALED=true the shell gate reads — opt-in, under the same rules as the rows above, exiting 5 (details). Without the opt-in a healed pass there is green.
• The repair surfaces — the Auto-heal button and auto-repair.
• AI test generation and planning runs, which self-heal unless the request turns it off.

Those runs carry healed: true on GET /api/v1/runs/:id, which reads whatever the run row holds.

fail-on-flaky is unaffected. Retries happen on every run, healer or no healer, so PASS ON RETRY is reported and gated today exactly as described above.

How the promise is kept:

• GitLab and Bitbucket run the same script, ci/vera-gate.sh, embedded verbatim in both templates.
• apps/server/src/services/ci-gating-parity.test.ts stands up a stub /api/v1, executes both the GitHub Action and the shell gate against every row of that table, and fails if their exit codes differ or if either template's embedded copy has drifted. pnpm test runs it, so the table above cannot quietly go stale.

6. The raw API (any CI system)

• POST /api/v1/projects/:id/run — body { "testId": "…" }, { "suiteId": "…" }, { "testIds": ["…", "…"] } (max 50; more is 400 too_many_test_ids and nothing runs — a result built from the first 50 of 84 would read like a complete one), or { "all": true }. Exactly one of them: combining testIds with testId / suiteId / all is rejected. Optional "environment": "staging" (named profile) and "baseUrlOverride": "https://pr-42.example.com" (wins over the environment — meant for per-PR preview URLs). Runs synchronously and returns the terminal result:

  {
    "runIds": ["…"],
    "status": "passed",
    "passed": 12,
    "failed": 1,
    "blockingFailed": 0,
    "quarantinedFailed": 1,
    "passedOnRetry": 2,
    "healed": 1,
    "reportUrl": "https://vera.example.com/projects/proj_x/tests"
  }

  Gate CI on status (already quarantine-aware: only non-quarantined failures fail it) or on blockingFailed. healed counts the runs that reached green only after self-heal rewrote a selector — never part of status, always reported, and always 0 from this endpoint today, because a run started here has no healer attached (why). Single-test runs return { "status", "blocking", "quarantined", "passedOnRetry", "healed", "reportUrl" } — gate on blocking.

• GET /api/v1/runs/:id — current status (+ passedOnRetry, healed). Unlike the run endpoint above this reads whatever the row holds, so it reports healed: true for a run that did heal: a vera-agent run with settings.agent.selfHeal on, a repair run, or an AI-generation run (where a run can heal).

• GET /api/v1/runs/:id/wait — long-polls up to 60s for a terminal status (202 if still running).

• GET /api/v1/runs/:id/junit — the run as JUnit XML (see §7).

• GET /api/v1/suite-runs/:id/junit — a suite run as JUnit XML (one <testsuite> per member run).

All require Authorization: Bearer vera_…. A revoked/invalid token → 401.

resp=$(curl -sS -X POST \
  -H "Authorization: Bearer $VERA_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"all": true}' \
  "$VERA_URL/api/v1/projects/$PROJECT_ID/run")
[ "$(echo "$resp" | jq -r '.status')" = "passed" ] || { echo "::error::Vera run failed"; exit 1; }

7. JUnit XML — render results in the PR checks UI

Every run (and every suite run) can be exported as JUnit XML, the format that test-report actions understand. Feed it to dorny/test-reporter, mikepenz/action-junit-report, or your CI's native JUnit ingester and you get a per-step pass/fail breakdown attached to the check — no scraping the JSON.

• GET /api/v1/runs/:id/junit → one <testsuite> (the test), one <testcase> per step.
• GET /api/v1/suite-runs/:id/junit → a <testsuites> wrapping one <testsuite> per member run.

Both respond Content-Type: application/xml with an attachment Content-Disposition, and require Authorization: Bearer vera_…. The session-authed browser variants (GET /api/runs/:id/junit, GET /api/suite-runs/:id/junit) return the same body for the dashboard.

What maps to what:

Sample GitHub Actions step — trigger a run, wait for it, download its JUnit, and publish it:

- name: Run Vera + publish JUnit
  run: |
    # trigger a single test (or suiteId / all:true) and capture the run id
    run_id=$(curl -sS -X POST \
      -H "Authorization: Bearer ${{ secrets.VERA_TOKEN }}" \
      -H "Content-Type: application/json" \
      -d '{"testId": "'"${{ vars.VERA_TEST_ID }}"'"}' \
      "${{ secrets.VERA_URL }}/api/v1/projects/${{ vars.VERA_PROJECT_ID }}/run" \
      | jq -r '.runIds[0]')

    # block until it reaches a terminal status, then fetch the JUnit report
    curl -sS -H "Authorization: Bearer ${{ secrets.VERA_TOKEN }}" \
      "${{ secrets.VERA_URL }}/api/v1/runs/$run_id/wait" > /dev/null
    curl -sS -H "Authorization: Bearer ${{ secrets.VERA_TOKEN }}" \
      "${{ secrets.VERA_URL }}/api/v1/runs/$run_id/junit" -o vera-junit.xml

- name: Publish test report
  if: always()
  uses: mikepenz/action-junit-report@v4
  with:
    report_paths: vera-junit.xml

The batch trigger ({"suiteId": "…"} / {"all": true}) returns a runIds array — fetch GET /api/v1/runs/<id>/junit per id and hand the reporter all the files, or, for suite runs recorded as a suite_runs row (dashboard- or scheduler-triggered), pull the single combined GET /api/v1/suite-runs/<suiteRunId>/junit.

8. Run only the impacted tests (vera check)

Everything above runs a fixed selection. On a pull request you usually want the selection to follow the diff — which is vera check --run, documented in full at change impact:

- uses: actions/checkout@v4
  with:
    fetch-depth: 0 # so `origin/<base_ref>` resolves

- name: Impacted tests
  run: npx vera-agent check --base origin/${{ github.base_ref }} --run
  env:
    VERA_API_URL: ${{ vars.VERA_API_URL }}
    VERA_PROJECT_ID: ${{ vars.VERA_PROJECT_ID }}
    VERA_TOKEN: ${{ secrets.VERA_TOKEN }}

Three things to know before you paste that anywhere:

• The variable names differ from the rest of this page. The CLI reads VERA_API_URL / VERA_PROJECT_ID / VERA_TOKEN; the vera-run action takes api-url / project / api-token inputs, and the shell gate reads VERA_URL / VERA_PROJECT / VERA_TOKEN. Only the token name is shared.
• fetch-depth: 0 is load-bearing. actions/checkout defaults to a shallow clone where origin/<base_ref> does not exist, and an explicit --base that does not resolve fails the step.
• --strict is opt-in and comes later. It fails the build on a business rule the diff touches with no adequate test, and on a project whose map coverage is still near zero it would be red on every PR. Turn it on once the numbers say you can — the reasoning is in §9 of the change-impact guide.

Exit codes are 0 clean · 1 usage/config · 2 could not complete · 3 a test failed · 4 --strict found a gap · 5 --fail-on-healed found a healed pass (only a --local run can heal). A sticky PR comment rendering this report is planned, not shipped; today it is a step that passes or fails and prints to the log, plus --json if you want to build your own.

Notes

• Tokens are org-scoped; a token only reaches its own org's projects.
• Store only ciphertext of the token on the server (SHA-256) — the plaintext is never retrievable after creation. Rotate by minting a new token and revoking the old one.
• Healed passes: healed is true/a count when a run reached green only after a stale selector was rewritten mid-run. It never changes status; gate on it with fail-on-healed / VERA_FAIL_ON_HEALED if you want a human to confirm the rewritten step still exercises what it was written for. Note that a CI-started run does not heal today, so that opt-in is currently inert — the full explanation. fail-on-flaky is unaffected.
• Quarantine/flake semantics: a quarantined test keeps running and reporting, but its failures never fail status; a run that failed then passed on a retry reports passedOnRetry. Both appear in the response so CI can surface them as warnings.