CI integration

Trigger Vera tests from CI with an API token and fail the build when a test fails — while quarantined tests, retry-flakes and healed passes surface as warnings instead of breaking the merge.

This page runs a fixed set: a test, a suite, or the whole project. To run only the tests a pull request's diff actually touches, see §8 and change impact.

1. Mint a token

In the dashboard, open Settings → CI Tokens (or POST /api/tokens with { "name": "ci" }). The vera_… token is shown once — store it as a CI secret (e.g. VERA_TOKEN). Also note your project id and your Vera base URL (VERA_URL) — e.g. the hosted deployment https://vera-agent.com.

2. GitHub Actions — the vera-run action (recommended)

name: E2E (Vera)
on: [pull_request]

jobs:
  vera:
    runs-on: ubuntu-latest
    steps:
      # ... deploy your PR preview and capture its URL ...
      - uses: mahmoodnasr/vera-run@v1 # once published; or vendor the dir and use `./.github/actions/vera-run`
        with:
          api-url: ${{ secrets.VERA_URL }}
          api-token: ${{ secrets.VERA_TOKEN }}
          project: ${{ vars.VERA_PROJECT_ID }}
          # optional:
          # suite: suite_xyz              # or `test: test_xyz`
          # environment: staging          # named Vera environment profile
          # preview-url: ${{ steps.deploy.outputs.url }}
          # fail-on-flaky: 'true'
          # fail-on-healed: 'true'

The action fails the job only on blocking failures: a failed quarantined test or a test that passed on retry produces a ::warning:: annotation and a job-summary table (with a report link) instead. When preview-url is omitted it falls back to VERCEL_URL / DEPLOY_PRIME_URL / DEPLOY_URL — so the same suite runs against every PR's preview deploy. See actions/vera-run for all inputs/outputs.

3. GitLab CI

Copy ci/gitlab/.gitlab-ci.yml into your project's .gitlab-ci.yml (or include: it), then:

1. Settings → CI/CD → Variables: add VERA_TOKEN = your vera_… token, Masked (and Protected if your MR branches are protected).
2. Set VERA_URL and VERA_PROJECT in the job's variables: block.

vera-e2e:
  stage: test
  image:
    name: alpine/curl:latest
    entrypoint: ['']
  variables:
    VERA_URL: 'https://vera.example.com'
    VERA_PROJECT: 'proj_abc123'
    VERA_PREVIEW_URL: '$CI_ENVIRONMENT_URL' # review-app URL, when your deploy job sets `environment:`
  script:
    - | # the embedded gate script — see the template for the full body
  artifacts:
    when: always
    reports:
      junit: vera-junit/*.xml # per-step results in the MR test widget
      dotenv: .vera/result.env # VERA_STATUS/VERA_PASSED/... for later jobs

The job needs only curl and a POSIX shell — no Node, no jq, no install step. It deliberately does not set allow_failure: true: the script decides what is blocking, and marking the whole job non-blocking would hide real failures too.

4. Bitbucket Pipelines

Copy ci/bitbucket/bitbucket-pipelines.yml into bitbucket-pipelines.yml (or merge its step into yours), then:

1. Repository settings → Repository variables: add VERA_TOKEN = your vera_… token, Secured.
2. Edit the export VERA_URL= / export VERA_PROJECT= lines at the top of the step (Bitbucket has no per-step variables: block).

The step writes JUnit XML to test-results/, which Bitbucket picks up automatically and renders in the build's Tests tab. Bitbucket's default image already has curl.

5. Gating parity across CI providers

All three surfaces gate identically. This is a product promise, not a coincidence: a quarantined test that reddened your GitLab pipeline while the same test only warned on GitHub would make quarantine worthless.

Opt into failing on retry-flakes with fail-on-flaky: 'true' (GitHub) or VERA_FAIL_ON_FLAKY: 'true' (GitLab/Bitbucket). Quarantined failures stay non-gating either way — quarantine is the escape hatch, and an escape hatch that can still fail the build is not one.

The two opt-ins are independent: fail-on-flaky never gates a healed pass, and fail-on-healed never gates a flake. Both are exercised as their own rows in the parity suite.

Green is not one thing

A run that reports passed got there in one of three ways, and Vera tells you which:

Healing only ever rewrites the locator, never the expectation — the assertion still had to hold against your app, which is why a healed pass is not a failure. What it can hide is a semantic change: if a Delete customer button disappears and the healer binds the step to a neighbouring control, a broad assertion can still pass. The interaction the test was written to exercise is gone, and a pass/fail column cannot show you that.

So healed is reported beside the status rather than folded into it, and the run detail names every rewritten selector (before → after). Teams gating a release branch typically set fail-on-healed: 'true' there and leave it off for feature branches: a heal is then a merge-blocking prompt for a human to confirm the test still means what it meant.

How the promise is kept:

• GitLab and Bitbucket run the same script, ci/vera-gate.sh, embedded verbatim in both templates.
• apps/server/src/services/ci-gating-parity.test.ts stands up a stub /api/v1, executes both the GitHub Action and the shell gate against every row of that table, and fails if their exit codes differ or if either template's embedded copy has drifted. pnpm test runs it, so the table above cannot quietly go stale.

6. The raw API (any CI system)

• POST /api/v1/projects/:id/run — body { "testId": "…" }, { "suiteId": "…" }, { "testIds": ["…", "…"] } (max 50; more is 400 too_many_test_ids and nothing runs — a result built from the first 50 of 84 would read like a complete one), or { "all": true }. Exactly one of them: combining testIds with testId / suiteId / all is rejected. Optional "environment": "staging" (named profile) and "baseUrlOverride": "https://pr-42.example.com" (wins over the environment — meant for per-PR preview URLs). Runs synchronously and returns the terminal result:

  {
    "runIds": ["…"],
    "status": "passed",
    "passed": 12,
    "failed": 1,
    "blockingFailed": 0,
    "quarantinedFailed": 1,
    "passedOnRetry": 2,
    "healed": 1,
    "reportUrl": "https://vera.example.com/projects/proj_x/tests"
  }

  Gate CI on status (already quarantine-aware: only non-quarantined failures fail it) or on blockingFailed. healed counts the runs that reached green only after self-heal rewrote a selector — never part of status, always reported. Single-test runs return { "status", "blocking", "quarantined", "passedOnRetry", "healed", "reportUrl" } — gate on blocking.

• GET /api/v1/runs/:id — current status (+ passedOnRetry, healed).

• GET /api/v1/runs/:id/wait — long-polls up to 60s for a terminal status (202 if still running).

• GET /api/v1/runs/:id/junit — the run as JUnit XML (see §7).

• GET /api/v1/suite-runs/:id/junit — a suite run as JUnit XML (one <testsuite> per member run).

All require Authorization: Bearer vera_…. A revoked/invalid token → 401.

resp=$(curl -sS -X POST \
  -H "Authorization: Bearer $VERA_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"all": true}' \
  "$VERA_URL/api/v1/projects/$PROJECT_ID/run")
[ "$(echo "$resp" | jq -r '.status')" = "passed" ] || { echo "::error::Vera run failed"; exit 1; }

7. JUnit XML — render results in the PR checks UI

Every run (and every suite run) can be exported as JUnit XML, the format that test-report actions understand. Feed it to dorny/test-reporter, mikepenz/action-junit-report, or your CI's native JUnit ingester and you get a per-step pass/fail breakdown attached to the check — no scraping the JSON.

• GET /api/v1/runs/:id/junit → one <testsuite> (the test), one <testcase> per step.
• GET /api/v1/suite-runs/:id/junit → a <testsuites> wrapping one <testsuite> per member run.

Both respond Content-Type: application/xml with an attachment Content-Disposition, and require Authorization: Bearer vera_…. The session-authed browser variants (GET /api/runs/:id/junit, GET /api/suite-runs/:id/junit) return the same body for the dashboard.

What maps to what:

Sample GitHub Actions step — trigger a run, wait for it, download its JUnit, and publish it:

- name: Run Vera + publish JUnit
  run: |
    # trigger a single test (or suiteId / all:true) and capture the run id
    run_id=$(curl -sS -X POST \
      -H "Authorization: Bearer ${{ secrets.VERA_TOKEN }}" \
      -H "Content-Type: application/json" \
      -d '{"testId": "'"${{ vars.VERA_TEST_ID }}"'"}' \
      "${{ secrets.VERA_URL }}/api/v1/projects/${{ vars.VERA_PROJECT_ID }}/run" \
      | jq -r '.runIds[0]')

    # block until it reaches a terminal status, then fetch the JUnit report
    curl -sS -H "Authorization: Bearer ${{ secrets.VERA_TOKEN }}" \
      "${{ secrets.VERA_URL }}/api/v1/runs/$run_id/wait" > /dev/null
    curl -sS -H "Authorization: Bearer ${{ secrets.VERA_TOKEN }}" \
      "${{ secrets.VERA_URL }}/api/v1/runs/$run_id/junit" -o vera-junit.xml

- name: Publish test report
  if: always()
  uses: mikepenz/action-junit-report@v4
  with:
    report_paths: vera-junit.xml

The batch trigger ({"suiteId": "…"} / {"all": true}) returns a runIds array — fetch GET /api/v1/runs/<id>/junit per id and hand the reporter all the files, or, for suite runs recorded as a suite_runs row (dashboard- or scheduler-triggered), pull the single combined GET /api/v1/suite-runs/<suiteRunId>/junit.

8. Run only the impacted tests (vera check)

Everything above runs a fixed selection. On a pull request you usually want the selection to follow the diff — which is vera check --run, documented in full at change impact:

- uses: actions/checkout@v4
  with:
    fetch-depth: 0 # so `origin/<base_ref>` resolves

- name: Impacted tests
  run: npx vera-agent check --base origin/${{ github.base_ref }} --run
  env:
    VERA_API_URL: ${{ vars.VERA_API_URL }}
    VERA_PROJECT_ID: ${{ vars.VERA_PROJECT_ID }}
    VERA_TOKEN: ${{ secrets.VERA_TOKEN }}

Three things to know before you paste that anywhere:

• The variable names differ from the rest of this page. The CLI reads VERA_API_URL / VERA_PROJECT_ID / VERA_TOKEN; the vera-run action takes api-url / project / api-token inputs, and the shell gate reads VERA_URL / VERA_PROJECT / VERA_TOKEN. Only the token name is shared.
• fetch-depth: 0 is load-bearing. actions/checkout defaults to a shallow clone where origin/<base_ref> does not exist, and an explicit --base that does not resolve fails the step.
• --strict is opt-in and comes later. It fails the build on a business rule the diff touches with no adequate test, and on a project whose map coverage is still near zero it would be red on every PR. Turn it on once the numbers say you can — the reasoning is in §9 of the change-impact guide.

Exit codes are 0 clean · 1 usage/config · 2 could not complete · 3 a test failed · 4 --strict found a gap. A sticky PR comment rendering this report is planned, not shipped; today it is a step that passes or fails and prints to the log, plus --json if you want to build your own.

Notes

• Tokens are org-scoped; a token only reaches its own org's projects.
• Store only ciphertext of the token on the server (SHA-256) — the plaintext is never retrievable after creation. Rotate by minting a new token and revoking the old one.
• Healed passes: healed is true/a count when a run reached green only after a stale selector was rewritten mid-run. It never changes status; gate on it with fail-on-healed / VERA_FAIL_ON_HEALED if you want a human to confirm the rewritten step still exercises what it was written for.
• Quarantine/flake semantics: a quarantined test keeps running and reporting, but its failures never fail status; a run that failed then passed on a retry reports passedOnRetry. Both appear in the response so CI can surface them as warnings.