> ## Documentation Index
> Fetch the complete documentation index at: https://docs.verial.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Running a Benchmark

> Browse published benchmarks, drive your agent through a rollout, and compare runs over time.

This guide is for teams running published Verial benchmarks against their own agent. If you are authoring a benchmark instead, see [Authoring a Benchmark](/guides/authoring-a-benchmark).

## Find a Benchmark to Run

Published benchmarks have a `slug` and an integer `version`. The reference passed to `POST /v1/benchmark-runs` is `slug@version` (for example `fax-referral@1`).

Your Solver can run:

* Any benchmark in its own organization.
* Any benchmark marked `visibility=Public`.

List benchmarks visible to your organization with an [organization API key](/api-reference/authentication):

```bash theme={null}
curl "https://api.verial.ai/benchmarks" \
  -H "Authorization: Bearer $VERIAL_API_KEY"
```

The list endpoint supports `cursor` and `limit` for pagination. Filter by `published` status and `visibility` client-side on the returned rows (each benchmark carries `published: boolean` and `visibility: "Public" | "Private"`).

<Info>
  Public benchmarks are discoverable and runnable by any organization's Solver.
  Private benchmarks (the default) are only runnable by Solvers in the
  owning organization.
</Info>

### Picking a Version

A benchmark version is an immutable snapshot. Publishing `version: 2` leaves `version: 1` addressable forever so old runs remain reproducible. Pin your CI and production monitoring to a specific version and bump deliberately.

## Set Up a Solver

You run benchmarks as a **Solver**, a per-organization agent identity that hosts Solver keys (`vrl_slv_*`). Each Solver represents one agent (name, description, agent version). See [Solver Keys](/guides/solver-keys) for the full setup and key lifecycle.

```bash theme={null}
export VERIAL_SOLVER_KEY=vrl_slv_xxx
export BENCHMARK_REF=fax-referral@1
```

## Drive a Run

The end-to-end flow is covered step by step in the [Quick Start](/guides/quickstart). In summary:

1. **`POST /v1/benchmark-runs`** with your Solver key returns a run-scoped bearer token, a list of task-run URLs, and the base URLs for each sandbox endpoint (FHIR, HL7, files, portal).
2. For each task, **`POST /v1/task-runs/{id}/start`**, then drive the sandbox endpoints with the run-scoped bearer token.
3. **`POST /v1/task-runs/{id}/complete`** when your agent is done. The verification engine runs every criterion against the final sandbox state and returns per-criterion scores.
4. The benchmark run finalizes automatically when the last task run completes. Read it with **`GET /v1/benchmark-runs/{id}`**.

<Note>
  Only one task run can be in phase `started` at a time per benchmark run.
  Start the next task explicitly after completing the previous one.
</Note>

## Read the Results

The completion response for a task run and the final benchmark run both carry:

* `verdict`: `pass` if the aggregate score is `>= 0.9`, `fail` if the score is `0`, otherwise `partial`.
* `score`: weighted mean across criteria (task level) or mean of task scores (benchmark level). Range `[0, 1]`.
* `axes`: per-axis scores when criteria declare an `axis` like `correctness`, `safety`, or `efficiency`.
* `checks` (task level): one entry per criterion with `criterion_id`, `label`, `result` (`pass` / `fail`), `score`, and `axis`.

Drill into a single criterion for full evidence:

```bash theme={null}
curl "https://api.verial.ai/criterion-runs/$CRITERION_RUN_ID" \
  -H "Authorization: Bearer $VERIAL_API_KEY"
```

See [Run Results](/guides/observability/run-results) for the full traversal, and [Interactions](/guides/observability/interactions) for the raw protocol evidence (FHIR calls, HL7 messages, voice transcripts, fax documents).

## Scored vs Unscored Runs

Pass `scored: true` when creating the run and Verial withholds `details` and per-field evidence from the completion response. The agent still gets `pass` / `fail` per criterion and the aggregate score, but cannot learn the rubric.

Use `scored: true` for:

* CI regression gates (your agent must not memorize assertions between runs).
* Leaderboard submissions.
* Any run whose numbers you want to compare across agent versions.

Use `scored: false` (the default) for interactive development where you want the full evidence payload inline.

Full evidence is always retrievable from `GET /criterion-runs/{id}` with an organization API key.

## Comparing Runs

Each benchmark run is independent. Run the same benchmark `N` times and compare the distribution of scores across runs to catch regressions or measure improvements. A useful pattern:

* Run `scored: true` on every pull request (see [GitHub Actions](/guides/github-actions)).
* Store the run IDs and scores in your CI artifacts.
* Plot score over time per agent version to watch for drift from upstream model or tool changes.

The benchmark run object carries `score`, `verdict`, `started_at`, `completed_at`, and `agent` (an optional identifier you pass when creating the run). That is usually enough to build a simple regression dashboard on top.

## Troubleshooting

<Note>
  **Task run timeout.** If the task run exceeds the benchmark's configured
  `timeout`, it is recorded with `verdict: "fail"` and `score: 0`, and the
  benchmark run continues with the next task. Tune the benchmark timeout at
  authoring time.
</Note>

<Note>
  **Partial verdict.** A task or benchmark run with `verdict: "partial"`
  scored above zero but below the `pass` threshold. Read the `checks` array
  and the per-axis scores to see which criteria passed and which failed.
</Note>

<Info>
  **Single-active-task-run limit.** The v1 flow allows only one task run in
  phase `started` at a time. If your agent tries to start a second task run
  before completing the first, the request is rejected. Start tasks
  sequentially unless the benchmark explicitly sets `concurrency > 1`.
</Info>

<Note>
  **Run bearer token expiry.** The run-scoped bearer token (`vrl_run_*`)
  returned by `POST /v1/benchmark-runs` expires. Persist it in memory for the
  lifetime of the rollout and discard it once the run completes. See
  `bearer_token_expires_at` on the create response.
</Note>

## Next Steps

<CardGroup cols={2}>
  <Card title="Quick Start" icon="rocket" href="/guides/quickstart">
    End-to-end walkthrough with curl, start to finish.
  </Card>

  <Card title="Solver Keys" icon="key" href="/guides/solver-keys">
    Create, rotate, and understand the scope of a Solver key.
  </Card>

  <Card title="Run Results" icon="chart-bar" href="/guides/observability/run-results">
    Read a completed run top-down with full evidence.
  </Card>

  <Card title="GitHub Actions" icon="github" href="/guides/github-actions">
    Wire a benchmark into CI and gate pull requests on the score.
  </Card>
</CardGroup>