> ## Documentation Index
> Fetch the complete documentation index at: https://docs.verial.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmark Runs

> Rollouts of a benchmark, scored by the verification engine.

A **benchmark run** is a single execution of a [benchmark](/guides/concepts/benchmarks). The agent drives a rollout for each task against a provisioned playground. When a task run completes, the [verification engine](/guides/concepts/verification) runs each criterion against the final sandbox state and produces [Criterion Runs](/api-reference/resources/criterion-runs), a task score, and a verdict.

## Lifecycle

```mermaid theme={null}
graph LR
  C["Created<br/>(playground provisioning)"]:::state --> S["Started"]:::state
  S --> Co["Completed"]:::state
  S --> Ca["Cancelled"]:::state

  classDef state fill:#2563eb,color:#fff,stroke:#1d4ed8
```

1. **Created.** Benchmark run is created, a playground is provisioned, TaskRun rows are created in phase `created`.
2. **Started.** The agent calls `POST /v1/task-runs/{id}/start` on the first task.
3. **Completed.** All task runs are complete; the benchmark run score is the mean of task scores.

## Two Entry Points

| Flow                             | Auth                          | Typical caller                                                                         |
| -------------------------------- | ----------------------------- | -------------------------------------------------------------------------------------- |
| Internal `POST /benchmark-runs`  | Organization API key (Bearer) | Verial tooling                                                                         |
| Public `POST /v1/benchmark-runs` | Solver key (Bearer)           | External developers running a published benchmark from their own organization's Solver |

External integrations should use the **v1 flow**. See the [Quick Start](/guides/quickstart) for an end-to-end walkthrough.

## The v1 Flow

```mermaid theme={null}
graph TD
  Start["POST /v1/benchmark-runs"]:::step --> TokenOut["Returns bearer token<br/>+ task-run URLs<br/>+ sandbox endpoint paths"]:::step
  TokenOut --> AgentDrive["Agent drives rollout against<br/>/v1/benchmark-runs/{id}/fhir,<br/>/hl7, /files, /portal"]:::step
  AgentDrive --> Complete["POST /v1/task-runs/{id}/complete"]:::step
  Complete --> Verify["Verification engine runs criteria"]:::step
  Verify --> Score["Returns score, verdict, axes, checks"]:::step

  classDef step fill:#2563eb,color:#fff,stroke:#1d4ed8
```

## Scoring

### Criterion Score

Every criterion produces a `score` in `[0, 1]`.

### Task Score

Weighted mean across criteria:

```
task_score = sum(score_i * weight_i) / sum(weight_i)
```

### Per-Axis Scores

Criteria sharing an `axis` are aggregated into a per-axis weighted mean, returned in `axes` on the completion response.

### Benchmark Run Score

Mean of task scores.

### Verdict

* `pass` when the aggregate is `>= 0.9`
* `partial` when it is above zero
* `fail` otherwise

## Scored vs Unscored Runs

Benchmark runs can be created with `scored: true`. When the run is scored, completion responses omit `details` and per-field evidence to avoid leaking the rubric to the agent. You can always retrieve the full evidence later via `GET /criterion-runs/{id}`.

## Cancellation

A run can be cancelled via `POST /benchmark-runs/{id}/cancel`. Completed task results are retained.

## Comparing Runs

Run the same benchmark multiple times to measure improvements or regressions. Each run is independent.

## Next Steps

<CardGroup cols={2}>
  <Card title="Criteria" icon="check" href="/guides/concepts/criteria">
    Understand how typed assertions score task runs.
  </Card>

  <Card title="Quick Start" icon="rocket" href="/guides/quickstart">
    Drive a v1 benchmark run end to end.
  </Card>
</CardGroup>