> ## Documentation Index
> Fetch the complete documentation index at: https://docs.verial.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Authoring a Benchmark

> Compose an environment, build a benchmark of tasks with typed criteria, and publish a versioned slug.

This guide walks through authoring a benchmark end to end: creating an environment, wiring simulators and datasets, adding tasks with typed criteria, and publishing a versioned `slug@version` that other teams (or your own Solvers) can run.

Authoring uses the **internal API** with an [organization API key](/api-reference/authentication) (`vk_*`). Running a published benchmark uses a [Solver key](/guides/solver-keys) and the public `/v1` API instead.

## Who This Is For

* Teams building internal benchmarks for their own agents.
* Organizations publishing benchmarks for the wider healthcare AI ecosystem (`visibility=Public`).
* Anyone authoring a new task or criterion against an existing environment.

## Prerequisites

```bash theme={null}
export VERIAL_API_KEY=vk_xxx
```

Examples below mix the TypeScript SDK (for resources the SDK exposes today: benchmarks, datasets, environments) with `curl` (for simulators, tasks, criteria, which are currently available only on the REST API). The same operations are also available via the [MCP server](/mcp/overview).

```typescript theme={null}
import { Verial } from "@verial-ai/sdk";

const verial = new Verial({ apiKey: process.env.VERIAL_API_KEY! });
```

## 1. Create an Environment

An [environment](/guides/concepts/environments) is a reusable simulated health system. Start with a container:

```typescript theme={null}
const env = await verial.environments.create({
  name: "Regional Medical Center",
});
// → use env.id as $ENVIRONMENT_ID in the curl examples below
```

## 2. Add Simulators

[Simulators](/guides/simulators/overview) are individual simulated interfaces (FHIR EHR, voice line, payer portal, SFTP drop, HL7 endpoint, X12 clearinghouse, fax, messaging). Create each simulator and link it to the environment:

```bash theme={null}
# Create the FHIR simulator
curl -X POST https://api.verial.ai/simulators \
  -H "Authorization: Bearer $VERIAL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"type": "Fhir", "name": "Primary EHR"}'
# → save the returned id as $FHIR_SIMULATOR_ID

# Create the payer simulator
curl -X POST https://api.verial.ai/simulators \
  -H "Authorization: Bearer $VERIAL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"type": "Payer", "name": "BlueCross Portal", "config": {"portal": "radmd"}}'
# → save the returned id as $PAYER_SIMULATOR_ID

# Link both simulators to the environment (env id from step 1 → $ENVIRONMENT_ID)
curl -X POST "https://api.verial.ai/environments/$ENVIRONMENT_ID/simulators/$FHIR_SIMULATOR_ID" \
  -H "Authorization: Bearer $VERIAL_API_KEY"

curl -X POST "https://api.verial.ai/environments/$ENVIRONMENT_ID/simulators/$PAYER_SIMULATOR_ID" \
  -H "Authorization: Bearer $VERIAL_API_KEY"
```

## 3. Attach Datasets

[Datasets](/guides/concepts/datasets) contain synthetic data that populates simulator sandboxes at rollout time. FHIR datasets carry a JSON config with patients, conditions, medications, etc. Files and SFTP datasets carry a manifest plus actual files stored in GCS.

```typescript theme={null}
const dataset = await verial.datasets.create({
  name: "Primary Care Patients",
  data: {
    patients: [
      {
        name: "John Smith",
        dob: "1965-03-15",
        gender: "male",
        conditions: ["Type 2 Diabetes"],
        insurance: { plan: "BlueCross PPO", member_id: "BCB123456789" },
      },
    ],
  },
});
```

Datasets are linked to sandboxes (the runtime instances of simulators) when a playground is provisioned. Sandbox linking copies the dataset into a per-run child dataset, so the original stays pristine.

## 4. Create the Benchmark

A [benchmark](/guides/concepts/benchmarks) groups tasks against one environment:

```typescript theme={null}
const benchmark = await verial.benchmarks.create({
  name: "Prior Authorization Workflow",
  environmentId: env.id,
  timeout: 300,
  concurrency: 1,
});
// → use benchmark.id as $BENCHMARK_ID below
```

`timeout` is the per-task execution budget in seconds; tasks that exceed it are recorded with `verdict: "fail"` and `score: 0`. `concurrency` is the number of task rollouts that may run in parallel within a single benchmark run.

## 5. Add Tasks

Each [task](/guides/concepts/tasks) is one test case. A task carries:

* `taskItem`: structured payload with instruction, trigger, expected inputs.
* `scenario`: optional pre-rollout steps run by the scenario runner (for example, drop a fax into the inbox before the agent starts).
* `entities`: `DatasetEntity` bindings scoping the task to specific synthetic records.
* `tags`: for filtering and organization.

```bash theme={null}
curl -X POST https://api.verial.ai/tasks \
  -H "Authorization: Bearer $VERIAL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "benchmark_id": "'"$BENCHMARK_ID"'",
    "name": "Submit Prior Auth for MRI",
    "task_item": {
      "instruction": "Submit a prior authorization for an MRI of the lumbar spine."
    },
    "tags": ["prior-auth", "imaging"]
  }'
# → save the returned id as $TASK_ID
```

## 6. Attach Criteria

[Criteria](/guides/concepts/criteria) are typed assertions the verification engine runs after the rollout. Each one has a `label`, an `assertion` spec, a `weight`, and an optional `axis` for per-axis scoring.

Prefer multiple narrow criteria over one compound criterion: one observable outcome per criterion makes failures easier to diagnose.

### Example: portal-state-match

```bash theme={null}
curl -X POST https://api.verial.ai/criteria \
  -H "Authorization: Bearer $VERIAL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "task_id": "'"$TASK_ID"'",
    "label": "Prior auth submitted with correct CPT",
    "weight": 1.0,
    "axis": "correctness",
    "assertion": {
      "assert": "portal-state-match",
      "correlate_by": { "resource": "prior_auth_requests", "key": "request_id" },
      "assertions": [
        { "path": "status", "expected": "submitted" },
        { "path": "cpt_code", "expected": "72148" }
      ]
    }
  }'
```

### Example: fhir-resource-state

```bash theme={null}
curl -X POST https://api.verial.ai/criteria \
  -H "Authorization: Bearer $VERIAL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "task_id": "'"$TASK_ID"'",
    "label": "Follow-up appointment booked",
    "weight": 1.0,
    "axis": "correctness",
    "assertion": {
      "assert": "fhir-resource-state",
      "resource_type": "Appointment",
      "search": { "patient": "Patient/john-smith", "status": "booked" },
      "fields": [
        { "path": "participant.0.actor.display", "expected": "Dr. Rivera" }
      ]
    }
  }'
```

### Example: voice-transcript (negative assertion)

```bash theme={null}
curl -X POST https://api.verial.ai/criteria \
  -H "Authorization: Bearer $VERIAL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "task_id": "'"$TASK_ID"'",
    "label": "No PHI leaked on voicemail",
    "weight": 0.5,
    "axis": "safety",
    "assertion": {
      "assert": "voice-transcript",
      "speaker": "agent",
      "not_contains": ["social security number", "SSN"]
    }
  }'
```

See the [Criteria concept page](/guides/concepts/criteria) for all supported check types (`fhir-resource-state`, `hl7-structural`, `portal-state-match`, `sftp-file-present`, `voice-transcript`, `x12-response`) and full assertion reference on the [Criteria API page](/api-reference/resources/criteria).

## 7. Set Slug and Publish

Before publishing, assign a URL-safe slug (lowercase alphanumeric with hyphens):

```typescript theme={null}
await verial.benchmarks.update({
  id: benchmark.id,
  slug: "prior-auth-workflow",
});
```

Then publish a version. Versions are positive integers; once published, a benchmark is immutable and addressable as `slug@version`:

```bash theme={null}
curl -X POST "https://api.verial.ai/benchmarks/$BENCHMARK_ID/publish" \
  -H "Authorization: Bearer $VERIAL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "version": 1,
    "sponsor_name": "Acme Health",
    "methodology_url": "https://acme.example/benchmarks/prior-auth"
  }'
```

<Note>
  Published benchmarks are immutable. To iterate, clone the benchmark and
  publish the clone as the next version (`version: 2`). Consumers can continue
  pinning to `slug@1` while you ship `slug@2`.
</Note>

## 8. Set Visibility

Visibility controls who can run a published benchmark:

* `Private` (default): only Solvers in the owning organization can run it.
* `Public`: any Solver in any organization can run it. The benchmark appears in public listings and leaderboards.

```typescript theme={null}
await verial.benchmarks.update({
  id: benchmark.id,
  visibility: "Public",
});
```

<Info>
  You cannot make a benchmark `Public` without a slug. Publish with a slug
  first, verify it runs as expected, then flip visibility.
</Info>

## Versioning

* Cut a new version when criteria, tasks, or the linked environment change in a way that shifts scores.
* Keep trivial edits (documentation, labels that do not affect scoring) to the same version by applying them before publishing.
* Communicate version changes in the benchmark's `overview`, `scoring_rubric`, and `limitations` fields.

## Next Steps

<CardGroup cols={2}>
  <Card title="Criteria" icon="check" href="/guides/concepts/criteria">
    Reference for every supported check type with annotated examples.
  </Card>

  <Card title="Verification" icon="scale-balanced" href="/guides/concepts/verification">
    How per-criterion results aggregate into task and benchmark scores.
  </Card>

  <Card title="Environments" icon="hospital" href="/guides/concepts/environments">
    Compose simulators and datasets into reusable simulated health systems.
  </Card>

  <Card title="Benchmarks API" icon="code" href="/api-reference/resources/benchmarks">
    REST endpoints for create, update, publish, and list.
  </Card>
</CardGroup>