Skip to main content

​
Prerequisites

  • A Verial account with an API key
  • Node.js 18+ (for the SDK) or any HTTP client (for the REST API)
Set your API key: 
export VERIAL_API_KEY=vk_xxx

​
1. Install the SDK

npm install @verial-ai/sdk

import Verial from '@verial-ai/sdk'

const verial = new Verial({ apiKey: process.env.VERIAL_API_KEY })

​
2. Create a Simulator and Environment

First create a FHIR simulator, then an environment that links it: 
// Create a FHIR simulator
const simulator = await verial.simulators.create({
  type: 'FHIR',
  name: 'Primary Care EHR',
})

// Create an environment and link the simulator
const environment = await verial.environments.create({
  name: 'Primary Care Clinic',
})

await verial.environments.addSimulator({
  environmentId: environment.id,
  simulatorId: simulator.id,
})

​
3. Create a Benchmark with Tasks and Evals

A benchmark defines what to test. Create it linked to the environment, then add tasks with evals: 
const benchmark = await verial.benchmarks.create({
  name: 'Prior Auth Basics',
  environmentId: environment.id,
  timeout: 300,
})

// Add a task
const task = await verial.tasks.create({
  benchmarkId: benchmark.id,
  name: 'Submit Prior Auth',
  instruction: 'Submit a prior authorization for an MRI of the lumbar spine',
})

// Add evals to the task
await verial.evals.create({
  taskId: task.id,
  label: 'pa-submitted',
  assert: 'A prior auth request was submitted to the payer',
  weight: 1.0,
})

await verial.evals.create({
  taskId: task.id,
  label: 'correct-cpt',
  assert: 'The request includes CPT code 72148',
  weight: 0.5,
})

​
4. Start a Run

A run executes the benchmark. Verial provisions sandbox instances from the environment’s simulators, and your agent connects to perform tasks. 
const run = await verial.runs.create({
  benchmarkId: benchmark.id,
})

console.log(run.id)     // run_clxyz...
console.log(run.status) // "running"

​
5. Check Results

Once your agent completes the tasks (or the timeout is reached), retrieve the run to see results. 
const result = await verial.runs.get({ id: run.id })

console.log(result.status)  // "completed"
console.log(result.score)   // 0.85
console.log(result.verdict) // "pass"

​
Next Steps

Environments

Learn about simulator types and environment configuration.

Evaluations

Write effective evaluation criteria for your benchmarks.

SDK Reference

Install and configure the TypeScript SDK.

API Reference

REST API endpoints for all resources.