Skip to main content
Manage criteria, the typed assertions that the verification engine runs against a task run.
The MCP server currently still exposes the legacy evals tool. Use it for now to write criteria; the assert argument accepts a JSON-encoded assertion spec (see Evals legacy notes). A dedicated criteria tool that accepts the structured assertion object is planned.

​
Actions (planned)

ActionDescription
listList all criteria for a task
createCreate a new criterion
getGet a criterion by ID
updateUpdate a criterion
deleteDelete a criterion

​
Parameters

​
list

ParameterTypeRequiredDescription
taskIdstringyesTask ID to list criteria for

​
create

ParameterTypeRequiredDescription
taskIdstringyesTask ID to attach the criterion to
labelstringyesShort human-readable label
assertionobjectyesTyped assertion spec. See Criteria API
weightnumbernoRelative weight (default 1)
axisstringnoOptional scoring axis
inputEntityIdstringnoOptional DatasetEntity ID the criterion is scoped to

​
get

ParameterTypeRequiredDescription
idstringyesCriterion ID

​
update

ParameterTypeRequiredDescription
idstringyesCriterion ID
labelstringnoUpdated label
assertionobjectnoUpdated assertion spec
weightnumbernoUpdated weight
axisstringnoUpdated axis
inputEntityIdstringnoUpdated scoped DatasetEntity ID

​
delete

ParameterTypeRequiredDescription
idstringyesCriterion ID

​
Examples

​
Create a FHIR criterion

{
  "action": "create",
  "taskId": "task_xyz789",
  "label": "Appointment booked with correct provider",
  "weight": 1,
  "axis": "correctness",
  "assertion": {
    "assert": "fhir-resource-state",
    "resource_type": "Appointment",
    "search": { "patient": "Patient/john-smith", "status": "booked" },
    "fields": [
      { "path": "participant.0.actor.display", "expected": "Dr. Rivera" }
    ]
  }
}

​
Create a voice transcript criterion

{
  "action": "create",
  "taskId": "task_xyz789",
  "label": "Required disclosures",
  "weight": 0.5,
  "assertion": {
    "assert": "voice-transcript",
    "speaker": "agent",
    "contains": ["member ID", "date of birth"]
  }
}

​
List criteria for a task

{
  "action": "list",
  "taskId": "task_xyz789"
}