Evaluations

Evaluations let you test an agent before it reaches your customers. You build a suite of test cases — scripted conversations with expectations — and run them against a frozen snapshot of the agent's configuration. Every response is scored automatically, and results show which cases passed, which failed, and why.

Evaluations are available on every plan. Plan limits apply to how many suites, cases and daily runs you get.

Where to find it

Open an agent and select the Evaluations tab.

Suites and cases

A suite groups related cases for one agent, such as "Refund policy accuracy". Each case has:

• Turns — the scripted user messages, in order.
• Expectations — what the answers must satisfy:
  • Required facts — text that must appear
  • Forbidden claims — text that must never appear
  • Expected and forbidden tool calls
  • Expected knowledge sources the answer should be grounded in
  • Maximum latency and maximum response length
  • A reference answer and rubric for the AI judge
• Severity — low, medium, high or critical. Critical failures are highlighted and can block publishing.
• Fixtures — canned responses for side-effecting tools during the test.

Cases are versioned: editing one creates a new version, so past runs still reference the exact definition they were scored against.

You can also draft a case from a real conversation. The user turns are lifted into a case with email addresses and phone numbers redacted automatically.

Scoring

Each case is scored by a mix of deterministic checks and an optional AI judge:

A case that lists no evaluators runs all of them, including the judge. Scores are weighted per case and aggregated into a run-level overall score, pass rate and critical-failure count.

Running an evaluation

1. In Suites, select a suite and choose Run evaluation.
2. The estimate dialog shows the exact cost before anything is charged, checked against your remaining monthly allowance.
3. Confirm. Items run in the background and results appear under Runs.

What a run costs

Evaluations draw on the same monthly allowance as chat:

• 1 interaction per simulated user turn, plus 1 per attachment on that turn
• plus 1 interaction per case that uses the AI judge, per execution

So a single-turn case with the judge costs 2 interactions. A case using only deterministic checks costs no judge interaction. See interaction counting.

Comparing against a baseline

On higher plans you can run a suite against both the current configuration and an earlier snapshot, producing a regression matrix showing exactly what improved and what got worse. You can also repeat each case several times to measure consistency.

Human review

On any finished run item, owners and admins can record a verdict — pass, fail or needs work — optionally overriding an evaluator's score, with notes. Human review sits alongside the automated result and never overwrites it.

Publish gates

A publish gate is a per-agent policy checked at the moment a change reaches production.

Thresholds cover minimum overall score, maximum regressions, zero critical failures, and requiring that the evaluated snapshot matches the configuration going live.

For an agent without versioning the gate applies at activation; with versioning it applies at each publish.

Snapshots

Every run pins a snapshot of the agent's full configuration, so results stay reproducible after the agent changes. Knowledge index content is not versioned — re-embedding a source after a snapshot can change retrieval when you rerun it. A drift badge indicates when this has happened.

Plan limits

Suite limits count active suites across the workspace; case limits count active cases per suite. Archiving frees a slot. One run counts as one per day, and a cancelled run gives the slot back.

Related pages

• Agent versioning
• Interaction counting
• Guardrails