---
title: "Evaluations"
url: "/evaluations"
description: "Test an agent's quality before customers see it — suites, test cases, scoring, and publish gates."
productArea: evaluations
audience: ["business-user", "admin"]
tags: ["evaluations", "testing"]
lastReviewed: 2026-08-11
---
# Evaluations (/evaluations)
Evaluations let you test an agent before it reaches your customers. You build a **suite** of
**test cases** — scripted conversations with expectations — and run them against a frozen
snapshot of the agent's configuration. Every response is scored automatically, and results show
which cases passed, which failed, and why.
Evaluations are available on **every plan**. Plan limits apply to how many suites, cases and
daily runs you get.
Evaluation runs happen in a sandbox. The agent's real side-effect tools — email, WhatsApp,
notifications, scheduled tasks — are blocked or answered with fixtures, so a test run can never
message a real customer or change live data.
## Where to find it [#where-to-find-it]
Open an agent and select the **Evaluations** tab.
| Sub-view | Purpose |
| ---------------- | --------------------------------------------------------------- |
| **Suites** | Create suites, add cases, launch runs |
| **Runs** | Run history, per-case scores, turn-by-turn traces, human review |
| **Publish gate** | A policy that warns or blocks when scores are too low |
## Suites and cases [#suites-and-cases]
A **suite** groups related cases for one agent, such as "Refund policy accuracy". Each **case**
has:
* **Turns** — the scripted user messages, in order.
* **Objective checks** — rule-based pass/fail, with no AI involved and no extra interaction
cost:
* **Required facts** — text that must appear
* **Forbidden claims** — text that must never appear
* **Required tool calls** — each entry asserts the tool was called; optionally it must also
match specific arguments (partially or exactly) and be called at least (or at most) a given
number of times
* **Forbidden tool calls** — tools the agent must never call
* **Required call order** — tools must be called in a given order, not necessarily back-to-back
* **Maximum tool calls** — a total budget across the whole case, so an agent that loops or
over-fetches fails even when every individual call is allowed
* **Expected knowledge sources** the answer should be grounded in
* **Maximum latency** and **maximum response length**. A case scored by the AI judge waits on an
extra scoring pass, so leaving **Maximum latency** blank on one gives it a more forgiving
default cap rather than no cap at all. Set a value to override it.
* **AI judge** — a **reference answer and rubric** the judge scores the response against. Filling
either adds one interaction per case execution. A separate **Check knowledge faithfulness**
tick-box has an AI check verify the answer only asserts what the retrieved knowledge supports;
it also adds one interaction per case execution and is scored n/a when nothing was retrieved.
* **Severity** — low, medium, high or critical. Critical failures are highlighted and can block
publishing.
* **Tags** — free-form grouping, such as `happy-path` or `security`. The tags already used in the
suite are offered as suggestions.
* **Test setup** — optionally freeze the case's runtime context so time-dependent behaviour
("remind me tomorrow at 9") stays deterministic across reruns: a **simulated time**, entered
as wall-clock time in a chosen **timezone**, and a simulated **channel**. Test setup is also
where you pick which of the suite's fixtures the case runs against — see
[fixtures](#fixtures) below.
Only a **forbidden tool call** counts as a critical safety breach. Argument, count and order
misses are correctness failures — they lower the tool-correctness score and can fail the case,
but they never flag it as a safety violation.
Cases are **versioned**: editing one creates a new version, so past runs still reference the
exact definition they were scored against.
Each case row in the suite view shows a strip of dots for its last few runs, oldest to newest,
colour-coded pass/fail/critical, with a relative "last run" time — so a case that's flaky or has
just started failing stands out without opening it.
You can also draft a case **from a real conversation**. The user turns are lifted into a case and
the customer's email addresses and phone numbers are replaced automatically with sample values —
realistic ones, such as `sarah@example.com`, so the case still runs the way the original
conversation did. Nothing the customer actually wrote in those fields is kept, and no message is
sent to a sample address: evaluations never deliver email, WhatsApp or SMS.
## Fixtures [#fixtures]
A **fixture** is a canned response for one tool, defined per suite. When the agent calls that
tool during a run, the fixture's response is returned instead of executing the real tool — so a
case can exercise "create the order, then confirm it" without ever writing real data.
Each fixture has:
* **Name** — an optional label, such as "Order not found". It is shown in place of the tool id in
the suite's fixture list and in the per-case picker, which is what makes two fixtures on the
same tool tellable apart. It is only a label: matching never depends on it.
* **Tool** — picked from a searchable catalog, grouped into write tools (blocked by default in
evaluations), the tools of any [connected app](/skills-and-tools) the agent has access to
(always blocked — see below), and read tools (which already run for real). You can also type an
exact tool id, which is what to do for a custom tool or when a connected app's authorisation has
lapsed and its tools are temporarily missing from the list.
* **Match mode** — when the fixture applies:
* **Any call** to the tool
* **Partial** — every field in the matcher must equal the call's arguments; extra arguments
are ignored
* **Exact** — the call's arguments must equal the matcher exactly
* **Matcher** — the argument shape to match, as JSON. The chosen tool's field names appear as
clickable keys that insert into the matcher, and the editor rejects field names the tool
doesn't accept.
* **Response** — the JSON body the agent receives when the fixture matches. It is prefilled with
the tool's real example payload, so shaping a realistic response starts from a working
template.
By default every fixture in the suite applies to every case. A case can narrow this under **Test
data & tool responses** in its **Test setup** by ticking only the fixtures it uses. Ticking a
fixture only supplies the simulated result if the agent calls that tool — it doesn't require the
call. To assert the call happens, add a required tool call under the case's objective checks.
When a side-effecting tool is called and **no fixture matches**, the case's fallback (also set
in **Test setup**) decides what happens: the call is **blocked** (the default), answered with a
generic **safe default**, or the item is marked unsupported. Read-only tools always run for
real, so retrieval and data lookups behave like production.
### Connected apps always need a fixture [#connected-apps-always-need-a-fixture]
Tools that come from a connected app — Google Calendar, DocuWare, anything reached over MCP — are
the one exception to all of the above. They are **always** blocked during a run without a fixture:
reads included, and whatever the case's fallback is set to. A real call would reach live data in
somebody else's system, and the app's own description of which of its calls are safe to repeat is
not something we can verify, so nothing is taken on trust.
In practice that means a case has to cover every connected-app call it will make, not just the
final one. Booking an appointment usually needs a fixture for the calendar lookup that comes first
as well as for the booking itself — without the first, the agent never reaches the second, and the
case fails on a missing fixture rather than on the behaviour you were testing.
These fixtures have no example payload to start from, since the response shape belongs to the other
system rather than to KlicForge. Copy the shape from a real response where you can — inventing
field names produces a case that passes against a payload the agent will never actually see.
### Reading a run's trace [#reading-a-runs-trace]
Each tool call in a run's trace is badged so you can see exactly what happened: **sim** for a
fixture response, **blocked** when nothing matched — and the detail distinguishes "no fixture
exists for this tool" from "a fixture exists but its matcher didn't fire", so a case-authoring
gap never reads as an agent failure. Expanding a call shows its arguments and result as
syntax-highlighted JSON.
A **blocked** badge is clickable. It opens the fixture editor with the tool and the call's real
arguments already filled in, so the usual fix for a blocked call is to click it, paste a realistic
response, save, and re-run.
## Scoring [#scoring]
Each case is scored by a mix of deterministic checks and an optional AI judge:
| Evaluator | Checks |
| ---------------------- | ------------------------------------------------------------------------------------------ |
| Task success | Required facts present, forbidden claims absent |
| Tool correctness | Required tools called with matching arguments, counts and order; a forbidden call critical |
| Knowledge grounding | The expected knowledge sources were actually retrieved |
| Knowledge faithfulness | The answer only asserts what the retrieved knowledge supports (opt-in) |
| Latency | The slowest turn is within the cap (judged cases default to a more forgiving one) |
| Reliability | No turn errored |
| Conciseness | Final response within the length cap |
| AI judge | A rubric-based quality score |
Knowledge grounding and knowledge faithfulness answer different questions. Grounding asks whether
retrieval found the right sources; faithfulness reads the answer and asks whether it stuck to what
those sources actually say. An agent can score 100% on grounding and still invent details, so the
two are scored separately. Faithfulness is skipped (scored "n/a") when nothing was retrieved —
that failure is grounding's to report.
A case that lists no evaluators runs **all** of them, except the two that cost an extra
interaction: the AI judge runs only when you supply a rubric or reference answer, and knowledge
faithfulness only when you tick it on. Scores are weighted per case and aggregated into a
run-level overall score, pass rate and critical-failure count.
Results also group the dimensions into three at-a-glance tiers — **execution** (right tools,
right arguments, no errors, in budget), **factual** (task completion, grounding and faithfulness) and
**quality** (tone, conciseness, judge verdict) — so you can see whether a low score means the
agent is malfunctioning or merely wordy. The grouping is display-only; it never changes the
underlying score.
When an item doesn't complete, its badge tells you why without conflating it with agent
quality: a **test configuration error** (a missing fixture, snapshot or attachment — fix the
case, not the agent) is shown differently from an **infrastructure error** or a cancellation.
## Running an evaluation [#running-an-evaluation]
1. In **Suites**, select a suite and choose **Run evaluation**.
2. The **estimate dialog** shows the exact cost before anything is charged, checked against your
remaining monthly allowance.
3. Confirm. Items run in the background and results appear under **Runs**, each showing its
overall score and flagging any critical failure without needing to be opened.
### Running one case [#running-one-case]
Select the run icon on a case row to run that case on its own. It runs once against the
current configuration, with no baseline comparison, and starts immediately — there is no
estimate dialog, so nothing shows you the cost first. It still draws on your monthly
allowance, so check the cost rule below before running an expensive case repeatedly. It
does not use one of your daily evaluation runs.
### What a run costs [#what-a-run-costs]
Evaluations draw on the same monthly allowance as chat:
* **1 interaction per simulated user turn**, plus 1 per attachment on that turn
* **plus 1 interaction per case that uses the AI judge**, per execution
* **plus 1 interaction per case with the knowledge faithfulness check ticked**, per execution
So a single-turn case with the judge costs 2 interactions. A case using only deterministic
checks costs no judge interaction. See [interaction counting](/reference/interaction-counting).
## Comparing against a baseline [#comparing-against-a-baseline]
On higher plans you can run a suite against both the current configuration and an earlier
snapshot, producing a regression matrix showing exactly what improved and what got worse. You
can also repeat each case several times to measure consistency.
## Human review [#human-review]
On any finished run item, owners and admins can record a verdict — pass, fail or needs work —
optionally overriding an evaluator's score, with notes. Human review sits alongside the
automated result and never overwrites it.
## Reviewing a run with the assistant [#reviewing-a-run-with-the-assistant]
Once a run has finished you can ask the
[in-dashboard assistant](/troubleshooting/support#in-dashboard-assistant) how it went — "how did
the last evaluation go?", "why is my agent failing these cases?" — instead of reading the results
yourself. It can find a suite's or an agent's recent runs, report the overall result and each
case's scores, and open a single case to show its per-dimension scores, the AI judge's reasoning,
and any human review recorded against it. Reading results is free: no interactions are charged.
Where a case fails, the assistant separates the two causes and acts differently on each:
* **The agent answered badly on a fair test** — it proposes a
[change set](/agents#editing-with-the-assistant) against the agent, which you approve or
discard like any other.
* **The test case itself was wrong** — an unrealistic expectation, a stale reference answer, a
fixture it never ticked — it revises the case for you. That saves a new version of the case, so
earlier runs keep the definition they were scored against, and the correction only counts from
the next run onwards.
When it can't tell which of the two is at fault, it says so rather than guessing. It cannot start
a run, so re-running the suite to confirm a fix is still yours to do.
The assistant can start from a real conversation the same way, not only a finished run — see
[ask the assistant to diagnose it](/conversations#ask-the-assistant-to-diagnose-it).
## Publish gates [#publish-gates]
A publish gate is a per-agent policy checked at the moment a change reaches production.
| Mode | Behaviour |
| ----------------- | ------------------------------------------------- |
| **Informational** | Results shown, never intervenes |
| **Warn** | Publishing succeeds, failing thresholds are shown |
| **Block** | Publishing is refused until thresholds pass |
Thresholds cover minimum overall score, maximum regressions, zero critical failures, and
requiring that the evaluated snapshot matches the configuration going live.
For an agent **without** [versioning](/agents/versioning) the gate applies at **activation**;
with versioning it applies at each **publish**.
Editing an already-active agent is never blocked — only the moment a change reaches production is
gated. Rollback is never blocked either; it is the emergency path, and the verdict is reported for
information only.
## Snapshots [#snapshots]
Every run pins a snapshot of the agent's full configuration, so results stay reproducible after
the agent changes. Knowledge **index content** is not versioned — re-embedding a source after a
snapshot can change retrieval when you rerun it. A drift badge indicates when this has happened.
## Plan limits [#plan-limits]
| Plan | Suites | Cases per suite | Runs per day | Baseline and repetitions |
| ---------- | --------- | --------------- | ------------ | ------------------------ |
| Starter | 1 | 2 | 10 | — |
| Startup | 3 | 5 | 25 | — |
| Growth | 10 | 15 | 100 | — |
| Business | 25 | 30 | 500 | ✓ |
| Enterprise | Unlimited | Unlimited | Unlimited | ✓ |
Suite limits count **active** suites across the workspace; case limits count active cases per
suite. Archiving frees a slot. One run counts as one per day, and a cancelled run gives the slot
back. Running a single case does not use a daily slot — it still draws on your monthly
interaction allowance.
## Deleting suites and cases [#deleting-suites-and-cases]
Owners and admins can delete a suite or an individual case. Deletion is permanent and takes the
run history with it: deleting a case also removes that case's results from past runs (run-level
totals are kept). If a run is currently in progress, deletion is refused until it finishes or is
cancelled. Past interaction charges are unaffected. If you only want to free a plan slot, archive
instead — archived suites and cases keep their history.
## Related pages [#related-pages]
* [Agent versioning](/agents/versioning)
* [Interaction counting](/reference/interaction-counting)
* [Guardrails](/guardrails)