Evaluations

Evaluations let you test an agent before it reaches your customers. You build a suite of test cases — scripted conversations with expectations — and run them against a frozen snapshot of the agent's configuration. Every response is scored automatically, and results show which cases passed, which failed, and why.

Evaluations are available on every plan. Plan limits apply to how many suites, cases and daily runs you get.

Where to find it

Open an agent and select the Evaluations tab.

Suites and cases

A suite groups related cases for one agent, such as "Refund policy accuracy". Each case has:

• Turns — the scripted user messages, in order.
• Objective checks — rule-based pass/fail, with no AI involved and no extra interaction cost:
  • Required facts — text that must appear
  • Forbidden claims — text that must never appear
  • Required tool calls — each entry asserts the tool was called; optionally it must also match specific arguments (partially or exactly) and be called at least (or at most) a given number of times
  • Forbidden tool calls — tools the agent must never call
  • Required call order — tools must be called in a given order, not necessarily back-to-back
  • Maximum tool calls — a total budget across the whole case, so an agent that loops or over-fetches fails even when every individual call is allowed
  • Expected knowledge sources the answer should be grounded in
  • Maximum latency and maximum response length. A case scored by the AI judge waits on an extra scoring pass, so leaving Maximum latency blank on one gives it a more forgiving default cap rather than no cap at all. Set a value to override it.
• AI judge — a reference answer and rubric the judge scores the response against. Filling either adds one interaction per case execution. A separate Check knowledge faithfulness tick-box has an AI check verify the answer only asserts what the retrieved knowledge supports; it also adds one interaction per case execution and is scored n/a when nothing was retrieved.
• Severity — low, medium, high or critical. Critical failures are highlighted and can block publishing.
• Tags — free-form grouping, such as happy-path or security. The tags already used in the suite are offered as suggestions.
• Test setup — optionally freeze the case's runtime context so time-dependent behaviour ("remind me tomorrow at 9") stays deterministic across reruns: a simulated time, entered as wall-clock time in a chosen timezone, and a simulated channel. Test setup is also where you pick which of the suite's fixtures the case runs against — see fixtures below.

Only a forbidden tool call counts as a critical safety breach. Argument, count and order misses are correctness failures — they lower the tool-correctness score and can fail the case, but they never flag it as a safety violation.

Cases are versioned: editing one creates a new version, so past runs still reference the exact definition they were scored against.

Each case row in the suite view shows a strip of dots for its last few runs, oldest to newest, colour-coded pass/fail/critical, with a relative "last run" time — so a case that's flaky or has just started failing stands out without opening it.

You can also draft a case from a real conversation. The user turns are lifted into a case and the customer's email addresses and phone numbers are replaced automatically with sample values — realistic ones, such as [email protected], so the case still runs the way the original conversation did. Nothing the customer actually wrote in those fields is kept, and no message is sent to a sample address: evaluations never deliver email, WhatsApp or SMS.

Fixtures

A fixture is a canned response for one tool, defined per suite. When the agent calls that tool during a run, the fixture's response is returned instead of executing the real tool — so a case can exercise "create the order, then confirm it" without ever writing real data.

Each fixture has:

• Name — an optional label, such as "Order not found". It is shown in place of the tool id in the suite's fixture list and in the per-case picker, which is what makes two fixtures on the same tool tellable apart. It is only a label: matching never depends on it.
• Tool — picked from a searchable catalog, grouped into write tools (blocked by default in evaluations), the tools of any connected app the agent has access to (always blocked — see below), and read tools (which already run for real). You can also type an exact tool id, which is what to do for a custom tool or when a connected app's authorisation has lapsed and its tools are temporarily missing from the list.
• Match mode — when the fixture applies:
  • Any call to the tool
  • Partial — every field in the matcher must equal the call's arguments; extra arguments are ignored
  • Exact — the call's arguments must equal the matcher exactly
• Matcher — the argument shape to match, as JSON. The chosen tool's field names appear as clickable keys that insert into the matcher, and the editor rejects field names the tool doesn't accept.
• Response — the JSON body the agent receives when the fixture matches. It is prefilled with the tool's real example payload, so shaping a realistic response starts from a working template.

By default every fixture in the suite applies to every case. A case can narrow this under Test data & tool responses in its Test setup by ticking only the fixtures it uses. Ticking a fixture only supplies the simulated result if the agent calls that tool — it doesn't require the call. To assert the call happens, add a required tool call under the case's objective checks.

When a side-effecting tool is called and no fixture matches, the case's fallback (also set in Test setup) decides what happens: the call is blocked (the default), answered with a generic safe default, or the item is marked unsupported. Read-only tools always run for real, so retrieval and data lookups behave like production.

Connected apps always need a fixture

Tools that come from a connected app — Google Calendar, DocuWare, anything reached over MCP — are the one exception to all of the above. They are always blocked during a run without a fixture: reads included, and whatever the case's fallback is set to. A real call would reach live data in somebody else's system, and the app's own description of which of its calls are safe to repeat is not something we can verify, so nothing is taken on trust.

In practice that means a case has to cover every connected-app call it will make, not just the final one. Booking an appointment usually needs a fixture for the calendar lookup that comes first as well as for the booking itself — without the first, the agent never reaches the second, and the case fails on a missing fixture rather than on the behaviour you were testing.

These fixtures have no example payload to start from, since the response shape belongs to the other system rather than to KlicForge. Copy the shape from a real response where you can — inventing field names produces a case that passes against a payload the agent will never actually see.

Reading a run's trace

Each tool call in a run's trace is badged so you can see exactly what happened: sim for a fixture response, blocked when nothing matched — and the detail distinguishes "no fixture exists for this tool" from "a fixture exists but its matcher didn't fire", so a case-authoring gap never reads as an agent failure. Expanding a call shows its arguments and result as syntax-highlighted JSON.

A blocked badge is clickable. It opens the fixture editor with the tool and the call's real arguments already filled in, so the usual fix for a blocked call is to click it, paste a realistic response, save, and re-run.

Scoring

Each case is scored by a mix of deterministic checks and an optional AI judge:

Knowledge grounding and knowledge faithfulness answer different questions. Grounding asks whether retrieval found the right sources; faithfulness reads the answer and asks whether it stuck to what those sources actually say. An agent can score 100% on grounding and still invent details, so the two are scored separately. Faithfulness is skipped (scored "n/a") when nothing was retrieved — that failure is grounding's to report.

A case that lists no evaluators runs all of them, except the two that cost an extra interaction: the AI judge runs only when you supply a rubric or reference answer, and knowledge faithfulness only when you tick it on. Scores are weighted per case and aggregated into a run-level overall score, pass rate and critical-failure count.

Results also group the dimensions into three at-a-glance tiers — execution (right tools, right arguments, no errors, in budget), factual (task completion, grounding and faithfulness) and quality (tone, conciseness, judge verdict) — so you can see whether a low score means the agent is malfunctioning or merely wordy. The grouping is display-only; it never changes the underlying score.

When an item doesn't complete, its badge tells you why without conflating it with agent quality: a test configuration error (a missing fixture, snapshot or attachment — fix the case, not the agent) is shown differently from an infrastructure error or a cancellation.

Running an evaluation

1. In Suites, select a suite and choose Run evaluation.
2. The estimate dialog shows the exact cost before anything is charged, checked against your remaining monthly allowance.
3. Confirm. Items run in the background and results appear under Runs, each showing its overall score and flagging any critical failure without needing to be opened.

Running one case

Select the run icon on a case row to run that case on its own. It runs once against the current configuration, with no baseline comparison, and starts immediately — there is no estimate dialog, so nothing shows you the cost first. It still draws on your monthly allowance, so check the cost rule below before running an expensive case repeatedly. It does not use one of your daily evaluation runs.

What a run costs

Evaluations draw on the same monthly allowance as chat:

• 1 interaction per simulated user turn, plus 1 per attachment on that turn
• plus 1 interaction per case that uses the AI judge, per execution
• plus 1 interaction per case with the knowledge faithfulness check ticked, per execution

So a single-turn case with the judge costs 2 interactions. A case using only deterministic checks costs no judge interaction. See interaction counting.

Comparing against a baseline

On higher plans you can run a suite against both the current configuration and an earlier snapshot, producing a regression matrix showing exactly what improved and what got worse. You can also repeat each case several times to measure consistency.

Human review

On any finished run item, owners and admins can record a verdict — pass, fail or needs work — optionally overriding an evaluator's score, with notes. Human review sits alongside the automated result and never overwrites it.

Reviewing a run with the assistant

Once a run has finished you can ask the in-dashboard assistant how it went — "how did the last evaluation go?", "why is my agent failing these cases?" — instead of reading the results yourself. It can find a suite's or an agent's recent runs, report the overall result and each case's scores, and open a single case to show its per-dimension scores, the AI judge's reasoning, and any human review recorded against it. Reading results is free: no interactions are charged.

Where a case fails, the assistant separates the two causes and acts differently on each:

• The agent answered badly on a fair test — it proposes a change set against the agent, which you approve or discard like any other.
• The test case itself was wrong — an unrealistic expectation, a stale reference answer, a fixture it never ticked — it revises the case for you. That saves a new version of the case, so earlier runs keep the definition they were scored against, and the correction only counts from the next run onwards.

When it can't tell which of the two is at fault, it says so rather than guessing. It cannot start a run, so re-running the suite to confirm a fix is still yours to do.

The assistant can start from a real conversation the same way, not only a finished run — see ask the assistant to diagnose it.

Publish gates

A publish gate is a per-agent policy checked at the moment a change reaches production.

Thresholds cover minimum overall score, maximum regressions, zero critical failures, and requiring that the evaluated snapshot matches the configuration going live.

For an agent without versioning the gate applies at activation; with versioning it applies at each publish.

Snapshots

Every run pins a snapshot of the agent's full configuration, so results stay reproducible after the agent changes. Knowledge index content is not versioned — re-embedding a source after a snapshot can change retrieval when you rerun it. A drift badge indicates when this has happened.

Plan limits

Suite limits count active suites across the workspace; case limits count active cases per suite. Archiving frees a slot. One run counts as one per day, and a cancelled run gives the slot back. Running a single case does not use a daily slot — it still draws on your monthly interaction allowance.

Deleting suites and cases

Owners and admins can delete a suite or an individual case. Deletion is permanent and takes the run history with it: deleting a case also removes that case's results from past runs (run-level totals are kept). If a run is currently in progress, deletion is refused until it finishes or is cancelled. Past interaction charges are unaffected. If you only want to free a plan slot, archive instead — archived suites and cases keep their history.

Related pages

• Agent versioning
• Interaction counting
• Guardrails