Projects › Eval Harness
Watch the demo
Recorded on the live dashboard: both assistants' gates, a regression caught, a failing case, and the judge checked against hand labels.
Read the transcript
Eval Harness: a simulated client project, part three of five.
One harness now checks both of the hotel group's assistants the same way: the same cases, metrics and gates, with ninety-five percent intervals on every rate.
It reproduces both projects' published numbers from their own recordings, and fresh re-runs at the pinned versions pass every gate.
Then I made a realistic bad change on a throwaway branch: two passages instead of five, to save tokens. Retrieval only fell from ninety-eight to ninety-three percent, inside the interval.
The gate still failed, because two questions that always passed now always fail. The message names them. The branch was never merged.
Here is one: a spa question about a sixteen-year-old. The right passage dropped out, and every metric shows it.
An AI judge rates tone and clarity. I labelled forty replies by hand. It agreed ninety-five percent of the time, but kappa was zero: my labels had no failures, so agreement proved nothing. No judge score is trusted, and none is a gate.
The code, the reports and every decision are linked on screen.
Free hosting: if the app is asleep, click ‘wake up’ and give it about 30 seconds.
Eval Harness
One evaluation system for both of the hotel group's AI assistants: the same cases, metrics, statistics and regression gates, so a prompt, model or retrieval change can't lower quality without a red check.
Independent project: a simulated client engagement for the same fictional hotel group as stages 1 and 2. Documents, guests and emails are synthetic. Every number below comes from a saved run in the repo; the business figures are projections from stated assumptions.
Why evals rot
Both assistants were measured carefully when they shipped. Then the changes start: a cheaper model, a shorter prompt, “send 2 passages instead of 5 to save tokens”. Each change is reasonable, and each project had its own eval scripts, its own report format and nothing in CI that would fail.
One of those changes quietly stops the staff assistant from finding the spa's age rule. Nobody notices until a front-desk agent books a 16-year-old for a facial.
The approach
- One case bank: 181 labelled cases. 161 imported from both projects without changing what they test; 20 new held-out cases where coverage was thin (cross-property leaks, Spanish, a superseded document, injections, safety wording).
- Recorded first: saved outputs are re-scored with no API call, so CI is free and the same every time. Live mode runs a project's real pipeline at a pinned commit.
- The projects' own metrics: copied byte for byte at the pinned commits, behind one registry, so the harness reproduces both projects' published numbers from the same recordings.
- Honest statistics: a 95% interval on every rate, counted over distinct cases; repeated runs to find flaky cases.
- Gates that name names: hard 100% gates for escalation, prompt injection and forbidden promises, floors, the baseline interval, and any case that always passed and now always fails.
- A judge only where code can't judge: completeness, tone and clarity, trusted only if it agrees with my hand labels beyond chance.
Architecture
- CasesOne schema for both projects; dev, held-out and visitor splits.
- RunnersRecorded outputs by default; or the real pipeline at a pinned commit, exported so the checkout is never touched.
- Metrics + judgeThe projects' own deterministic checks; an LLM judge only for completeness, tone and clarity.
- StatisticsWilson intervals, run-to-run consistency, flaky cases, categories.
- GatesCandidate vs baseline: hard 100%, floors, the interval, stable flips.
- Dashboard and CIReports in the repo, a read-only dashboard, and one CI job in each project's repo.
Results
From the projects' own recordings, the harness reproduces their published numbers exactly: stage 1 at 98% answer accuracy and 27 of 27 correct refusals, stage 2 at 100% on escalation, injection and forbidden promises. Then it re-ran both live at their pinned versions, 3 runs each, on 2026-10-14.
| Measure | Stage 1, 54 questions | Stage 2, 40 emails |
|---|---|---|
| Gate against the saved baseline | Pass | Pass |
| Hard gates: escalation, injection, no promises | 100% | 100% |
| Answer accuracy / requests found | 98% (90–100%) | 100% (91–100%) |
| Strict retrieval / facts grounded | 98% (88–100%) | 100% (91–100%) |
| Expected source cited | 98% (88–100%) | 94% (72–99%) |
| Same verdict in all 3 runs | 98% | 90% |
| Median / 95th percentile | 1.6 s / 3.3 s | 4.3 s / 8.9 s |
| Cost per 1,000 | $0.17 | $1.30 |
The 20 new held-out cases found real gaps, reported rather than tuned away. Stage 1 answered 67% of its 13 correctly (interval 40–86%): Spanish questions are blocked before they reach the model, injected text pushed the resort-fee passage out of the search results, and the safety rule misses “smoke coming from under a door” and “incendio”. Stage 2's 7 new emails kept escalation and injection at 100%.
The judge calibration story
An LLM judge (Claude Haiku 5.5, versioned rubric, structured output) rates completeness, tone and clarity. Before any of its numbers count, I labelled 40 stratified replies by hand, pass or fail, without seeing its verdicts.
I passed all 40. The judge agreed on 38: 95% agreement, which sounds good. Cohen's kappa was 0.00. With no failures in my labels, agreement can't be told apart from a judge that always says “pass”. So no judge metric is trusted, and none is a gate.
- One rubric revision fixed the item it targeted (stage 1's fixed “couldn't find this, check with a manager” sentence is clear, not vague). Agreement stayed 95%, kappa 0.00. A second revision could only have fitted these 40 labels.
- Judging the same 40 prompts three times, the judge changed its verdict on one item: a disagreement that comes and goes on identical input is noise.
- On the full re-runs the judge fails 13–17% of replies on clarity, stricter than my labels. Which of us is right needs labels that contain real failures.
The regression it caught
On a throwaway branch I made a realistic change to stage 1: send 2 retrieved passages instead of 5, to cut input tokens. Strict retrieval fell from 98% to 93%, which is still inside the baseline's interval (88–100%). An interval-only gate would have passed it.
The harness failed it anyway: two questions passed in every baseline run and failed in every run after the change, q13 (“Can a 16 year old book a facial?”) and q35. In recorded mode (saved answers, retrieval recomputed, no API) 1 gate failed; in live mode 4 did, each naming the cases. The branch was deleted, never pushed and never merged.
What the eval taught me
- Agreement isn't evidence. 95% agreement with kappa 0.00 meant my sample had nothing to disagree about. Sampling replies a deterministic check had flagged gave me wrong facts, not badly written replies.
- An interval alone misses small, real regressions. Two lost questions out of 45 stay inside it; “always passed, now always fails” caught them.
- Re-running unchanged code failed the gate. One model miss in a 3-case group broke a 90% floor. Floors now need 10 or more cases.
- The graders were wrong four times: a cited policy description read as a promise, two correct injection rejections, and a Spanish “50 $”. Each fix is a test with the exact sentence.
Trade-offs
| Choice | Why | What it costs |
|---|---|---|
| Copy each project's metric code at a pinned commit | One definition per metric; published numbers reproduce exactly | A project fix reaches the harness only when the pin moves |
| Recorded mode by default | Free, deterministic CI | A model change needs a live run to be measured |
| Intervals over distinct cases | Three runs of one question aren't three samples | Wide intervals on small groups, shown rather than hidden |
| Stable flips fail a build | Catches the small, real regression an interval misses | One real flip on a model metric can be noise; the message names it for a person |
| The judge never gates until trusted | An unverified judge would gate on its own opinion | No automated tone or clarity score yet |
Engineering
- Tests: 86 tests at 93% coverage, including every metric passing and failing, kappa and interval math, gates, the judge with a stand-in model, and every dashboard page.
- CI: lint and format, a dependency audit, a 90% coverage gate, recorded-mode gates and a re-score of every saved report. Stages 1 and 2 each gained one CI job that installs the harness at v1.0.0; their own tests and numbers are unchanged.
- Cost: the whole project's API spend was $0.38: two live re-runs, the demo and 318 judge calls.
- Decisions: 19 recorded, each with context, trade-off and result. Decision log →
Business case (a projection)
Projected from stated assumptions and shown as a range; nothing here is a measured saving. The base case assumes 4 changes a month per assistant, 3 hours to evaluate one by hand against 30 minutes with the harness, $90 an hour, and one shipped regression a quarter. Only the API cost of a full re-run ($0.29) is measured.
| Scenario | Net value |
|---|---|
| Pessimistic | $1,493 |
| Base | $26,273 |
| Optimistic | $67,613 |
What's next
- Calibration labels that contain real failures, and a second labeler
- Held-out questions and emails written by front-desk staff
- Stage 4, Production Deployment: run history in a database, monitoring, sign-in and personal-data redaction