← All notes

Evals

My eval said 96%. It was wrong.

How a generous metric hid retrieval misses, and why I published the lower number.

October 9, 2026 · 3 min read · Evals

The number I almost shipped

The first thing I measured on my hotel assistant was retrieval. Before Claude writes an answer, search has to hand it the right material, so for each labeled question I asked: is the right document in the top five results?

Keyword search had a rough start. After I renamed the loyalty program to “Guest Rewards”, the score dropped from 93% to 86%, because questions about guests started matching the loyalty documents on the word “guest”. A small stemmer fixed the word forms, so “cats” matched “cat” and “smoking” matched “smoke”.

The score went to 96%. Ten points from one change. It looked like a number I could put in a README.

What it was actually measuring

One of the test questions is in Spanish: “¿Se permiten perros en el hotel?” Keyword search shares no words with the English documents, so every passage scored zero.

It passed anyway. With every score tied at zero, the results came back in file order, and the right document happened to be first.

That's when I looked at what the check actually asked. It counted a hit if the right document appeared anywhere in the top five. But a policy document has many sections: pets, the pool, parking, airport transfers. Staff don't need the right file. They need the passage that holds the answer.

Making it strict

Each test question now names a distinctive phrase that must appear in a retrieved passage from the expected document. For the dog question at the beach resort, that phrase is “up to 40 lbs”:

{"id": "q01", "property": "beach-resort", "question": "Can a guest bring their dog?", ..., "expected_source": "beach-resort/policies.md", "expected_text": "up to 40 lbs"}

Multi-part questions list one phrase per part, and every part has to be found. The scoring function is three lines:

def retrieval_hit(item: dict, results: list) -> bool:
    texts = item["expected_text"] if isinstance(item["expected_text"], list) else [item["expected_text"]]
    return all(any(r["source"] == item["expected_source"] and t in r["text"] for r in results) for t in texts)

A passage only counts if it comes from the right document and contains the answer. Luck with file order no longer helps.

What dropped, and what it revealed

Two points lower, and every one of them real. The two misses had the same shape: the question shares no words with the passage that answers it.

One was the Spanish question. The other was “A guest left a jacket”. The answer is in the Lost and Found procedure, and that procedure never uses the word “jacket”. Keyword search matched nothing at all.

Those are not edge cases at a front desk. Staff ask in their own words, and sometimes in their own language.

What I fixed next

First, search stopped returning passages that match nothing. A question with no matching words now gets a refusal without calling the model, instead of five arbitrary passages. That turned a lucky pass into an honest refusal.

Then I measured hybrid search, which adds meaning-based embeddings to the keyword ranking. The embeddings ranked the Lost and Found passage first for the jacket question, and strict retrieval rose to 97% (33 of 34). The Spanish question still failed, because the embedding model is English-only.

The test set later grew to 54 questions in 16 categories, including prompt injection, cross-property leaks and wrong facts in the question. When I tested it the way a first-time visitor types, I found that “checkout” never matched “Check-out”, so I gave compound words one spelling in both questions and documents.

Today strict retrieval is 98% (44 of 45) in both search modes. The one miss is still the Spanish question, and it's documented as a known limitation rather than hidden.

What I'd tell a client

A flattering number moves the risk to the rollout. If the pilot starts at “96%” and the staff find the misses on day three, the next number nobody believes is the right one.

A stricter, lower number does the opposite. It tells you which questions fail and why, and what to fix next, before a guest is standing at the desk. It also makes every later improvement mean something, because it's measured against the same strict check.

Three takeaways

• Measure what the user needs. Here that's the passage with the answer, not the file name.
• Test that your metric can fail. A check that passes a question with zero matches is measuring luck.
• Read the raw results, not just the score. The same habit later caught my answer grader passing a wrong “yes”.

Everything here is in the project's repository: the scoring code, the labeled questions, the saved eval reports and the decision log.

The repository →
The case study →

Have a question about this? Email me or find me on LinkedIn.

← All notes