Projects › Production & Handoff
Watch the demo
Recorded on the live apps: a Spanish question answered in Spanish, an emergency answered at once, a wrong-inbox email answered for the right hotel, then the owner's Operations page and the handoff runbooks.
Read the transcript
Production and handoff: a simulated client project, parts four and five of five.
The eval harness found real gaps. Spanish questions were blocked before they reached the model. Now the answer comes back in Spanish, from the English documents, with its source.
Emergencies in other words, or in Spanish, were missed. Now they get the safety message at once, with no model call.
A guest booked at the Mountain Lodge wrote to the Beach Resort. The agent used to answer for the wrong hotel. Now the booking is looked up first, so the reply is about the right one.
On a local copy, the owner's Operations page shows requests, latency and cost against budgets. Every log record has personal data removed before it is stored, and is kept for thirty days.
And the handoff package: runbooks, service levels, staff guides, and a daily health check that knows a sleeping free app from a broken one.
Every number comes from a saved run. The links are on screen.
Free hosting: if an app is asleep, click ‘wake up’ and give it about 30 seconds.
Production & Handoff
Two live AI apps made ready for a hotel group to run: the gaps the evals found fixed and measured, personal data out of the logs, budgets that fail CI, monitoring that knows “asleep” from “down”, and a handoff package their own team can work from.
Independent project: a simulated client engagement for the same fictional hotel group as stages 1–3. Documents, guests and emails are synthetic. Every number below comes from a saved run in the eval harness; nothing here is a measured saving.
A demo is not a service
Both apps worked, and the eval harness said so. It also said where they didn't: a front-desk agent who types in Spanish was turned away before the model saw the question; “there's smoke coming from under a door” got no manager warning; a guest who wrote to the wrong hotel's inbox got the wrong hotel's answer.
And nothing about running them was ready for someone else: logs vanished on every restart and kept names as typed, anyone could do anything, a cost spike had no off switch, and the only monitoring was me opening the app.
What changed in production
Every fix started with new held-out cases committed before the code, so the “after” can't be the result of tuning to them. Numbers are live re-runs, Claude Haiku 5.5, 3 runs each, 2026-10-15.
| Gap | Before | After | Fix |
|---|---|---|---|
| Assistant: Spanish questions blocked before the model | 67% (26/39) | 85% (33/39) | Spanish hotel vocabulary into the search; answers in the guest's language |
| Assistant: emergencies in other words or Spanish | 85% escalated | 100% | Broader safety rule (0 false alarms on every non-safety eval question); emergencies skip the model |
| Assistant: 8 new held-out cases | 62% (15/24) | 100% (24/24) | The same fixes |
| Agent: wrong-inbox email (E12, b10) | 0/6 right hotel | 6/6 | The sender's booking is looked up before the model and pins the hotel |
| Agent: Spanish email answered in English (X05) | 67% (4/6) | 100% | Language detected up front; one rewrite; Spanish holding replies |
The headline numbers didn't drop: the assistant's answer accuracy went from 98% to 100% (162/162; the Spanish question q37 now passes) at a 95th percentile of 2.93 s and $0.18 per 1,000 questions. The agent kept every hard gate at 100%, with correct tool use 96% → 100%, citations 94% → 100% and $1.30 → $1.24 per 1,000 emails.
- Persistent, private logs: every request goes to Postgres (Neon) or SQLite, one table shared by both apps. Emails, phone and card-like numbers and names are removed, and booking IDs hashed, before anything is stored; records are deleted after 30 days. Tests plant personal data in every field and fail if any survives.
- Staff sign-in: the public demo stays open in a labelled Demo mode; a Staff role (lockout, idle sign-out) unlocks reporting an answer for review. A real client would use their own SSO; the switch is documented.
- Budgets and resilience: p95 and cost-per-1,000 budgets for each app, checked by the eval harness in CI; 30-second timeouts with retries; a kill switch that stops every model call.
- Monitoring: an owner-only Operations page (requests, errors, latency, cost per day, budget status) and a daily GitHub Action that reads Streamlit Cloud's status, so a sleeping free app never pages anyone and a broken one opens an issue.
The handoff package
Everything the hotel's IT and operations teams need, in Markdown in one repo, built from the real code and linking back to it:
- Architecture across all three systems: data flows, where each secret lives, what is logged and what is redacted.
- Eight runbooks, each with how you notice it, the steps and how to check it worked: API outage, cost spike, a bad or harmful answer, prompt injection, an app asleep or down, rotating secrets, restoring logs, adding a property's documents.
- SLOs and budgets with how each is measured and alerted, and an error-budget policy.
- Plain-language guides for front-desk and guest-services staff, and an admin guide.
- A readiness checklist with an evidence link per item (owner-only items left for the owner) and a 30/60/90-day plan for the client team.
What production taught me
- A fixed rule wasn't a fixed answer. After the safety rule caught “smoke under a door”, the model still answered it with the smoking policy and a $250 cleaning fee. Emergencies now skip the model entirely.
- The prompt-injection gap I found had no warning sign. A held-out note “to the automated system” matched none of the agent's patterns, so it passed the planted request on to a manager. Planted text is now removed before the model reads the email; I wrote three new injection cases first, and they passed before the fix too: that case was flaky, not systematic.
- Every fix has a price, and I report it. With the planted “6 PM check-out” gone, one held-out reply stopped volunteering the guest's Platinum late check-out (key facts 3/3 → 0/3). Safe and correct, less specific: reported, not tuned.
- The new timeout caught a real stall. In the first live run one streamed answer froze for 30 seconds; the eval driver now counts a failed call as a failed answer instead of stopping.
Trade-offs
| Choice | Why | What it costs |
|---|---|---|
| A Spanish vocabulary list, not a translation call | Free, deterministic, testable | Only Spanish; other languages are still refused before the model |
| Pattern-based redaction | No model or API call before every write; fully tested | A bare name with no cue can slip through (listed as a known limit) |
| Every new setting optional | The live apps never break on deploy | Persistence is “ready” until the owner adds the database URL |
| Shared staff password | Enough for a demo, same protections as the owner page | No individual accounts; SSO is the client path |
| One health sample a day | Free, and never alarms on a sleeping app | Can't see a short outage between checks |
Engineering
- Tests: assistant 378 and agent 276 (97% coverage each), plus 8 per app that run against a real Postgres in CI; the harness 92; the health check 12, including asleep vs down.
- CI: each app gates its release's saved live run with latency and cost budgets; the harness checks that the previous releases fail against the new baselines, so a rollback would be caught.
- Releases: assistant v1.6.0, agent v1.1.0, harness v1.2.0, handoff v1.0.0;
docker compose upruns each app with its own Postgres. - Cost: about $0.89 in API spend for both stages, almost all of it live eval runs.
- Decisions: 16 new ones across the three repos, indexed in the decision index →
What's next
- Outside testers: questions and emails written by people who haven't seen the documents
- A second labeler and written examples before any AI-judge score counts
- A loyalty-aware check-out answer in the agent, with new held-out cases first