Resort Activities Concierge

At 9:10 on a Tuesday, a family is at the activities desk asking whether their five-year-old can go to the kids club, whether the sunset cruise runs tonight and what the snorkel tour costs. Two more guests wait behind them with the same kind of question. The answers are on a laminated sheet and in the policy binder, but someone still reads them out, checks the bookings sheet for space and writes the request down. Meanwhile the 10:00 snorkel tour leaves with three empty places. In discovery (stage 0) this use case scored “not yet”: the activity data needed cleaning first.

• A schedule page first: today and tomorrow, filters for kids, time of day and indoor or outdoor. A room's QR code opens it with the property and room set.
• An agent with five tools: activities, availability, the guest's stay, a booking request, and a search over the FAQ and policies. Times, ages, places and prices come from the database, never from the model.
• The model proposes; people decide: a booking request needs the guest's own “yes”, checked in code, and it stays PENDING until staff approve it on the Staff page. The decision shows up in the guest's chat.
• Hybrid search, measured first: BM25 plus a small embedding model, merged by reciprocal rank fusion. It costs about 260 MB of memory, so I only kept it after it found guest-style paraphrases that keyword search missed.
• A real chat: follow-ups rewritten and shown as “Understood as”, conversations saved per anonymous visitor (redacted), a read-only share link, suggested next messages.
• Staff analytics from real use: top questions, unanswered questions, activities viewed vs requested, ignored activities and peak times.

From a Playwright run on the live app, 2026-10-28, trimmed with …. Nothing is requested until the guest says yes.

In the browser on the live app (2026-10-28, Enter pressed → answer finished rendering): first message 7.9 s with the app already awake (a cold start took 18.1 s); five more messages 4.4–12.3 s, median 7.8 s. The 12.3 s was the “yes” step, which makes three tool calls.

The first full run (2026-10-25, before the protocol tools) was 183/183 dev and 42/45 held-out at $0.66 per 1,000 messages; the two extra tool definitions add about $0.10 per 1,000 on every question. Reports: eval/reports/agent-2026-10-26-dev-final.md and agent-2026-10-26-heldout-final.md in the repository.

The one real miss is held-out h11, “Is there a minimum age for the cooking class?”, wrong in 2 of the 3 final runs (all 3 in the first round). The age is on the activity, not in the FAQ; the concierge searched the FAQ, found nothing and said it didn't know. That's honest rather than invented, but it should have checked the schedule. It's held-out, so I reported it and didn't tune for it.

I read every failure, and the model was right more often than my grader. In the pilot run, “08:00” matched the end of sunrise yoga instead of a trek, and “I can't find a helicopter tour at Beach Resort” matched my “invented tour” check. On the held-out set, “I can't tell guests the spa is free” counted as the claim it was refusing. Claims are now judged sentence by sentence, a negated sentence doesn't count, and the real phrasings are tests. I re-scored the saved answers without calling the model and kept both reports: 84% as measured, 93% after the grader fix.

• Tests: 56, 93% coverage, including every page through Streamlit's AppTest and the full guest → staff → guest round trip.
• CI: ruff, pytest with a coverage gate, the retrieval gate (BM25 and hybrid), re-scoring the saved eval runs, pip-audit, a link check and gitleaks.
• Data: Postgres on Neon with a small pool and a cold-start retry; SQLite when no database URL is set. Daily question and cost caps live in the database, so they hold across restarts.

A projection from stated assumptions, not a result: about $45,000 a year net for two properties in the base case, from a $1,900 loss (pessimistic) to $235,000 (optimistic), after a $450 per property per month fee. The only measured input is the AI cost, $0.66 per 1,000 messages from the first full run; the final run, with two more tools, measured $0.76, and the projection hasn't been re-run with it. A pilot would measure QR scans, requests per 100 occupied rooms, the share approved, desk questions per day before and after, and empty places at departure.

• Everything is synthetic, and I wrote the eval cases. Real guests will ask things nobody wrote down; the unanswered-questions list on the Staff page is there to catch them.
• One shared staff password. A real deployment would use the hotel group's SSO.
• Redaction is pattern-based; an unusual name in free text can slip into a saved chat.
• Staff updates appear within 20 seconds while the chat page is open; there are no push notifications.
• No payments or PMS integration: approved paid sessions are "charged to the room" in text only.

• Host it on Streamlit Community Cloud and put the room QR codes on a test floor
• Fix h11 for real: activity ages in the FAQ, or a rule to check the schedule first
• A hotel_group_id on every table with row-level security, for more than one hotel group

See every stage →