Skip to content
Back to search

booking-pace-forecast

Booking-pace guest forecast + staff planner with a Claude-powered assistant, MCP server and evals that check every number it quotes. Beats a weekday baseline by 53–68% at 7 days out.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/rebekaolar-cyber--booking-pace-forecast/badge.svg)](/listings/rebekaolar-cyber--booking-pace-forecast)

HTML

<a href="/listings/rebekaolar-cyber--booking-pace-forecast"><img src="/api/listings/rebekaolar-cyber--booking-pace-forecast/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 374 lines
  1. # Booking pace forecast
  2. [![tests](https://github.com/rebekaolar-cyber/booking-pace-forecast/actions/workflows/tests.yml/badge.svg)](https://github.com/rebekaolar-cyber/booking-pace-forecast/actions/workflows/tests.yml)
  3. [![eval](https://github.com/rebekaolar-cyber/booking-pace-forecast/actions/workflows/eval.yml/badge.svg)](https://github.com/rebekaolar-cyber/booking-pace-forecast/actions/workflows/eval.yml)
  4. **Upload a reservations export and get the next two weeks: guests per day, a likely range, the
  5. unusual days and how many staff to schedule. Then ask an assistant about it, in English or Dutch.**
  6. - **A forecast that reads the reservation book.** On 119,390 real hotel bookings, its error 7 days
  7. ahead is 53-68% smaller than the usual weekday average ([`reports/results.md`](reports/results.md),
  8. bookings: [`data/README.md`](data/README.md)).
  9. - **An AI assistant that only quotes numbers from the forecast tools.** Claude tool use with five
  10. read-only tools, in a plain loop with no framework ([`agent/loop.py`](agent/loop.py)).
  11. - **An eval that checks it.** 50 questions, a scorer in plain code that traces every number back to
  12. a tool result, and a hand audit of every failure ([report](evals/reports/2026-11-16-live-eval.md)).
  13. - **An MCP server.** The same tools work in Claude Desktop or any other MCP client
  14. ([`docs/mcp.md`](docs/mcp.md)).
  15. **Live demo:** _deploying — link coming_
  16. ![Planning the next two weeks](docs/demo.png)
  17. ## The AI assistant
  18. The app's third tab is a chat assistant for the manager: "How many bar staff do I need on
  19. Saturday?", "Any unusual days in the next two weeks?", "Wordt het druk aanstaande vrijdag?". It
  20. answers from the sample city hotel's forecast, where "today" is Thu 1 Jun 2017. It never sees an
  21. uploaded file.
  22. ![Architecture: the chat tab, the eval runner and MCP clients all reach the forecast through the same five read-only tools](docs/architecture.png)
  23. **How a question flows**
  24. 1. The question goes to the Claude Messages API with the system prompt
  25. ([`prompts/v2.md`](prompts/v2.md)), the five tool definitions
  26. ([`agent/contract.py`](agent/contract.py)) and the last 4 questions and answers, so "and
  27. Sunday?" works.
  28. 2. When the model asks for tools, [`agent/loop.py`](agent/loop.py) runs them through `run_tool`
  29. and sends the JSON results back. This repeats until the model answers.
  30. 3. The tools are read-only: `get_today_and_data_range`, `get_forecast`, `get_staffing`,
  31. `get_flagged_days` and `get_accuracy`. A bad request (a past date, an unknown role, more than
  32. 28 days) comes back as `{"error": "<plain sentence>"}` that the model can explain.
  33. 4. Every answer shows a "How I got this" box with the tools called, the time, the tokens and the
  34. cost. Every run is logged to `logs/runs.jsonl` ([`agent/run_log.py`](agent/run_log.py)).
  35. **Guardrails**
  36. - **Numbers only from tools.** The prompt allows only numbers from a tool result in the same
  37. conversation, and no totals or averages the model works out itself.
  38. - **Ranges, not single numbers.** A forecast is always given with its likely range (low-high).
  39. Percentages are quoted exactly, to one decimal, so a day never sounds flagged when it isn't.
  40. - **The question's language.** An English question gets an English answer, a Dutch one a Dutch
  41. answer.
  42. - **Out of scope gets a short no.** Pricing, other businesses, past dates or dates beyond the
  43. forecast: it says what it can answer instead.
  44. - **Caps.** At most 6 model calls and 12 tool calls per question, and 700 output tokens per call
  45. ([`agent/loop.py`](agent/loop.py)). The chat tab allows 10 questions per session and 200 per day
  46. across all sessions ([`app_pages/assistant.py`](app_pages/assistant.py)).
  47. **Three real answers**
  48. These come from the first full live run (Haiku, prompt v1,
  49. [`2026-11-16_haiku_v1.txt`](evals/reports/logs/2026-11-16_haiku_v1.txt)). The scorer of that day
  50. failed them on false alarms. The hand audit found them correct, and the current scorer passes them
  51. ([`tests/test_live_eval_v1.py`](tests/test_live_eval_v1.py) replays them). The CI logs print only
  52. failed answers, so answers that passed the first time are not stored in the repo.
  53. *English.* "How many bar staff do I need on Saturday 3 June 2017?"
  54. > Sat 3 Jun: you need **3 bar staff**.
  55. >
  56. > - Forecast: 138 guests (likely range 129–147)
  57. > - Rule: 1 bar staff per 60 guests, rounded up
  58. >
  59. > Based on: staffing rules, forecast.
  60. Tools: `get_today_and_data_range`, `get_staffing {"date": "2017-06-03", "role": "bar"}`,
  61. `get_forecast {"start_date": "2017-06-03", "end_date": "2017-06-03"}`
  62. *Dutch, and a refusal.* "Hoeveel gasten verwachten we op 15 juli 2017?" (How many guests do we
  63. expect on 15 July 2017?)
  64. > Ik kan voor **za 15 jul 2017** geen gastenaantal geven. Mijn voorspelling reikt tot en met **wo 28 jun 2017**, want vandaag is do 1 jun 2017.
  65. >
  66. > Ik kan wel de verwachting geven voor elke dag tot en met 28 jun, inclusief de bandbreedte. Wil je dat?
  67. >
  68. > Gebaseerd op: datumbereik van de data.
  69. Tools: `get_today_and_data_range`. 15 July is past the 28-day horizon, so it declines and offers
  70. any day up to 28 June instead.
  71. *"How sure are you?"* "How sure are you about the forecast for Saturday 17 June 2017?"
  72. > **Sat 17 Jun: forecast 157 guests, likely range 141-173.**
  73. >
  74. > - 154 guests are already booked.
  75. > - The usual for a Saturday is 136, so this day is flagged busy (+15.9%).
  76. > - Confidence is moderate. The day is about two weeks out. In backtests at 14 days ahead, the average error was 10.6 guests, and the 80% range was ±16 guests. Weekday averages missed by about 26.5 guests, so the forecast is much better than a typical-Saturday guess.
  77. > - Caution: over the 14 days after the export, only 64% of actual results landed inside the range, below the 80% target. Treat the range as a bit optimistic.
  78. >
  79. > Based on: forecast, accuracy check.
  80. Tools: `get_today_and_data_range`, `get_forecast {"start_date": "2017-06-17", "end_date": "2017-06-17"}`,
  81. `get_accuracy`
  82. ## How well it works
  83. Live runs on 16 Nov 2026: 50 questions each, real forecast, scored by the current scorer. Every
  84. failed answer was then checked by hand. An earlier v1 run, scored before 16 scorer fixes, is left
  85. out because it is not comparable. All numbers below are copied from
  86. [`evals/reports/2026-11-16-live-eval.md`](evals/reports/2026-11-16-live-eval.md); the full CI output
  87. is in [`evals/reports/logs/`](evals/reports/logs/).
  88. | | Haiku, prompt v1 | Haiku, prompt v2 | Sonnet, prompt v2 |
  89. |---|---|---|---|
  90. | Passed all checks | 38/50 (76%) | 43/50 (86%) | 39/50 (78%) |
  91. | Every number traced to a tool | 40/50 (80%) | 44/50 (88%) | 42/50 (84%) |
  92. | Right tools called | 49/50 (98%) | 50/50 (100%) | 48/50 (96%) |
  93. | Refused when it should | 7/8 (88%) | 8/8 (100%) | 6/8 (75%) |
  94. | Answered in the question's language | 49/50 (98%) | 49/50 (98%) | 50/50 (100%) |
  95. | Answers with a real mistake (by hand) | 4/50, plus 2 unclear | 4/50 | 2/50, plus 2 unclear |
  96. | Answers with a wrong or invented number (by hand) | 0/50, plus 1 unclear | 1/50 | 2/50 |
  97. | Latency p95 | 5.546 s | 5.964 s | 6.213 s |
  98. | Cost per 100 questions | $0.0871 | $0.0888 | $1.5430 |
  99. Models: `claude-haiku-5-5` and `claude-sonnet-5-5`. Cost is worked out from the token counts and the
  100. list prices in [`agent/run_log.py`](agent/run_log.py).
  101. **What I found.** The model almost never invents numbers: by hand, 0 to 2 answers in 50 had a wrong
  102. number, and each of those was a wrong count of flagged days, with every per-day number right. The
  103. automated pass rate rose from v1 to v2 (38 to 43), but the hand-checked real mistakes did not
  104. change (4 and 4); most of the gap was scorer false alarms. Haiku did about as well as Sonnet at about
  105. 1/17 of the cost: 4 against 2 real mistakes and 1 against 2 wrong numbers is within run-to-run
  106. variation, so these runs do not show either model to be more accurate. The real mistakes
  107. were counting flagged days wrong, English questions answered partly or fully in Dutch (Haiku
  108. only), staffing numbers worked out by the model itself, and garbled words ("минимaal",
  109. "nanaast"). The caveat: 50 questions and one run per setup, so a difference of a few answers is
  110. within run-to-run variation, and only failed answers were checked by hand.
  111. **What the scorer checks** ([`evals/scoring.py`](evals/scoring.py), plain code, no second model)
  112. - **Every number is traced.** Each number in the answer must match a tool output for the day the
  113. sentence is about, a value that holds for every day (such as accuracy), or a number from the
  114. question.
  115. - **The right tools, with the right arguments.** Compared with the expected calls for each question.
  116. - **Refusals.** Out-of-scope questions must get a clear decline.
  117. - **Language.** The answer must be in the question's language.
  118. - **The key numbers are there.** The answer must contain the numbers the question is about,
  119. recomputed from the same tools at eval time.
  120. **Known limits.** The scorer is a text check, not a judge, so it misses some mistakes and flags
  121. some correct answers:
  122. - A sentence that names no day is checked against every day of the run, so a number from the wrong
  123. day can pass there.
  124. - Counts of days are only checked in the form "N busy/flagged days". "Two are busy" (there were 3)
  125. passed.
  126. - Short date lists ("Wed 7, Sat 10 and Fri 16 Jun"), the 80% target wording and the 15% threshold
  127. wording still cause false alarms.
  128. - Dutch fragments in an English answer, wrong weekday labels and garbled words are not caught.
  129. - Numbers repeated from the question always pass, even with another meaning.
  130. **Run it again**
  131. ```bash
  132. python -m evals.run --backend real --prompt v2 # needs ANTHROPIC_API_KEY
  133. python -m evals.run # demo forecast data, still calls the model
  134. python -m evals.compare evals/results/A.json evals/results/B.json # two runs side by side
  135. ```
  136. In GitHub Actions, [`eval.yml`](.github/workflows/eval.yml) runs the live eval on every push to
  137. `main` when the `ANTHROPIC_API_KEY` secret is set. Run it by hand from the Actions tab
  138. (workflow_dispatch) with three inputs: `sample` (number of questions, default 10), `model`
  139. (default `claude-haiku-5-5`) and `prompt` (default `v2`). It fails the build when a run ends in an
  140. error or fewer than 80% of answers have every number traced. Flags above that line show up as
  141. warnings on the run, to check by hand, because the scorer still has the false alarms listed above.
  142. ## The forecast
  143. ### Use it on your own reservations
  144. 1. **Export your reservations** from your booking system (TheFork, Zenchef, Resengo, OpenTable, a
  145. hotel PMS) as CSV or Excel: one row per reservation, the last 3 months or more plus everything
  146. booked ahead. Three columns are enough: visit date, number of guests and the day it was made. A
  147. cancellation date and a status column make it better. Names and contact details aren't needed.
  148. 2. **Upload it.** Columns are recognised in English and Dutch (`Datum`, `Aantal personen`,
  149. `Aangemaakt op`, `Geannuleerd op`), with commas or semicolons, with or without times. Statuses
  150. like *cancelled / geannuleerd* and *no-show / niet verschenen* are understood. Anything it
  151. can't place, you pick from a dropdown.
  152. 3. **Say what kind of place it is.** A hotel's guests nearly all book ahead. A restaurant also gets
  153. walk-ins, so the app adds an estimate on top of the reservations: by default walk-ins are 40% of
  154. all guests on Friday to Sunday and 20% Monday to Thursday, and you can change both. At 40%, 60
  155. guests expected from reservations become 100 in total. If your file already lists walk-ins, or
  156. you upload real daily totals from your till, nothing is added.
  157. 4. **Read the plan.** For each day: booked so far, the usual for that weekday, the forecast, a
  158. likely range, a flag on days 15% or more busier or quieter than usual, and staff per role from
  159. your own rules. Download it as Excel or CSV.
  160. Every plan is checked on your own data first. The app replays your last 8 weeks, rebuilding the
  161. book as it stood each morning, and shows how far off the forecast would have been at each
  162. distance. The likely range is the error that 8 in 10 of those past days stayed within.
  163. ![The plan with staffing](docs/plan-table.png)
  164. No file at hand? The app has a sample: a real city hotel's bookings as they stood on 31 May 2017,
  165. shown next to what really happened afterwards.
  166. ### Why this matters
  167. Restaurants and hotels decide every day how many people to schedule and how much to order. Most
  168. plan from a fixed weekly pattern ("Fridays are busy"). The reservation book often shows weeks in
  169. advance that a day will be unusual. This project measures how much that is worth.
  170. ### Results
  171. Mean absolute error in **guests per day**, lower is better. Forecasts for 1 Mar - 31 Aug 2017, made
  172. only from what was known on the morning of the forecast. The ML model was trained on Jul 2016 -
  173. Feb 2017 and never saw the test months. Source: [`reports/results.md`](reports/results.md) and
  174. [`reports/results.json`](reports/results.json).
  175. **City hotel** (140 guests arriving per day on average)
  176. | Days ahead | Weekday average | Same day last year | **Booking pace (rule)** | Gradient boosting (ML) |
  177. |---:|---:|---:|---:|---:|
  178. | 1 | 26.5 | 35.8 | **4.8** | 6.4 |
  179. | 7 | 26.5 | 35.8 | **8.6** | 8.9 |
  180. | 14 | 27.5 | 35.8 | **10.7** | 11.0 |
  181. | 30 | 32.0 | 35.8 | 18.3 | **16.7** |
  182. **Resort hotel** (75 guests arriving per day on average)
  183. | Days ahead | Weekday average | Same day last year | **Booking pace (rule)** | Gradient boosting (ML) |
  184. |---:|---:|---:|---:|---:|
  185. | 1 | 21.5 | 28.9 | **6.6** | 6.7 |
  186. | 7 | 21.5 | 28.9 | **10.2** | 10.6 |
  187. | 14 | 22.2 | 28.9 | 13.1 | **12.3** |
  188. | 30 | 21.6 | 28.9 | 16.4 | **14.3** |
  189. ![Forecast error by days ahead](reports/error_by_horizon.png)
  190. 1. **The book is the signal.** Correcting for booking pace cuts the error by 69-82% one day out and
  191. still by 24-43% a month out.
  192. 2. **Simple beats clever up to a week.** The two-line rule beats gradient boosting at every horizon
  193. from 1 to 7 days, at both hotels. A manager can check it by hand, which matters when it decides
  194. who gets a shift.
  195. 3. **ML earns its place further out.** A month ahead the book is still thin, and the model's use of
  196. seasonality and last year pays off: 9-13% better than the rule at 30 days.
  197. ![Forecast against actual arrivals](reports/forecast_vs_actual.png)
  198. ### How it works
  199. For a day `d`, standing `h` days before it ([`src/pace/models.py`](src/pace/models.py)):
  200. 1. **Weekday average.** Average the guests on the last 6 same weekdays that are already over.
  201. 2. **Booking pace.** Count the guests in the book for `d` now. Compare that with what was in the
  202. book for those 6 earlier days at the same distance before them.
  203. 3. **Correct.** Add the difference to the average, capped at ±50% so one odd group booking can't
  204. run away with the forecast.
  205. ![How the booking curve gave the answer away](reports/booking_curve.png)
  206. *28 April 2017, 7 days ahead: a large group was already in the book. The weekday average said
  207. 162, the booking pace said 243, and 241 guests came ([`reports/forecasts.csv.gz`](reports/forecasts.csv.gz)).*
  208. The reservation book is rebuilt from the raw bookings as a `day × days-before` matrix with a
  209. difference array ([`src/pace/data.py`](src/pace/data.py)). A booking counts from the day after it
  210. was made until it is cancelled.
  211. ### Proof it's honest
  212. Backtests leak the future easily. These checks guard against it and run on every push:
  213. - **The book only shows bookings that existed at forecast time.** A booking made on the forecast
  214. day itself doesn't change that forecast ([`tests/test_book.py`](tests/test_book.py)).
  215. - **Cancellations leave the book on their cancellation date. No-shows stay in the book but don't
  216. count as guests.** Each has a test.
  217. - **Strict time split:** training ends 28 Feb 2017, testing starts 1 Mar 2017.
  218. - **The published numbers are what the code produces.** A test reruns the whole backtest and fails
  219. if `reports/results.json` differs.
  220. - **An old export can't see the future.** A plan made from an export date gives the same answer
  221. whether or not later bookings are in the file ([`tests/test_plan.py`](tests/test_plan.py)).
  222. The app's second tab replays the full benchmark: pick a hotel, a distance and any day, and see the
  223. booking curve behind the number.
  224. ![Looking inside one day](docs/demo-day.png)
  225. ### Limits
  226. - **These are hotels, not restaurants.** Hotel guests book further ahead and rarely walk in, so
  227. the 1-day gain is bigger than a restaurant with walk-ins would see. The 7- and 14-day numbers are
  228. the fairer comparison.
  229. - **The restaurant walk-in estimate is a fixed share, not a forecast.** The 40% and 20% defaults are
  230. assumptions, not measured here, and the likely range only covers the reservations part. Real
  231. daily totals from the till replace the estimate with what really happened.
  232. - **Two hotels, one country, 2015-2017.** The method transfers; the exact percentages may not.
  233. - **Arrivals, not occupancy.** The target is guests arriving per day, the closest match to covers
  234. per evening in a restaurant.
  235. ### Where this is used
  236. The same rule runs every night in a staffing and scheduling system I built for a restaurant in
  237. Enschede. There it turns the forecast into staff per role from the manager's own rules, with a
  238. staff app for schedules, shift swaps and clocking in. That repo is private because it belongs to
  239. the restaurant.
  240. ## Run it yourself
  241. ```bash
  242. git clone https://github.com/rebekaolar-cyber/booking-pace-forecast
  243. cd booking-pace-forecast
  244. pip install -r requirements.txt -r requirements-agent.txt -r requirements-dev.txt
  245. streamlit run streamlit_app.py # the app, three tabs
  246. python -m pytest -q # 444 tests, none of them calls the API
  247. PYTHONPATH=src python -m pace.backtest # rebuilds reports/results.md
  248. PYTHONPATH=src python -m pace.charts # redraws the charts
  249. ```
  250. **API key for the assistant.** Put it in `.streamlit/secrets.toml`:
  251. ```toml
  252. ANTHROPIC_API_KEY = "..."
  253. ```
  254. or set the `ANTHROPIC_API_KEY` environment variable. `secrets.toml` is in `.gitignore` and is never
  255. committed. Without a key the chat tab says so and switches its input off; the other two tabs work.
  256. **MCP server.** `python -m agent.mcp_server` serves the five tools over stdio. Add
  257. `BPF_BACKEND=fake` for demo data. Claude Desktop setup: [`docs/mcp.md`](docs/mcp.md).
  258. ## Project layout
  259. ```
  260. agent/ the assistant
  261. contract.py the five read-only tools and run_tool
  262. loop.py the tool-use loop on the Messages API
  263. real_backend.py the tools on the real forecast; fake_backend.py for tests
  264. mcp_server.py the same tools over MCP
  265. run_log.py one JSON line per run, with token cost
  266. app_pages/ assistant.py (chat tab), ui.py (the app's look), demo_assistant.py
  267. evals/ questions.yaml (50 questions), scoring.py, run.py, compare.py
  268. reports/ the hand-audited live eval and its CI logs
  269. prompts/ v1.md, v2.md: the assistant's instructions, one file per version
  270. src/pace/ the forecast
  271. data.py load bookings, rebuild the reservation book per day
  272. models.py weekday average, same day last year, booking-pace rule
  273. ml.py gradient boosting on the same signals (scikit-learn)
  274. backtest.py replay Mar-Aug 2017 at 1-30 days ahead, write reports/
  275. charts.py the README charts
  276. plan.py read any reservations export, plan, self-check, staffing
  277. tests/ no-leakage checks, export reading, reproducible results, the loop,
  278. the scorer, the MCP server, the app itself
  279. docs/ screenshots, architecture.png, mcp.md
  280. reports/ benchmark results and charts
  281. data/ the hotel booking dataset
  282. static/ the app's fonts and icon
  283. streamlit_app.py the app
  284. ```
  285. ---
  286. Built by **Rebeka Olar**, Business Information Technology student at the University of Twente (NL).
  287. The agent layer (assistant, MCP server and eval) was built with Claude Code; I designed and
  288. reviewed it.
  289. **Data:** Antonio, N., de Almeida, A. & Nunes, L. (2019). *Hotel booking demand datasets.* Data in
  290. Brief, 22, 41-49. [doi:10.1016/j.dib.2018.11.126](https://doi.org/10.1016/j.dib.2018.11.126).
  291. Licensed [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). The authors removed all hotel
  292. and customer identifiers. `data/hotels.csv.gz` is the unmodified CSV, gzipped.
  293. Code: MIT licence. Fonts: Bricolage Grotesque and Instrument Sans, both under the SIL Open Font
  294. License 1.1 (licence texts in `static/fonts/`).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.