Module 90: AI Application Project
Deployment
🏠The everyday picture
A new menu item is tried first with regular customers who give honest feedback, then offered to everyone, with the kitchen watching how it goes in the first weeks.
📘What it really is
-
Deploy with your Level 6 practices: CI with tests and evals (Module 71), staging first, smoke tests, approved production release, rollback ready (Module 70), secrets in the secrets store (Module 77), monitoring and alerts (Modules 72, 89).
-
Roll out gradually:
- internal pilot: a few trusted staff, with the feature flag on just for them; collect feedback daily;
- review the pilot's conversations (privately and carefully), thumbs-down answers and "couldn't find" questions; fix knowledge and prompts; re-run evals;
- wider release to the whole team, with a short guide: what it's for, what it isn't, how to check sources, how to report problems;
-
customer-facing use only later, if ever, after its own risk assessment and much more testing.
-
Train people: the assistant is a helper; staff stay responsible for what they tell customers. Encourage checking sources, especially for anything important.
-
Document limits and risks openly in
assistant-limits.md: what it covers, what it doesn't, known weaknesses, the data it uses and sends, and who owns it. -
After launch: weekly reviews (Module 89); monthly eval runs and cost reviews; quarterly knowledge-base review; and an incident process for harmful answers (Module 80).
-
When models change (new versions, retirements): run your evals on the new model in staging before switching; keep the model name in configuration so switching and rolling back are easy.
🖱️Try it
Deploy the assistant to staging with your CI/CD pipeline, run smoke tests and the eval suite, then release it to production behind
a feature flag for a pilot group (yourself and one or two others). Write the staff guide and assistant-limits.md. After a week of
pilot use, write a short pilot review with three improvements, made and re-tested.
✅Quick check
Why start with an internal pilot?
Show the answer
To find problems with a small, trusted group before everyone relies on it.
What should you do before switching to a new model version?
Show the answer
Run your evals on it in staging and compare with the current model.