Every enterprise now has a GenAI demo that impressed a leadership meeting. Very few have the same workload running in production with SLOs, cost controls and an audit trail. The gap between those two states is not model quality — it is platform engineering. This is the checklist we work through when we take a GenAI workload to production.
Serving: treat models like services, not scripts
- Latency and availability SLOs defined per use case — a chatbot tolerates 2 seconds; an autocomplete does not tolerate 200 ms.
- A real serving stack (vLLM, KServe, or managed endpoints) with health checks, autoscaling and graceful degradation, not a notebook behind a Flask route.
- Version pinning for models and prompts. "The model changed under us" is the new "works on my machine."
- Fallback routing: when the primary model times out or hits quota, requests degrade to a cheaper model or a cached answer instead of failing.
Evaluation: your regression tests now include prompts
Model upgrades, prompt edits and RAG index changes all silently change behavior. Production GenAI needs an evaluation harness the same way production code needs CI: a golden dataset of real queries, scored automatically on every change, with a quality gate that blocks regressions before deploy.
- A golden set of 100–500 real, anonymized queries with graded expected outcomes
- Automatic scoring on every prompt/model/index change — LLM-as-judge is fine to start
- Quality gates wired into the same CI/CD that deploys the change
RAG: retrieval quality is measurable — measure it
Most "the model is hallucinating" complaints are retrieval failures. Instrument retrieval precision and answer groundedness separately from generation quality, version your embeddings and chunking strategy, and rebuild indexes as a pipeline, not a one-off script someone ran in March.
Cost: GPUs and tokens need budgets before launch
- Scale-to-zero on GPU node pools — idle inference GPUs are the fastest way to torch an AI budget
- Per-workload token budgets with alerts, not one shared key for the whole company
- Model routing: small models for classification and extraction, frontier models only where reasoning quality pays for itself
- Prompt and response caching — duplicate questions are 30–60% of real traffic in most internal tools
Governance: what security will ask before sign-off
If AI agents touch internal systems, they need identities, scoped permissions and audit logs like any other principal — this is where MCP-based tool access with non-human identity governance earns its keep. Expect questions about data residency, prompt injection handling, PII in logs, and who can change the system prompt. Having answers written down is the difference between a two-week and a two-quarter approval.
Observability: traces, not vibes
Every request should carry a trace from user input through retrieval, generation and post-processing, with token counts and latency at each hop (OpenTelemetry has GenAI semantic conventions now — use them). When quality drops, you want to know whether the model, the index or the prompt changed, in minutes, from a dashboard.
If your GenAI workload passes this list, ship it. If it fails more than a couple of items, that is normal — it is exactly the gap our MLOps/LLMOps practice closes, and our free AI readiness audit will tell you which gaps matter for your specific workload.