"It works in the demo" has stopped meaning anything to me. Almost everything works in a demo — the demo is curated, the inputs are friendly, and nobody is trying to break it. The question that actually predicts whether a system survives production is a narrower one: what happens on the inputs nobody planned for? Over a few years of shipping AI features, I've collapsed that question into five I ask before signing off on anything.
1. What happens when it's wrong?
Not if — when. Every model is wrong some percentage of the time, so the design question is what the failure looks like to the user. A wrong movie recommendation costs nothing. A wrong medical dosage suggestion or a wrong invoice total costs a lot. The acceptable error rate isn't a fixed number — it's a function of how expensive a single wrong answer is, and that should decide how much human review sits between the model and the consequence.
2. Can I see why it made this decision?
Not full interpretability — that's an unsolved research problem for most model classes. I mean something more modest: can I trace an output back to the inputs that produced it? For a RAG system, that's "which chunks were retrieved." For a classifier, that's confidence scores and nearest training examples. If the answer is "no, it's a black box end to end," debugging a bad outcome in production becomes guesswork, and guesswork doesn't scale past the second incident.
3. What's the fallback when confidence is low?
Systems that always answer, even when they shouldn't, are more dangerous than systems that sometimes say "I'm not sure." A well-designed AI feature has an explicit low-confidence path — defer to a human, ask a clarifying question, or return a narrower, hedged answer. If the only two states are "confident answer" and "confident-sounding wrong answer," that's a design gap, not a model limitation.
4. How was it evaluated, and against what baseline?
"We tested it and it seemed good" is not an evaluation. A real answer names a metric, a test set that resembles production traffic (not hand-picked examples), and a baseline — the current human process, the previous system version, or a naive heuristic. Without a baseline, "90% accuracy" is a number with no meaning: 90% might be a huge win or a regression, depending entirely on what it's being compared against.
5. Who is accountable when it fails in production?
This is the question teams skip because it's organizational, not technical. If a model-driven decision causes harm — a wrong charge, a wrongly denied request, a bad medical suggestion — there needs to be a named owner and a defined escalation path before launch, not improvised after the first incident report. "The model did it" is not an incident response.
Using it as a gate, not a vibe check
I run through these five in order, in writing, before any AI feature ships — not as a philosophical exercise but as a literal checklist attached to the launch doc. None of them require deep ML expertise to answer. What they require is treating "does it work" as a much smaller question than "is it safe to depend on," and refusing to ship until both have real answers.
I write about what actually holds up when AI features meet real users, in my newsletter, AI Shipped. New issue every week.