AI in Production: What the Hype Skips

The demo is always impressive. The production system is a different conversation.

AI capability has outpaced AI reliability infrastructure. The gap between what a model can do in a controlled demonstration and what it consistently does in production — at scale, across the full distribution of real inputs, without human review of every output — is where most AI deployments encounter their actual problems.

For regulated environments, that gap has compliance implications that generic AI adoption advice doesn't address.

The production reliability problem

Language models are probabilistic systems. The same input can produce different outputs at different times, depending on sampling parameters, model updates, and infrastructure state. In many applications this is fine — variation in a marketing copy suggestion doesn't matter much. In applications where consistency, accuracy, or auditability are requirements, it matters significantly.

McKinsey's 2024 State of AI report found that organizations cite "inaccurate outputs" and "explainability" as the top barriers to AI deployment in production settings. These aren't edge-case concerns — they're the gap between demo performance and production performance that appears as soon as the input distribution widens beyond the cases the demo covered.

Hallucination in regulated contexts. AI hallucination — confidently stated outputs that are factually incorrect — is a known characteristic of current language models, not a bug that future versions will eliminate. For most use cases, the risk is manageable with appropriate human review. In regulated contexts where AI outputs inform compliance decisions, patient care, financial transactions, or legal determinations, hallucination has liability implications that require explicit mitigation strategies — not assumptions that the model is accurate enough.

Model drift. Foundation models are updated by their providers, and updates can change output characteristics in ways that break downstream applications. A prompt that reliably produces the correct structured output format in January may not produce the same format after a model update in March. Production AI systems need monitoring that detects behavioral drift, not just infrastructure availability monitoring.

The long-tail input problem. Demos select inputs that showcase model capabilities. Production systems receive the full distribution of real user inputs, which includes phrasing, edge cases, languages, and contexts that demos don't cover. Performance on selected demo inputs is a ceiling estimate for production performance, not an average.

What regulated environments require

Audit trails for AI decisions. Any AI system that influences decisions in a regulated context needs to produce a record of what input it received, what output it produced, and what version of the model produced it. "The AI said so" is not sufficient documentation for a compliance audit. The audit trail requirements are not fundamentally different from those for any other system that influences regulated decisions — the difference is that AI systems are often built without this infrastructure because it wasn't required in the demo phase.

The intersection with security-first architecture is direct: audit logging as a structural feature, not a retrofit, is what makes AI systems auditable without a remediation project.

Human-in-the-loop for high-stakes outputs. For decisions with material consequences — credit decisions, clinical recommendations, legal determinations, compliance classifications — AI outputs should be treated as decision support, not decision automation, until reliability has been validated at the specific level of precision the domain requires. The temptation to automate everything misses the liability that attaches when an automated system produces an incorrect output with material consequences.

Data governance for training and retrieval. Retrieval-augmented generation (RAG) systems — AI that answers based on your internal documents — are only as accurate as the documents they retrieve from. If the document set contains outdated, incorrect, or inconsistent information, the AI amplifies those errors with confident delivery. Data governance — ensuring the information the AI retrieves is current, accurate, and properly scoped — is a prerequisite for RAG reliability, not an optional addition.

Vendor risk in AI supply chain. The AI system is rarely just the model. It's the model, the API provider, the orchestration framework, the vector database, the embedding service, and the deployment infrastructure. Each is a vendor with its own availability SLA, data processing terms, and security posture. The same vendor due diligence that applies to any third-party system with access to sensitive data applies here — with the added consideration that AI vendor relationships often involve sending your data to train or improve models, which has its own data classification implications.

---

Related reading

• The Problem with Generic Tech Recommendations
• If the Recommendation Came Before the Questions, It's Not Advisory
• "This Pricing Expires Friday" and Other Reasons to Slow Down
• Stop Watching What Everyone Else Is Building
• What I Tell Every CEO Who Asks Me About AI

Frequently asked questions about ai in production

Is AI appropriate for regulated industries at all?

Yes, with appropriate architecture. The use cases where AI adds genuine value in regulated environments — document summarization, pattern detection, draft generation for human review, support routing — are real. The error is treating AI as an off-the-shelf automation solution rather than a component that requires the same integration, testing, and monitoring discipline as any other production system.

How do you evaluate an AI vendor for production use?

The same way you evaluate any vendor with access to sensitive data: data processing terms, retention policies, security posture, incident response, SLA for uptime and support. Add AI-specific questions: how does the model handle inputs that fall outside expected distributions? What is the escalation path when the model produces outputs of low confidence? What happens to data submitted for inference — is it used for training?

What's a reasonable starting point for AI in a regulated business?

Internal-facing applications with human review of outputs. Use AI to draft, summarize, and surface — not to decide. Build the audit infrastructure before you scale adoption. Validate accuracy on a representative sample of your actual input distribution before drawing conclusions from demo performance. The companies that have the most durable AI advantages are the ones that built the infrastructure right the first time, not the ones that moved fastest.