Your In-App AI Feature Survived the Demo. Now What? Production Patterns for LLM Features That Hold Up After Day 90

Demos are optimized for the happy path. One model, curated documents, a single concurrent user, and questions the team already practiced. Production is the opposite: partial retrieval hits, ambiguous prompts, rate spikes, provider blips, and answers that must survive scrutiny from people who did not sit in the design meeting.

That gap is the demo tax. You paid it when you optimized for conviction instead of ownership. The five failure modes we see most often after launch are predictable:

• Cost: every session hits a frontier model for work a cheaper model could handle.
• Quality drift: retrieval degrades as content grows, and nobody notices until support tickets rise.
• Missing citations: answers look confident without sources the user can open and verify.
• Vendor lock-in: the feature is hard-wired to one provider SDK, so an outage or pricing change becomes an incident.
• No approval UX: the model proposes irreversible actions with nothing between draft and commit.

“A demo proves possibility. Production proves you can still say no, switch, measure, and recover.”

If you only take one idea from this article, take this: treat the demo as a prototype gate, not a launch ceremony. Everything below is how we design for the day the applause ends.

Teams that skip evaluation ship vibes. Teams that write the definition of good first ship features they can defend. Before production traffic, we want three artifacts in writing: an offline eval set, numerical thresholds, and a list of actions that never auto-execute.

Build an offline eval set that looks like real use

Pull representative queries from support history, sales questions, and internal workflows. Include the boring ones and the hostile ones. A useful set covers: expected answers with known source documents, questions that should refuse (out of scope, policy, or insufficient data), and red-team prompts that try to override system instructions or extract secrets.

Metrics that matter for product AI

• Answer quality on the eval set, plus retrieval recall when RAG is involved.
• Latency p95 against a budget the UI can live with (streaming helps, but slow first tokens still feel broken).
• Cost per request and cost per successful task, not just token averages in a spreadsheet.
• Refusal rate on cases that should refuse, and false-refuse rate on cases that should answer.
• User feedback hooks ready before launch, so wrong answers feed the next eval cycle.

The gate is binary on purpose. If thresholds fail, the feature stays behind a flag and the team expands fixtures. Live traffic is not how you discover that your retrieval chunking is wrong. This mirrors how we run AI-Integrated Solutions engagements: use-case and risk review first, then prototype and benchmark, then productize.

Red lines before any model call

Write down what the feature must never do alone: billing changes, deletes, privilege grants, legal or medical send-outs, bulk exports of sensitive data. Those become product rules and automated checks, not hopes in a system prompt. If it is not enforced in code and UX, it is not a red line.

Hardcoding one frontier model for every call is a product and finance risk. Simple classification does not need the same model as multi-step synthesis. A provider outage should not take the entire surface offline. Routing is how you make those facts structural.

In production we treat the path as: user action in the app, intent and complexity classification, router selection, provider adapter, then streaming response with structured logs. Failover sits next to the adapter, not as a weekend hotfix.

Route by task class

• Simple: classify, rephrase, short FAQ style answers. Prefer a fast, low-cost model.
• Complex: multi-hop reasoning, long synthesis, ambiguous product questions. Reserve frontier capacity.
• Structured: JSON extraction, constrained tool steps, schema-bound outputs. Prefer models and decoding paths that hold format.

Unified adapter layer

The application should speak one contract. We use TanStack AI as a multi-provider adapter layer so swapping xAI, OpenAI, Anthropic, or another backend is a configuration change, not a rewrite of the chat UI and server handlers. That same idea applies whether you use TanStack AI or another gateway: the product must not import one vendor’s SDK into every call site.

What you should not route blindly: high-stakes summaries where a single audited model version is a compliance requirement, or flows where silent model switching would invalidate an eval baseline. Pin those paths. Log the pin. Review pins on a schedule.

Chat without retrieval can still be useful for general drafting. For product answers about your domain, ungrounded generation is a liability. Retrieval-augmented generation only earns trust when the UI and the access model match how people verify work.

Chunking and metadata for how users ask

Split content on meaning, not arbitrary character counts alone. Attach metadata that filters retrieval: product area, audience, locale, document type, last updated. Query construction should use those filters when the user context already implies them (role, tenant, selected account).

Access control at retrieval time

Login is not enough. If a user cannot open a document in the app, the model must not answer from it either. Enforce role and tenant scope in the retrieval layer, not only in the page shell. This is the same discipline we apply to enterprise platforms with SSO and RBAC: authorization travels with the data path.

Always show sources

Every grounded answer should surface citations the user can open. Design the UI for verify, not just read. On our own site, the Ask assistant is built around knowledge the site actually publishes; the same principle applies inside client products. If you want the implementation story, see Building Ask: How We Put Our Entire Website Into an AI Assistant.

Freshness is an operations problem

Content changes. If your index does not, answer quality decays while the model still sounds confident. Build re-index jobs, document watchers, or publish hooks so new and edited material becomes answerable on a known schedule. Measure stale-hit rates the way you measure error rates.

When pure chat without retrieval is the wrong product: policy questions, account-specific status, anything where a wrong answer creates liability. In those cases, no retrieval means no feature, not a freer model.

Enterprise buyers are pushing agents into applications faster than governance catches up. The teams that scale are not the ones that remove humans from every step. They define autonomy boundaries and encode them in the interface.

Human-in-the-loop is not an apology for incomplete automation. It is how you ship useful AI without gambling the business on a single model completion.

Three tiers that map to UI

• T0 Suggest: low-impact help (summarize, rephrase). Auto-run is fine; review is optional.
• T1 Draft: proposed CRM updates, ticket replies, form fills. The system produces a draft. Apply or send is explicit.
• T2 Act: refunds, deletes, contract sends, role changes. Blocked until confirm, with an audit record of who, what, when, and context.

UX patterns that work

Draft not commit. Suggestion chips instead of silent mutations. Diff views before apply. Approval queues for high-risk actions. Streaming text is fine; silent side effects are not. If the model can change state, the user must see the proposed change and own the confirmation.

This product stance pairs with platform readiness. If agents and tools will call your system later, design the same boundaries into APIs and audit logs. Our enterprise practice treats agent-callable surfaces and hand-off as day-one architecture, not a retrofit. See Enterprise Web Development and our note on agent-callable APIs when the surface must serve machines as well as people.

If you cannot answer which model served a request, what it cost, how long it took, and whether retrieval hit, you do not own the feature. You are renting a demo that happens to be in production.

Log the decisions, not only the text

• Prompt or policy version identifiers
• Model id and provider after routing
• Route decision (simple, complex, structured, failover reason)
• Latency to first token and total duration
• Token or cost estimate per request
• Retrieval hit rate and source ids when RAG runs
• User feedback (thumbs, “this was wrong,” ticket link)

Dashboards product and eng can both read

Cost and quality should not live only in an ML notebook. Product owners need error rates, cost per session, and refusal spikes in the same place eng looks during an incident. When quality drops after a content import or a model bump, the signal has to be obvious without a research sprint.

Feature flags are part of the architecture

Ship with the ability to disable the surface, pin a model, force a cheaper route under budget pressure, or fall back to non-AI UX. Those controls belong next to the router, not as a scramble after a invoice shock. Kill switches are product quality.

Feed user corrections back into the eval set. A wrong answer that never becomes a fixture will return. Day 90 health is a loop: observe, expand fixtures, re-run the gate, adjust routing and retrieval, ship again behind flags.

Use this as a shipping scorecard. If half the list is still “we will get to it,” you are still in demo territory.

• Offline eval set with happy path, refusal, and red-team cases
• Written thresholds for quality, latency, cost, and safety
• Red lines encoded in code and UX (not only in prompts)
• Multi-model routing with a unified provider adapter
• Failover path tested, not theoretical
• Retrieval scoped by role and tenant when data is private
• Citations visible and openable in the UI
• Re-index or freshness process with a known cadence
• Risk tiers (suggest / draft / act) mapped to screens
• Audit log for T2 actions
• Structured logs: model, route, latency, cost, retrieval
• Feature flags: disable, pin model, force cheap route
• Feedback path that updates the eval set

If you need a partner to build the surface, that is our AI-Integrated Solutions work: multi-provider LLMs, RAG, streaming UI, guardrails, and observability. If the larger platform still lacks SSO, audit hooks, or agent-ready APIs, start with Enterprise Web Development. If the team is still choosing tools and workflow before product AI, AI & Tech Strategy Consultation is the shorter path to a plan you can execute.

For data that cannot leave the building, our private on-premise AI service is a different path with the same respect for citations and human verification. Cloud product AI and private office AI answer different constraints; the production discipline (eval, scope, observe) still applies to both.