6. Evaluation, Safety & Production Readiness

Test cases pulled from the customer's actual usage patterns catch failure modes a synthetic or generic test set never surfaces — the messy, specific edge cases real users produce.

A high aggregate score can hide a small number of severe failures. Read results broken down by category and severity, not just an overall percentage — a 95% pass rate is meaningless if the failing 5% are the highest-stakes cases.

An evaluation set showing 95% overall pass rate can still mean the system isn't ready — if the failing cases are concentrated in exactly the scenario the customer cares most about.

Offline catches known failure modes before anyone is affected. Online catches the failure modes you didn't anticipate — the ones real, unpredictable usage surfaces that a fixed test set never could.

(Side-by-side split)

A threshold should be set deliberately based on the use case's risk tolerance — a customer-facing legal summary needs a much higher bar than an internal draft-generation tool. The same threshold across every feature ignores that risk isn't uniform.

Two features at the same company legitimately warranting different acceptance thresholds is normal, not inconsistent — the deciding factor is what happens when each one is wrong.

Direct injection is an explicit override attempt typed by the user. Indirect injection is harder to catch — malicious instructions hidden inside a retrieved document, email, or webpage the model processes as part of its normal workflow.

You don't need to run the red-team exercise yourself, but you need to read results by severity: a jailbreak producing mildly off-brand text is a different risk tier than one that exfiltrates another customer's data.

A RAG system that retrieves and processes external content (Section 5) is exposed to indirect injection risk by design — the retrieval pipeline itself is a potential attack surface, not just the chat input box.

Guardrails catch outputs before they reach the user. Fallback paths activate when the primary approach fails. Review queues route uncertain cases to a human. Escalation paths handle high-stakes situations. Graceful degradation prevents a hard crash.

(Decision tree)

A single test run rarely exercises failure paths — they only get proven out under real, sustained production load, which is exactly why they need to be designed deliberately rather than assumed to work.

Input tokens (including retrieved context from RAG), output tokens, and per-token pricing all multiply against request volume. A per-request increase that looks negligible in one test call compounds dramatically at scale.

A deployment's cost tripling after a seemingly minor retrieval setting change is a scale problem, not a billing error — the fix is auditing what changed in context size per request, not assuming something broke.

A model upgrade can silently change output style or reliability on your specific use case, even when the provider frames it as a strict improvement overall. General improvement claims don't guarantee improvement on your specific task shape (Section 4's model selection lesson).

Track model cards for changes, re-run your own evaluation set against any new version before switching production traffic, and have a migration plan rather than being forced into an emergency one when a version gets deprecated.

A provider announcing a new model as a strict improvement is not a green light to switch immediately — re-testing against your own eval set first is what catches a use-case-specific regression before customers do.

Who approves a production change, and who can view sensitive outputs or alter configuration, need to be explicit — not assumed defaults that no one actually decided on.

Being able to reconstruct and prove what the system did after the fact is what compliance, incident review, and customer trust actually depend on — not just correct behavior at the time.

When a customer's compliance team asks "can you prove the system didn't do X on this date," the audit trail is the entire answer — there's no substitute for it after the fact.

Combine the evaluation approach from earlier in this section with a governance specification: who approves production changes, what gets logged, how long logs are retained, and who can access them.

A log confirms something happened. A trace shows the full path of a request through the system — retrieval, prompting, tool call, model response — pinpointing exactly where in a multi-step pipeline something went wrong.

(Side-by-side split)

Debugging a multi-step RAG or agent pipeline (Section 5) with only isolated error logs means guessing which stage failed — a full trace answers that question directly.

Real severity — not assumed, panic-level severity — determines the right response speed. Treating every incident as maximally urgent burns trust and attention that a genuinely critical incident later needs.

A good RCA is specific: what happened, when detected, the immediate fix, the true root cause, and concrete preventive changes. "We'll be more careful" prevents nothing — a specific root cause and a specific fix does.

This closes the loop with Section 5's RAG diagnosis and Section 9's live diagnosis under pressure — the same methodical instinct, now applied to writing up what happened after the fact so it doesn't happen the same way twice.

Document what happened, when it was detected, the immediate fix, the true root cause, and specific preventive changes — blameless and specific, not a vague commitment to be more careful.