How much validation does your AI actually need? Start with Context of Use

Two AI systems sit in the same building. One scores cold-chain risk on shipments; the other drafts deviation root-cause analyses and can flag a deviation’s severity in the quality system. A team that validates both the same way is either wasting months on the first or under-controlling the second — and only one of those mistakes shows up in an inspection.

The difference isn’t the technology. It’s the Context of Use. Get the classification right and everything downstream — how much evidence you gather, which controls you build, what an inspector will expect — falls out of it. Get it wrong and you either burn validation budget you didn’t need or ship a finding waiting to happen. So before you validate a GxP AI system, classify it. This is how.

Why classification comes first

Watch enough teams validate AI and the same pattern emerges independently: risk-classify by context of use first, then scale everything else to that. It’s the opening move of the converging validation recipe for a reason — classification is the dial that sets how much credibility evidence you owe.

The dial has a name: model risk = model influence x decision consequence. How much does the model drive the decision (versus a human who meaningfully reviews it), times how serious the decision is (patient safety, product quality, data integrity). The same model can pass for a low-stakes use and fail for a high-stakes one. That’s the whole game — and it’s why “is this model validated?” is the wrong question. The right one is “validated for what?”

This is also the business case. Classification is what lets you right-size the spend: over-classify and you pour validation effort into a low-risk advisory tool; under-classify and you put an unvalidated system near a GMP record and inherit a 483. The classification step is cheap. Both errors it prevents are expensive.

The vocabulary problem (and the bilingual key)

Here’s what makes this hard: four different stakeholders will classify the same system using four different frameworks, and they don’t share an axis.

Framework	Whose language	Classifies by	Output
FDA — Context of Use	US regulatory / QA	model influence x decision consequence	a risk level on a sliding scale
EU Annex 22	EU GMP / CSV	model type x criticality	permitted-in-critical vs non-critical only
GAMP 5	CSV / validation	software category + risk	Category 3 / 4 / 5 lifecycle rigor
EU AI Act	legal / compliance	use-case risk tier	minimal / limited / high-risk / prohibited

They overlap but they are not interchangeable. FDA asks how much evidence does this job need? Annex 22 asks is this model type even allowed near critical GMP? GAMP asks how custom is the software? The AI Act asks what’s the societal risk tier? A defensible classification answers all four — and the “bilingual” move that makes you useful in the room is translating one into the others. The procedure below produces all four outputs from one pass.

The classification procedure

Five steps. Run them in order; the early answers constrain the later ones.

Step 1 — State the question of interest and the Context of Use. One sentence each. The question of interest is the decision the output supports (“should this deviation be classified minor or major?”). The COU is the specific role and scope of the model in answering it (“drafts a proposed severity for SME review; does not finalize”). Credibility is always assessed relative to a COU, never in the abstract. If you can’t write the COU in one sentence, you can’t classify the system yet.

Step 2 — Score model influence. How much does the model actually drive the decision? Low: a human independently reaches the same decision and the model is advisory. Medium: the human reviews every output but anchors on it. High: the model’s output is acted on with no meaningful human gate (it writes to a system, triggers an action, or is rubber-stamped at volume). The presence of structural human-in-the-loop — powers withheld from the tool, low-confidence cases held in a queue — lowers influence; a human who “checks” 200 outputs an hour does not.

Step 3 — Score decision consequence. If the output is wrong and acted on, what’s the worst credible harm? Score against the GxP triad: patient safety, product quality, data integrity. Advisory search over public literature is low; altering a batch-disposition-adjacent record is high.

Step 4 — Place it on the matrix. Influence x consequence gives you a risk band:

	Consequence: Low	Consequence: Med	Consequence: High
Influence: High	Medium	High	Critical
Influence: Med	Low	Medium	High
Influence: Low	Low	Low	Medium

Step 5 — Map the band onto the other three lenses. Now translate:

• Model-type eligibility gate (Annex 22) first. Is the model static + deterministic, or dynamic / probabilistic / GenAI? Under the current Annex 22 draft, GenAI/LLMs and probabilistic models are excluded from critical GMP outright — so a high-band GenAI system can’t simply be “validated harder” into a critical role; it must be re-scoped to non-critical with documented oversight, or the model type changed.
• GAMP category — configured product (Cat 4) vs custom/bespoke (Cat 5) sets lifecycle rigor.
• EU AI Act tier — systems used in medicinal-product manufacturing often land high-risk.

The output of the pass is a single line: “COU X; influence Y x consequence Z = risk band B; Annex 22 {permitted-critical | non-critical only}; GAMP Cat C; AI Act {tier}.” That line is your classification of record.

The worksheet

Steal this. One page, filled at Step 5, attached to the validation plan:

Field	Entry
System / pilot name	—
Question of interest	the decision the output supports
Context of Use (one sentence)	the model’s specific role + scope + where the human gate is
Model type	static-deterministic / dynamic / probabilistic / GenAI
Model influence (L/M/H)	+ one line of justification
Decision consequence (L/M/H)	+ which of patient-safety / product-quality / data-integrity
Risk band	from the matrix
Annex 22 eligibility	permitted-in-critical / non-critical-only
GAMP category	3 / 4 / 5
EU AI Act tier	minimal / limited / high-risk
Evidence rigor that follows	light / standard / full credibility package

Worked examples

Two systems from a composite mid-size biologics company (“Northbrook Biologics”), classified end to end. Same word — “AI” — opposite regimes.

System A — cold-chain risk classifier. A static, deterministic ML model scores inbound shipments for excursion risk; a logistics analyst reviews the score before any action.

• COU: advisory risk score on shipments, reviewed by an analyst before disposition.
• Influence: Low (analyst decides; model informs). Consequence: Medium (product quality, but human-gated and not irreversible).
• Risk band: Low–Medium. Annex 22: permitted in critical (static + deterministic clears the eligibility gate). GAMP Cat 4. AI Act: limited. → Standard validation rigor; no exotic controls.

System B — deviation-investigation agent. A GenAI agent reads deviation reports, retrieves similar past cases (RAG), drafts a root cause, and has a tool to flag severity in the quality system.

• COU: drafts severity + RCA for SME review — but the severity-flag tool means it can write to a regulated record.
• Influence: High (it can act on the record, and at volume a reviewer rubber-stamps). Consequence: High (severity misclassification is a data-integrity and patient-safety issue).
• Risk band: Critical. Annex 22: GenAI is excluded from critical GMP — so this COU is not permitted as designed. It must be re-scoped to non-critical (remove the write-capability, make it draft-only with a hard human gate) to be allowed at all. GAMP Cat 5. AI Act: high-risk. → Full credibility/validation package, and a design change before it can ship.

The payoff: System B’s problem isn’t “validate it more.” It’s that the classification revealed the COU is non-compliant by design. You learned that in an afternoon with a worksheet — not in month three of validation, or worse, in an inspection.

So what: classification sets the evidence burden

The point of classifying is that the rigor follows from the band. Low band: light evidence, standard CSV. Critical band: the full package — and for an LLM/agent that means the security controls double as validation evidence. Identity and access at the gateway, input/output guardrails, a human-approval gate on high-risk tools, red-team results read as OQ evidence, and a tamper-evident audit trail. The Context of Use is the scaffold all of that hangs on: it tells you which controls you owe and how much proof each one needs.

That’s the bridge to the rest of this work — what a CSV validator should look for in an LLM gateway is the control layer; security is now a GMP requirement is why those controls are now validation deliverables. Classification is the step that comes before both.

The agent edge

One honest complication. Context of Use is clean for a static model with one output. It gets slippery for an agent that calls tools — because each tool is its own little COU with its own influence and consequence. “Send an email” and “flag deviation severity” can’t share one classification. The useful framing is per-tool COU: score influence x consequence for each action the agent can take, and let the highest-risk tool set the controls. That’s an open problem worth its own piece — for now, if your “system” is an agent, classify its tools, not just the agent.

What this doesn’t cover

The “critical GMP application” line is still undefined even after the Annex 22 consultation — the gate everything hinges on remains fuzzy. This post also doesn’t cover reclassification on drift (a model that degrades can cross a band), supplier-provided model qualification, or the full GAMP lifecycle. For those, start with the FDA AI credibility guidance, EU GMP Annex 22, GAMP 5 (2nd ed.), the NIST AI RMF, and the EU AI Act high-risk provisions.

---

I’m a pharma-systems consultant who helps CSV/QA and AI-platform teams get GenAI and agents into GxP workflows defensibly — starting with exactly this classification step. If you’re staring at a pilot the business wants in production and no one’s agreed how much validation it owes, get in touch.