How much validation does your AI actually need?
GMP AI risk turns on influence, consequence, uncertainty, complexity, and whether a failure is easy to see. Workshop input, not final guidance.
Two AI systems sit in the same building. One scores cold-chain risk on shipments; the other drafts deviation root-cause analyses and can flag a deviation’s severity in the quality system. A team that validates both the same way is either wasting months on the first or under-controlling the second, and only one of those mistakes shows up in an inspection.
What separates them is the Context of Use. Get the classification right and everything downstream falls out of it:
- how much evidence you gather
- which controls you build
- what an inspector will expect
Get it wrong and you either burn validation budget you didn’t need or ship a finding waiting to happen. So before you validate a GxP AI system, classify it. This is how.
Why classification comes first
Watch enough teams validate AI and they land on the same order independently: risk-classify by context of use first, then scale everything else to that. It’s the opening move of the converging validation recipe for a reason: classification is the dial that sets how much credibility evidence you owe.
Model risk = model influence x decision consequence.
How much does the model drive the decision (versus a human who meaningfully reviews it), times how serious the decision is (patient safety, product quality, data integrity). The same model can pass for a low-stakes use and fail for a high-stakes one. That’s why “is this model validated?” is the wrong question. The right one is “validated for what?”
This is also the business case. Classification is what lets you right-size the spend:
- Over-classify and you pour validation effort into a low-risk advisory tool.
- Under-classify and you put an unvalidated system near a GMP record and inherit a 483.
- The classification step is cheap. Both errors it prevents are expensive.
Influence and consequence set the band. Uncertainty, complexity, and detectability, covered in the risk section, change how much evidence that band requires.
The vocabulary problem (and the bilingual key)
Here’s what makes this hard: four different stakeholders will classify the same system using four different frameworks, and they don’t share an axis.
FDA Context of Use
Whose language: US regulatory / QA
Classifies by: model influence x decision consequence
Output: a risk level on a sliding scale. Asks how much evidence does this job need?
EU Annex 22
Whose language: EU GMP / CSV
Classifies by: model type x criticality
Output: permitted in critical vs non-critical only. Asks is this model type even allowed near critical GMP?
GAMP 5
Whose language: CSV / validation
Classifies by: software category + risk
Output: Category 3 / 4 / 5 lifecycle rigor. Asks how custom is the software?
EU AI Act
Whose language: legal / compliance
Classifies by: use-case risk tier
Output: minimal / limited / high-risk / prohibited. Asks what’s the societal risk tier?
They overlap but they are not interchangeable. A defensible classification answers all four. The “bilingual” move that makes you useful in the room is translating one into the others. The procedure below produces all four outputs from one pass.
The classification procedure
Five steps. Run them in order; the early answers constrain the later ones.
Step 1 State the question of interest and the Context of Use
One sentence each. The question of interest is the decision the output supports (“should this deviation be classified minor or major?”). The COU is the specific role and scope of the model in answering it (“drafts a proposed severity for SME review; does not finalize”). Credibility is always assessed relative to a COU, never in the abstract. If you can’t write the COU in one sentence, you can’t classify the system yet.
Step 2 Score model influence
How much does the model actually drive the decision?
- Low A human independently reaches the same decision and the model is advisory.
- Medium The human reviews every output but anchors on it.
- High The model’s output is acted on with no meaningful human gate (it writes to a system, triggers an action, or is rubber-stamped at volume).
Structural human-in-the-loop (powers withheld from the tool, low-confidence cases held in a queue) lowers influence. Say a reviewer is asked to “check” 200 outputs an hour: that does not.
Step 3 Score decision consequence
If the output is wrong and acted on, what’s the worst credible harm? Score against the GxP triad: patient safety, product quality, data integrity. Advisory search over public literature is low; altering a batch-disposition-adjacent record is high.
Step 4 Place it on the matrix
Influence x consequence gives you a risk band:
Step 5 Map the band onto the other three lenses
- Annex 22 eligibility, first Is the model static + deterministic, or dynamic / probabilistic / GenAI? Under the current Annex 22 draft, GenAI/LLMs and probabilistic models are excluded from critical GMP outright. A high-band GenAI system can’t simply be “validated harder” into a critical role; it must be re-scoped to non-critical with documented oversight, or the model type changed.
- GAMP category Configured product (Cat 4) vs custom/bespoke (Cat 5) sets lifecycle rigor.
- EU AI Act tier Systems used in medicinal-product manufacturing often land high-risk.
Update, Oct 9, 2026: EMA’s Oct 2, 2026 workshop report shows the Annex 22 drafting group has discussed letting generative AI and LLMs into GMP applications, as long as they meet the annex and sit under a fully documented, risk-based control strategy. This is a report, not a decision. The draft text has not changed, and the report notes open concerns about what that control strategy needs.
Update, Oct 17, 2026: This page now adds uncertainty, complexity, and detectability from the same report, on EMA’s June 30, 2026 workshop, in a new section below. The influence x consequence band is unchanged. The report is input to the Annex 22 drafting group, not final guidance.
How do you assess the risk of an AI system in GMP?
Score intended use, model influence, decision consequence, uncertainty, complexity, and whether a failure can be seen in time. Higher uncertainty, higher complexity, and harder detection each call for more validation. If uncertainty cannot be controlled, not using the system can be a valid outcome. The factors come from EMA’s Annex 22 workshop report, which is input to the drafting group, not final guidance. How much each one moves validation is my working rule, set out below.
The band on this page is still influence times consequence. As of the workshop report (EMA/156789/2026, on the June 30, 2026 workshop), three more factors sit beside influence: “Participants also emphasised that uncertainty, complexity, model influence and detectability are important considerations when evaluating AI systems used in GMP-regulated activities” (page 14). Influence is already Step 2, so this section adds the other three.
The report says what happens next: “The information presented during the workshop, including the consolidated responses provided by the participating industry associations, will be carefully reviewed and considered by the Annex 22 drafting group” (page 14).
| Factor | What raises it | What it means for validation effort |
|---|---|---|
| Uncertainty | Something material is still unknown, such as drift or an untested failure mode | Validation gets more formal, with monitoring and a further verification |
| Complexity | Several steps, several tools, or a novel approach make the system hard to follow | More formal controls and deeper validation |
| Detectability | A wrong output can’t be seen before it affects a GMP decision or record | Evidence moves up and needs a control that can actually see the failure. A rushed accept is not that control |
One of the session’s key messages kept the option to stop: “A valid QRM outcome may still be ‘do not use’ where uncertainty cannot be controlled” (page 6). QRM is quality risk management.
Because the report backs these factors without mandating a single risk scoring method, the rule below is my working rule for this page. It is not EMA’s method, and it is not a validated one.
- Start from the band the matrix gives you. If uncertainty is high, complexity is high, or detection is hard, move evidence up one level, from light to standard or from standard to full.
- Two or three unfavorable readings still move it one level. This isn’t a point score.
- A favorable reading adds nothing and takes nothing away, so a high consequence system never drops to light evidence.
- If the band already calls for the full package, stay there. Uncontrollable uncertainty can mean not using the system for that job, which sits outside the evidence levels.
None of this changes Step 5. A generative model is still outside critical GMP under the current Annex 22 draft.
The worksheet
Steal this. One page, filled at Step 5, attached to the validation plan:
- System / pilot name the system’s name
- Question of interest the decision the output supports
- Context of Use (one sentence) the model’s specific role + scope + where the human gate is
- Model type static-deterministic / dynamic / probabilistic / GenAI
- Model influence (L/M/H) + one line of justification
- Decision consequence (L/M/H) + which of patient-safety / product-quality / data-integrity
- Uncertainty (Low/High) + one line; High moves evidence up one level
- Complexity (Low/High) + one line; High moves evidence up one level
- Detectability (Easy/Hard) + one line; Hard moves evidence up one level
- Risk band from the matrix
- Annex 22 eligibility permitted-in-critical / non-critical-only
- GAMP category 3 / 4 / 5
- EU AI Act tier minimal / limited / high-risk
- Evidence rigor that follows light / standard / full credibility package
Detectability is Easy or Hard on the form, not High or Low. Everywhere else on this page High means more risk, and high detectability means less.
Two examples
Say a team at a mid-size biologics company runs two AI systems. Here is each one classified end to end. Same word, “AI”, but opposite regimes.
System A Low to Medium Cold-chain risk classifier
A static, deterministic ML model scores inbound shipments for excursion risk; a logistics analyst reviews the score before any action.
- COU Advisory risk score on shipments, reviewed by an analyst before disposition.
- Influence Low (analyst decides; model informs).
- Consequence Medium (product quality, but human-gated and not irreversible).
- Uncertainty Low (a static model with a defined job; no measured error rate is claimed).
- Complexity Low (one score, one analyst, no tool that writes a record).
- Detectability Easy, assuming the analyst can compare the score with the shipment record before acting.
- Verdict Annex 22: permitted in critical (static + deterministic passes the eligibility gate). GAMP Cat 4. AI Act: limited. Standard validation rigor; no exotic controls.
System B Critical Deviation-investigation agent
A GenAI agent reads deviation reports, retrieves similar past cases (RAG), drafts a root cause, and has a tool to flag severity in the quality system.
- COU Drafts severity + RCA for SME review, but the severity-flag tool means it can write to a regulated record.
- Influence High (it can act on the record, and at volume a reviewer rubber-stamps).
- Consequence High (severity misclassification is a data-integrity and patient-safety issue).
- Uncertainty High (a generated draft can state something the file doesn’t support, and those failures aren’t characterized yet).
- Complexity High (it reads reports, retrieves past cases, drafts a cause, and can write a severity flag).
- Detectability Hard while the write tool and the volume review remain.
- Verdict Annex 22: GenAI is excluded from critical GMP, so this COU is not permitted as designed. It must be re-scoped to non-critical (remove the write-capability, make it draft-only with a hard human gate) to be allowed at all. GAMP Cat 5. AI Act: high-risk. Full credibility/validation package, and a design change before it can ship.
The payoff: the classification revealed that System B’s COU is non-compliant by design. More validation would not have fixed that. You learned it in an afternoon with a worksheet, not in month three of validation, or worse, in an inspection.
Classification sets the evidence burden
The point of classifying is that the rigor follows from the band:
| Band | Evidence that follows |
|---|---|
| Low | Light evidence, standard CSV |
| Low to Medium (System A) | Standard validation rigor; no exotic controls |
| Critical (System B) | The full package. For an LLM/agent, the security controls double as validation evidence |
At the critical band those controls are:
- identity and access at the gateway
- input/output guardrails
- a human-approval gate on high-risk tools
- red-team results read as OQ evidence
- a tamper-evident audit trail
My working rule from the risk section adds one step on top of the band table.
The Context of Use is the scaffold all of that hangs on: it tells you which controls you owe and how much proof each one needs.
That’s the bridge to the rest of this work. What a CSV validator should look for in an LLM gateway is the control layer; security is now a GMP requirement is why those controls are now validation deliverables. Classification is the step that comes before both.
FAQ
Is the EMA workshop report a GMP requirement?
No. The June 30, 2026 workshop collected industry input for the Annex 22 drafting group. It is not Annex 22, and it does not change the current draft. Until the draft text changes, its limits on generative AI in critical GMP still stand. Name the report as a workshop input.
Do uncertainty, complexity, and detectability change the risk band?
On this page they change the evidence, and the band stays model influence times decision consequence. Low uncertainty, low complexity, and easy detection do not lower it. High uncertainty, high complexity, or hard detection moves evidence up one level, That step is my working rule. The report does not require one scoring method.
If a person reviews every output, is a bad output easy to detect?
Only when the person can see a bad output before it matters. Comparing the model answer to the source record, with time and authority to reject it, is real detection. Accepting a large pile of outputs in a hurry is not. Influence is who decides. Detectability is whether a wrong output can be seen.
Does high uncertainty mean the system cannot be used?
Not by itself. The industry presentation at the workshop argued that uncertainty should lead to more knowledge, more formal validation, monitoring, and repeat checks instead of an automatic ban. Not using the system can be a valid quality risk management outcome when the uncertainty cannot be controlled for that intended use.
Does this replace Context of Use and the influence score?
No. Context of Use is still the first step, and influence times consequence is still the band. The new section adds uncertainty, complexity, and detectability, and says how each one changes the amount of validation. The five steps of the procedure stay as they were.
Open questions
- Agents that call tools. Context of Use is clean for a static model with one output. For an agent, each tool is its own little COU: “send an email” and “flag deviation severity” can’t share one classification. The useful framing is per-tool COU: score influence x consequence for each action, and let the highest-risk tool set the controls. If your “system” is an agent, classify its tools, not just the agent.
- The critical GMP line. It is still undefined even after the Annex 22 consultation, so the gate everything hinges on remains fuzzy.
- Not covered here. Reclassification on drift (a model that degrades can cross a band), supplier-provided model qualification, and the full GAMP lifecycle. For those, start with the FDA AI credibility guidance, EU GMP Annex 22, GAMP 5 (2nd ed.), the NIST AI RMF, and the EU AI Act high-risk provisions.
Related reading: list your AI tools and risk tiers first.
I’m a pharma-systems consultant who helps CSV/QA and AI-platform teams get GenAI and agents into GxP workflows defensibly, starting with exactly this classification step. If you’re staring at a pilot the business wants in production and no one’s agreed how much validation it owes, get in touch. You can also get help sizing validation for your AI tool.