When a GxP AI tool gets it wrong

Regulatory status checked October 19, 2026. Annex 22 and the revised Annex 11 are still drafts, and so is the US Food and Drug Administration (FDA) guidance on AI for regulatory decisions. The incident and deviation rules I rely on are final and in effect today. Where a section leans on a draft, I say so there. The working rules are mine.

A wrong AI output in a good manufacturing practice (GMP) workflow is a quality event. It is a deviation when it leaves a wrong record or a decision the procedure did not allow. Stop that use, keep the prompt, model version, inputs, outputs and logs, assess the impact, find the cause, and release any fix through change control.

This note walks that sequence for the quality assurance (QA), CSV (computerized system validation) or AI platform lead who already runs a large language model (LLM) or an agent in a GMP process. Throughout, the tool proposes and a trained, authorized person decides and is accountable for the record. It ends with one case carried through every step, a severity rating that came out too low.

What counts as an AI incident

None of the sources I checked defines “AI incident” as a term, and you don’t need one. EU GMP Annex 11, the computerized systems annex in effect since 2011, already covers it in section 13:

“All incidents, not only system failures and data errors, should be reported and assessed. The root cause of a critical incident should be identified and should form the basis of corrective and preventive actions.”

My working definition is this. An AI incident is a model output, or a failure to produce one, that could affect a GMP decision, a GMP record, or the control that is supposed to protect either. A guardrail that fires and stops the output is an incident even when the bad text never lands anywhere. A wrong answer in a sandbox that cannot touch a record is a test result, and if that test was part of validation, it still gets written down.

When that incident is a deviation

A deviation is the GMP word you already use. EU GMP Chapter 1, section 1.8(vii) asks that “Any significant deviations are fully recorded, investigated with the objective of determining the root cause and appropriate corrective and preventive action implemented”. In the United States, 21 CFR 211.100(b) says “Any deviation from the written procedures shall be recorded and justified.”

“Significant” is the regulation’s word, and not every model error is significant. This is where I draw the line, as my working rule.

If the wrong output let a defective batch reach the market, or a complaint comes in, the complaint and recall rules in EU GMP Chapter 8 apply on top of the deviation.

How you notice a bad output

A bad output reaches you by one of three routes. A guardrail catches it, a person reviewing the output catches it, or monitoring finds it after the fact. The workshop’s reliability session put it plainly: “Controls should be layered across prevention, detection and containment” (workshop report, section 1.4, page 10). The report also notes that controls built for known failures don’t cover the ones nobody anticipated, which is why monitoring after the fact still has a job.

A person only counts as detection when they can see a wrong output before it matters. A reviewer who rubber-stamps a severity score is not a control. My note on how much validation your AI needs covers how detectability changes the controls you need, and my note on LLM gateways covers what a gateway has to stop when a tool can write a severity flag.

The draft Annex 22 expects performance and input monitoring for the models it covers (sections 10.3 and 10.4). Those sections don’t govern an LLM today.

What you do when a guardrail fails or flags uncertainty

Stop that use, hold any outputs nobody has acted on, and send the case to a trained, authorized person who can reject the output. The drafting group asked this question itself, and the report does not answer it with a mechanism.

"When guardrails fail or detect uncertainty, what mechanisms are required to ensure timely escalation and prevention of GMP impact?"

The session supported documented escalation paths, and the report says Annex 22 may need to define terms such as escalation, fallback, safe state and monitoring threshold (page 10). Until the annex says more, I work through these steps in order.

1. Stop that use. Don’t leave it running “while we look.”
2. If the guardrail fired, believe the stop. A low-confidence or “undecided” flag stops this workflow until a trained, authorized person deals with the case. Section 9.2 of the draft Annex 22 has a related idea, written for models that predict or classify. Applying it to an LLM is my call.
3. Hold any outputs nobody has acted on yet. If a person already acted on one, start the impact assessment the same day.
4. Escalate to the process owner and the quality unit. Chapter 1 and 21 CFR 211.22(a) already give the quality unit the authority to make sure errors “have been fully investigated.”

The workshop report says “Guardrails may reduce risk but do not eliminate the need for risk-proportionate human oversight or accountability” (page 7). A guardrail failure is when that oversight has to be real.

What evidence you keep

Keep the prompt, the model name and version, the inputs including any retrieved sources, the output, the time, whether a guardrail fired, and what the person did next. If any of those is missing, write the gap into the investigation.

The full field list is in An ALCOA+ audit trail for LLM calls. The current rule that matches it is Annex 11 (2011), section 9, which says “Consideration should be given, based on a risk assessment” to building in an audit trail of GMP-relevant changes and deletions. Where there is one, the record has to be readable later by someone who wasn’t in the room. Annex 11 says “convertible to a generally intelligible form.”

How you assess impact and find the cause

Impact starts with the records that changed and the decisions made on the output. Then ask whether other uses of the same model, prompt and data could fail the same way. If a batch record or a specification is involved, 21 CFR 211.192 says “The investigation shall extend to other batches of the same drug product and other drug products that may have been associated with the specific failure or discrepancy.”

For cause, look at the model, the data, the prompt and the process, and include the human review in the process. “The model hallucinated” describes the symptom, and the root cause sits behind it. A cause sounds like “the repeat-batch rule was not in the prompt” or “the reviewer accepted the score without checking it against the record.” Chapter 1 is direct about the easy answer (section 1.4(xiv), page 4):

“Where human error is suspected or identified as the cause, this should be justified having taken care to ensure that process, procedural or system-based errors or problems have not been overlooked, if present.”

How deep you go follows the risk. ICH Q10, section 3.2.2 says the effort, formality and documentation of the investigation “should be commensurate with the level of risk, in line with ICH Q9.”

How a fix goes back into use

A fix goes back into use through change control, and a prompt edit or a new model version counts as a change just like any other. The corrective action should address the cause you named. A retrieval change or a new guardrail threshold goes through the same route. Annex 11 section 10, still current, says changes to a computerized system “should only be made in a controlled manner in accordance with a defined procedure.” The draft Annex 22, section 10.1 puts it this way.

“Any change to the model itself, the system, or the process in which it is used, including any change to physical objects the model is using as input, should be documented and evaluated to determine if the model needs to be retested. Any decision not to conduct such retest should be fully justified.”

The FDA draft guidance of January 2025, still marked “Not for implementation,” says in section IV.B: “For example, in pharmaceutical manufacturing, it is important that changes to the AI model or changes in manufacturing that may impact the performance of the AI model be evaluated by the manufacturer’s change management system within their pharmaceutical quality system.”

The tool doesn’t go back into the GMP workflow until that change is approved. Re-release means that approval, recorded. After that, Chapter 1 and ICH Q10 expect you to monitor whether the action worked, and the revised draft Annex 11 would have periodic review follow up “Actions from incidents, problems and deviations” (draft Annex 11, section 14.2(vi), page 12). For what a model change or drift triggers more broadly, see AI validation in GxP doesn’t have a settled playbook yet, and for who owns a routing rule when the model version changes, see The GenAI routing layer.

Example of a severity rating that was too low

Say a tool proposes a severity

A tool reads a deviation record and proposes a severity. The written procedure says the severity is major when the same failure has happened on another batch in the last 90 days. The record in front of the tool includes that earlier batch. The tool returns minor.

If the reviewer catches it

The reviewer compares the proposal with the record, rejects minor and sets major. That is an incident, because the tool was wrong. It is not a deviation of that record, because the recorded severity is the one the procedure requires and a person set it. The incident still asks whether the next wrong call would be caught. If the honest answer is no, the control is the problem.

If nobody catches it

The tool writes minor straight into the record, and that direct write is itself the control gap. Or the reviewer accepts minor, and the investigation is scoped as minor. That is a deviation, because the wrong severity is now the record. Stop the scoring use and keep that call.

The impact assessment covers every deviation the same tool scored while the fault was in place, including the ones nobody noticed.

Pick the cause from the evidence. The repeat-batch rule may have been missing from the prompt, the earlier batch may never have reached the input, or the reviewer may have accepted the score without checking. Corrective action and change control come before the tool touches a live deviation again.

How to classify a severity tool in the first place is covered in the GxP-use rows of Can you use ChatGPT, Claude or Copilot for GxP work? My reading is that a severity rating that can narrow an investigation is not a safe place to assume “non-critical.”

FAQ

Is every wrong AI output a deviation?

Not every wrong output is a deviation. I call it an incident when it could affect a GMP decision or record. It is a deviation when that output is in the record, someone acted on it, or a procedure was not followed. A rejected draft that never gets recorded is still an incident. Annex 11 says to report and assess incidents. Chapter 1 turns a significant deviation into an investigation. The boundary is my working rule.

Does the EMA workshop report require an AI incident procedure?

No. The June 30 and July 1, 2026 workshop collected input for the Annex 22 drafting group. The report says that input will be reviewed and used to inform the revision. It is not Annex 22, and it does not require an incident procedure. The drafting group asked how to escalate when guardrails fail. Use the report as input. The current draft still says generative models should not be used in critical GMP applications.

If a person accepts the low severity, whose deviation is it?

The process owner and the quality unit own it. A person who accepts a low severity without checking it against the deviation record has let the model decide. The wrong severity in the quality record is the deviation. Look at the model, the prompt, the data, and the review step. Do not close the record as human error unless you have shown that the process and the system were not the cause. Chapter 1 asks for that justification.