An ALCOA+ audit trail for LLM calls

Feel free to download the python code to play around with this yourself. It’s a small set of python files with a minimal streamlit app to help visualize the topic.

When a regulated GenAI system meets an inspector, three questions decide whether a model interaction is a record or just a log line:

1. Who did this, and on whose behalf?
2. What exactly happened — prompt, model, parameters, output?
3. Prove it hasn’t changed since.

This example shows which concrete field or control satisfies each ALCOA+ attribute — with the integrity claim made cryptographic (a hash chain).

Why do we care? - GenAI in pharma quality & manufacturing

For common use cases like an LLM that drafts or reviews a batch record, a deviation, a CAPA, or an SOP — the model’s output now feeds a record a QA lead signs off on, and an inspector can demand its full lineage years later. “The model summarized this deviation” is not enough: you have to prove who invoked it, what prompt and model version ran, and that the stored interaction hasn’t been edited since.

That is the same shape wherever an LLM sits on the path to a regulated decision — pharmacovigilance, clinical study report drafting, regulatory-submission QC.

The moment an AI output becomes part of a record with a retention obligation and a named human owner, a flat log file stops being evidence. This is an easily digestible example of the control that makes it evidence again.

Why this matters to you as a GenAI system dev or System Owner?

A complete audit trail turns an inspection finding into a non-event. The cost of not having one is a data-integrity 483 plus a full remediation cycle.

The ALCOA+ to field mapping (the point of this post)

ALCOA+	For an LLM call, means…	Concrete field / control
Attributable	who triggered it, on whose behalf	actor.principal + actor.agent + actor.auth — never anonymous
Legible	permanently human- & machine-readable	structured JSON, stable key names
Contemporaneous	recorded at the moment	timestamp_utc, set server-side at the call boundary
Original	first verbatim capture preserved	interaction.request/response verbatim, committed into record_hash
Accurate	matches what actually ran	exact model, params (temp 0), prompt_sha256, usage
Complete	nothing silently dropped	outcome.status; errors, retries, human overrides recorded too
Consistent	same shape & order	schema_version + monotonic seq + fail-closed validation
Enduring	durable & tamper-evident	append-only store + hash chain (prev_hash -> record_hash)
Available	retrievable for retention	retention block + export + reviewer view

The integrity rule, in one line

In our little example project provided - Edit, insert, reorder, or delete any record after the fact and verify.py recomputes the hashes, finds the break, and names the exact seq. That is the cryptographic proof of Original and Enduring that a flat log file can never give you.

record_hash = sha256( canonical_json( record_without_record_hash ) )
prev_hash[0] = GENESIS;   prev_hash[n] = record_hash[n-1]

See it run

The whole thing is standard-library Python with no API keys — a deterministic mock provider stands in for the model, so it’s clone-and-run. One command builds a trail, prints it, verifies it, and then tampers with it:

python demo.py

Three LLM calls just happened, and each became one record:

• seq 0 — a normal success.
• seq 1 — a call that failed upstream, and notice it was still recorded and then re-raised. A failed call must never be a missing record — that is ALCOA+ Complete.
• seq 2 — a success that a human reviewer overrode: the model said one thing, a QA lead corrected it, and that correction is itself an auditable event.

Each record’s prev is the previous record’s hash. Record 0’s prev is all zeros — the genesis link.

One record, the mapping made concrete

Go ahead and pull the first record out of the trail and the abstract table above turns into a single JSON object:

python -c "import json; print(json.dumps(json.loads(open('audit-trail.jsonl').readline()), indent=2))"

Every ALCOA+ attribute is a concrete field: Attributable → actor (who, which agent, how they authenticated — never anonymous); Contemporaneous → timestamp_utc, set server-side at the call boundary, not whatever the client claims; Original / Accurate → the verbatim request and response, the exact model, temperature 0, a hash of the prompt, token usage; Enduring → prev_hash and record_hash, the chain.

A validator reads this and can answer “who did what, when, with which model” without trusting me.

Log Tamper detection

Anyone can keep a JSON log. The question an inspector actually asks is prove it wasn’t changed. The demo copies the trail, edits record 0’s response text — exactly what someone “fixing” a log file would do — without recomputing its hash, then re-verifies both. The exit codes are real, which is what makes this a CI gate:

python verify.py audit-trail.jsonl          # PASS -> exit 0
python verify.py audit-trail.tampered.jsonl # FAIL -> exit 1, names seq 0, record_hash mismatch

The interactive viewer makes it visual: in the ⛓️ The chain tab, tamper with a record and watch that block and everything downstream turn red as the banner flips to FAIL.

Because each record commits the hash of the one before it, you can’t quietly change history — editing one record breaks every link after it. That is the cryptographic proof of Original and Enduring a flat log can never give you.

Why this is validation evidence, not just security

Security people see a tamper-evident log. An ITQ team sees something more specific:

• the exit code drops straight into a CI gate — a build can fail if the audit trail’s integrity is broken. That is an OQ-style control.
• the reviewer view (python demo.py --review) is the human-readable export an auditor reads — ALCOA+ Available.
• the schema fails closed — a malformed record is rejected, never stored — so you can’t end up with an “Accurate/Complete” trail full of garbage.

The same artifact is a security control and a piece of validation evidence. That overlap is the whole position. And the test names are the spec: tampered content is detected, a deleted record breaks the chain, the schema fails closed, the hash is canonical-order-independent.

pytest -q

What this is — and isn’t

• It is: the minimal, vendor-neutral teaching version. The artifact is understanding.
• It is: A small streamlit app created to illustrate the concepts covered
• It isn’t: a product. No DB server, no cloud, no auth system, no scale.