← All writing

AI agent audits · Audit trails · Regulated AI

How to audit AI agents: a practical playbook for regulated firms

Sooner or later someone hands you an agent system you did not build and asks whether it can be trusted. Here is the playbook, step by step, with the tests that separate governance from governance theatre.

The request usually arrives sideways. Risk finds out that operations has had an agent triaging cases for three months. An acquirer's deal team discovers the target's "AI platform" the week before exclusivity ends. Or a board, reading about the EU AI Act's fines, asks compliance a question nobody has an answer to: what are our agents actually doing, and could we prove it?

I have sat on the other side of this. I took agentic AI into production at a regulated trading venue, under my own technical authority, alongside an ISO 27001 programme - which means I have had to produce the evidence an auditor asks for, and I know which questions are easy to wave away and which ones are not. This piece is the second half of a pair: the observability article argues why the trace is the load-bearing artefact. This one is for the person holding the clipboard - internal audit, a risk function, or a deal team - who has to assess an agent system someone else built.

The timing is not academic. The EU AI Act's obligations for high-risk systems have been enforceable since 2 August 2026, with tamper-evident logging retained for at least six months under Article 12 and effective human oversight under Article 14. The FCA is running AI Live Testing with UK firms right now. And in US markets, firms are already being warned that a missing trace reads as a books-and-records problem, not a technology quirk. The audit is coming either way; better to run it yourself first.

Step 1: Inventory - find every agent, including the ones nobody registered

You cannot audit what you have not found, and you will not find everything by asking the platform team. Gartner published guidance in April 2026 on managing agent sprawl precisely because teams deploy agents with no central registration, no named owner and no monitoring. Shadow IT surfaced bad answers; a shadow agent holds credentials and takes actions.

For every agent, record four things: what it does, what credentials and API scopes it holds, who owns it, and what happens if it is switched off. The credential question is the one that bites. An agent with a service account inherited from a decommissioned integration is a finding on its own, before you look at anything it decides.

Step 2: The action space - classify by reversibility, not importance

List every action each agent can take, then sort the list by a single criterion: can this be undone? Reading a queue is reversible. Updating a record is usually reversible. Sending money, deleting data, or communicating with a customer is not.

Then check two things. First, permission scoping: does the agent hold only the permissions its action list requires, or does it run with something broad because that was easier at the time? Second, human checkpoints: every irreversible action should sit behind an explicit approval, and the approval should be recorded in the same trail as the action. A checkpoint that exists in a policy document but not in the execution path is the most common gap I see described, and the easiest to test for.

Step 3: The trace - run the reconstruction test

This is the centre of the audit, and it takes one meeting. Pick a production run from last month - not last night, last month - and ask the team to reconstruct it in front of you: what the agent was asked, what it reasoned at each step, every tool call with its full inputs and outputs, what came back, which model and version produced each decision, and how the run ended.

A team with a real audit trail does this in minutes, and the record is readable by someone outside engineering. A team without one starts assembling fragments from application logs, dashboards and memory. Both outcomes are informative. Only one of them is acceptable in a regulated setting, because "the model decided" is not an answer a regulator, an auditor or a risk committee will take - it is a non-answer from people with the authority to stop your deployment.

The follow-up question matters as much: is the reasoning captured for every consequential action, or only when the model felt like explaining itself? Models skip the explanation step routinely; if rationale is not enforced - as a required field, or via captured extended thinking - coverage will be far lower than anyone in the room believes.

Step 4: Tamper evidence - who can edit the log?

A trail that the writing process can rewrite is a debug log, not evidence. Article 12 of the EU AI Act asks for tamper-evident logging for a reason: after an incident, the record is only worth what it can withstand.

Three questions do the work. Are records chained - hashed with the hash of the record before them - so an edit or deletion breaks the chain? Is the log anchored somewhere the writing process cannot reach, such as an append-only store or a WORM bucket, rather than sitting in the same place it protects? And does anyone actually run the verifier - because a tamper-evident log nobody verifies provides exactly as much assurance as no log at all. The FINOS AI Governance Framework's mitigation on agent decision audit and explainability is a useful public reference point here if you need one to hold a vendor against.

Step 5: Evaluation - is anyone checking whether it is any good?

Observability tells you what the agent did; evaluation tells you whether it should have. Ask to see the evaluation suite, then ask three questions of it. Does it run continuously against sampled production traffic, or only before releases? When an incident happened, did the failing case become a permanent regression test? And is anyone watching the scores over time for drift, or is the suite a one-off artefact from the original build?

A team that cannot show you an evaluation suite is trusting the demo. A team whose suite has not changed since launch is trusting the past.

Step 6: Security - map it against the OWASP agentic Top 10

Agent security now has a public baseline: the OWASP Top 10 for Agentic Applications, published in December 2025. Use it as the checklist rather than inventing your own. The items that most often produce findings: prompt injection paths - every place untrusted content (emails, documents, web pages, tickets) enters the context of an agent that holds credentials; tool misuse - whether a compromised or confused agent can chain legitimate tools into an illegitimate outcome; and credential exposure - secrets in prompts, over-scoped tokens, and agents that can read the configuration that governs them.

Fewer than half of organisations run regular security testing on their agent workflows, by the surveys circulating this year. If the team cannot show you when the agent's toolset was last tested against injected instructions, that is the finding.

Step 7: The paperwork - does the governance match the system?

Finally, put the governance documents next to the system and check they describe the same thing. The most common failure mode in this space is not a missing policy - it is a handsome governance deck with nothing behind it: an oversight committee that has never seen a trace, a human-in-the-loop diagram with no approval step in the code, a retention policy for logs that are not actually retained.

This is also the test I would run as an acquirer, because it converts directly into deal language. Audit trails cannot be meaningfully retrofitted - the execution model has to generate structured evidence at each decision step - so a target whose agents run without them is not carrying a small remediation item. It is carrying a rebuild, and the price should say so.

What good looks like

When the system is right, the audit is boring. The inventory exists and matches reality. Irreversible actions sit behind recorded approvals. The reconstruction test takes ten minutes and produces a record a non-engineer can read. The log verifies. The evaluation suite has grown since launch because incidents fed it. The OWASP mapping has an owner and a date. The paperwork describes the system you just inspected.

None of that requires exotic tooling. It requires that the trail was built in from the first sprint, treated as an audit artefact rather than a debug log, and owned by someone. Which is why the audit doubles as a due diligence instrument: the same seven steps that satisfy a risk committee will tell an investor, faster than any management presentation, whether a company's agents are an asset or a liability. If you are assessing a target's AI claims more broadly, the companion pieces cover the full AI due diligence checklist and how to tell whether the AI is real or a wrapper.

Sources

I run AI agent security reviews and build agentic AI with audit trails regulators accept through Trustflux Ltd, alongside technology and AI due diligence for investors. The open-source practice tooling - md-to-jira, azure-cost-investigator and spec-check - is read-only by architectural guarantee, so you can inspect the standard before you engage it.

Written with Claude Code. The playbook is the artefact; the article is the receipt.