What Belongs in an AI Evidence Pack
A reviewer asked you to prove what your AI did. Here are the six parts an evidence pack needs, and the three things that get one rejected.
Who this guide is for
AI vendors preparing for audit, procurement or vendor-risk review
TL;DR — The deliverable is a self-contained artefact somebody can read without an account, forward to a colleague, and verify has not been altered. A dashboard login is not an evidence pack, because the reviewer's job is to attach something to a file.
There is a moment in most enterprise AI sales cycles where the conversation stops being about the product. A reviewer — vendor risk, internal audit, a regulator's examiner, or a customer's compliance lead — asks you to demonstrate that your AI behaved the way your documentation says it does.
What they want is a package. Something with a date on it, a defined scope, and a structure they recognise from every other control review they have run. Most AI vendors answer with a thread of email attachments assembled under time pressure, and the assembly is the part that takes weeks.
The six parts
1. Executive summary
One page, for somebody who will not read the rest. Scope — which systems, which period, which frameworks. Overall posture stated plainly. The number of controls assessed and how they came out. And the honest headline: what this pack does and does not cover. A summary that opens with an unqualified reassurance gets the rest of the document read with suspicion.
2. Control matrix
The core of the pack: every control in the framework, its status, and the evidence supporting that status. This is the structure a reviewer already thinks in, and matching it is most of what makes a pack easy to accept. One row per control, mapped to the framework's own identifier — EU AI Act Article 12, DORA's ICT third-party provisions, a SOC 2 criterion — because a reviewer testing a specific requirement should not have to translate your vocabulary into theirs.
3. Gap report, ranked by severity
What is not compliant, how bad it is, and what would fix it. Including the gaps is not a weakness — a pack with no gaps reads as a pack that was not looking. The ranking matters more than the completeness: a reviewer needs to know which three items to raise, not to be handed forty findings of undifferentiated importance.
Remediation guidance belongs next to each gap. It converts the pack from a verdict into a work plan, and it demonstrates that you understood the finding rather than merely detected it.
4. Findings — the underlying evidence
The layer beneath the matrix: the actual observations that produced each status. Which runs were examined, what was found, on what dates. A reviewer who accepts your matrix on trust is doing you no favour, because their own reviewer will not. Findings are what let them sample.
5. Framework coverage
Which requirements were assessed and which were out of scope, stated up front rather than inferred from absence. A pack covering nine of a framework's fourteen requirements is useful. A pack that silently covers nine and presents itself as covering the framework is a problem, and it is the kind of problem that ends a review badly.
6. Integrity manifest
A cryptographic hash — SHA-256 — over the pack's contents, so the recipient can verify that what they are reading is what you generated. This is the cheapest part to produce and the one that changes the character of the document most. It moves the pack from an assertion into something checkable, and it protects you as much as them: a pack that can be verified cannot be quietly edited by anyone downstream and attributed to you.
Three things that get a pack rejected
| What the pack does | What the reviewer concludes |
|---|---|
| Every control passes | It was not really assessed — nobody is fully compliant on a first pass |
| Controls pass with no evidence cited | The status is an assertion; I now have to test everything myself |
| Scope is vague or undated | I cannot tell what period this covers, so I cannot rely on it |
| It is a login to a dashboard | I cannot attach this to my file or send it to my committee |
| Numbers appear with no method | If this figure is unsourced, which others are? |
The first row is the counter-intuitive one and it is the most common. An all-green pack is read as evidence of a weak assessment rather than a strong control environment. Reviewers spend their working lives finding gaps; a document claiming there are none tells them the search was not serious.
The state most packs are missing
Compliance tooling generally reports two outcomes: pass or fail. That is sufficient for controls you can observe directly, and it misrepresents a large class of AI controls where the honest answer is that the available telemetry cannot demonstrate the control either way.
Collapsing that third situation into 'pass' produces false compliance — the failure mode that damages a pack's credibility permanently, because it is discovered by the reviewer rather than disclosed by you. Collapsing it into 'fail' is more honest but sends teams chasing defects that do not exist.
A pack that says 'we could not verify this control from the available evidence, and here is what telemetry would be needed' is stronger than one that quietly passes it. It tells the reviewer you know the difference between what you proved and what you assumed.
Generated, not assembled
The last structural point is about how the pack comes into existence. A hand-assembled pack is accurate on the day it is finished and decays from there — and the next review, three months later, starts from scratch.
- Generated from the run records themselves, so the pack reflects what happened rather than what somebody remembered.
- Reproducible, so the same period regenerates to the same result and a disputed finding can be re-examined.
- Scoped by date, so the pack can be regenerated for the window a reviewer actually asks about.
- Repeatable at low cost, because the second bank asks most of what the first one asked.
Where Dobby fits
Producing this artefact is what Dobby's Fintech AI Evidence module does. It connects to your AI activity out-of-band — reading run telemetry from CrewAI, LangChain, OpenAI, Google ADK or a custom SDK, never sitting in the request path — scans each run against EU AI Act, DORA and SOC 2, and exports a pack containing exactly the six parts above.
The per-control verdict has four states rather than two: compliant, violated, needs-review, and unverifiable. That fourth state is the design decision the rest of the product follows from — when the evidence is not sufficient to prove a control, the pack says so.
Non-custodial by design: for the Fintech AI Evidence module, raw prompt and response content is routed to a dataset you own rather than held by Dobby. Run metadata and verdicts are processed in Dobby's managed control plane.