Pass, Fail, and the Two States Your AI Compliance Report Is Missing
A control can be met, breached, ambiguous, or impossible to judge from the evidence you hold. Two-state reporting collapses the last two into a pass.
Who this guide is for
Risk, audit and AI platform leads
TL;DR — Before asking whether a control was met, a governance system has to ask whether it holds enough evidence to answer at all. Systems that skip the first question answer the second one anyway, and the answer they give is 'pass'.
Almost every compliance dashboard reports in two states. A control is green or it is red. That model is inherited from infrastructure compliance, where it works well: encryption at rest is either configured or not, and the system checking it can see the configuration directly.
AI controls break the model, because a large share of them are about behaviour rather than configuration — and behaviour is only visible through telemetry that varies enormously in what it captures.
The question underneath the question
Take a concrete control: no customer personal data is sent to an external model provider. To evaluate it, a system needs to see the payload of each call and the destination it went to.
Whether it can see those things depends on how the AI was instrumented. One integration emits full prompt content; another emits token counts and a model name. On the first, the control is genuinely evaluable. On the second it is not — and a two-state system evaluating it there has only two moves. Report red, and generate a false alarm about a violation nobody observed. Or report green, and assert compliance from an absence of evidence.
In practice the second move wins, because a red that turns out to be spurious gets escalated and tuned away, while a green nobody questions stays green. That is how a compliance report drifts into asserting things it cannot demonstrate.
Four states
| State | Meaning | Action |
|---|---|---|
| Compliant | The evidence was sufficient, and it shows the control was met. | Nothing. Cite it in the pack. |
| Violated | The evidence was sufficient, and it shows the control was breached. | Remediate the behaviour. |
| Needs review | The evidence was sufficient, and the outcome is genuinely ambiguous. | A human decides. This is a judgement, not a defect. |
| Unverifiable | The evidence was not sufficient to judge either way. | Fix the instrumentation, not the AI. |
The last two rows are the ones two-state systems collapse, and they are different problems with different owners. Needs-review is a decision waiting for a person. Unverifiable is a coverage gap — the AI may be behaving perfectly, and you cannot show it.
Separating them changes what a team does on a Monday morning. A report showing forty violations sends engineers to inspect forty behaviours. The same report split into eight violations and thirty-two unverifiable sends eight to engineering and one instrumentation task to the platform team — and the second reading is the accurate one.
Evidence sufficiency, stated plainly
The mechanism behind the four states is a comparison. Each control is decomposed into the evidence types required to evaluate it. Each integration is profiled for what it can actually observe. Before evaluating, the system checks whether what it has covers what the control needs.
control: "no personal data to an external provider"
requires: payload_visibility, destination_visibility, audit_persistence
integration A observes: payload_visibility, destination_visibility,
audit_persistence, model_version
-> sufficient -> evaluate -> compliant | violated | needs_review
integration B observes: model_version, token_counts, latency
-> insufficient -> unverifiable
missing: payload_visibility, destination_visibilityThe valuable part of an unverifiable verdict is the missing list. 'We could not evaluate this' is a shrug. 'We could not evaluate this because payload visibility is absent on this integration' is a work item with an owner and a definition of done.
Why a reviewer trusts it more
The intuition runs backwards here, so it is worth stating directly: reporting that you could not verify a control makes the rest of the report more credible, not less.
- It demonstrates the assessment had a way to fail. An instrument that cannot return a negative result is not measuring anything.
- It tells the reviewer where to sample. They were going to look for the soft spots regardless; a report that names them is a report that read like it was written by someone who went looking.
- It disclosed the gap rather than leaving it to be discovered. The same fact lands very differently depending on who says it first.
- It survives being wrong. A control that passed on absent evidence and later turns out to have been breached calls every other green into question.
An all-green report is the single most common reason a reviewer stops trusting a pack. Nobody is fully compliant on a first assessment, so a document claiming it reads as one that was not really looking.
What to ask a governance tool
Show me a control this system cannot evaluate on my setup. If the answer is that there are none, ask how it evaluates payload-level controls on an integration that does not emit payloads.
What does it report when telemetry is missing — and is that visually distinct from a pass in the exported artefact, not merely in the dashboard?
Does an unverifiable result name the specific evidence that was missing, or only that something was?
Can a control move from unverifiable to compliant by improving instrumentation alone, without changing the AI's behaviour? If not, the two problems are still conflated somewhere.
Where Dobby fits
This is the design decision the rest of Dobby follows from. Every control evaluation returns one of four states — compliant, violated, needs-review, unverifiable — and the evidence used and the evidence missing are both recorded alongside the verdict. Where the telemetry cannot demonstrate a control, the pack says so and names what would be needed.
Dobby connects to your AI activity out-of-band — reading run telemetry from CrewAI, LangChain, OpenAI, Google ADK or a custom SDK — and never sits in the request path. Because instrumentation varies by integration, evidence sufficiency is computed per run rather than assumed once at setup: the same control can be evaluable on one workload and unverifiable on another in the same organisation, and the evidence pack reports each honestly.
A compliance system's most important output is not the list of things that passed. It is the list of things it knows it could not check.