Change Management for AI: Proving What Was Live

Somebody tweaks a system prompt on a Thursday afternoon. Adds a tool. Nudges the temperature. The change is small, it works, and nobody writes it down because it was not a deploy.

Four months later a reviewer asks which version of that system served a customer decision in March. The honest answer is usually that nobody knows — and "nobody knows" is a change-management finding, not a documentation gap.

The question, as it is actually asked

Change management appears in every third-party risk questionnaire, and for conventional software the answers are routine: releases go through CI, approvals live in the pull request, rollback is a documented procedure. Teams answer it from muscle memory.

AI breaks the muscle memory in one specific place. A prompt change alters behaviour as much as a code change and frequently ships outside the release process entirely — through an admin UI, a config table, or someone with the right permissions and a good reason. The governance the organisation built for code does not reach it.

A configuration you can edit without producing a version is a configuration you cannot testify about.

Why a git repo does not close it

The natural instinct is to keep agent configuration in git and sync it on deploy. That genuinely helps, and it stops being sufficient the moment anyone can change behaviour through the product itself — which is usually the point of having a product.

What the obligation actually needs is versioning in the system of record: every edit writes a new row carrying the full configuration, the identity of whoever made it, and a timestamp. Nothing is overwritten. The current state is a query, not a file.

-- append-only: every edit is a new row, nothing is updated in place
SELECT *
FROM agent_versions
WHERE agent_id = @agent_id
  AND created_at <= @decision_time
ORDER BY version DESC
LIMIT 1;
-- "which configuration served this decision" becomes a point-in-time query

Three properties fall out of that shape. History is never lost, because nothing is deleted. Any past moment is reconstructable, because the query takes a timestamp. And rollback is an insert rather than a restore — you write the old configuration forward as a new version, which keeps the audit trail honest about the fact that a rollback happened.

Diff by field, not by blob

Storage format is half of it; the other half is whether a human can read the change. A blob diff of version 41 against 42 shows a wall of JSON in which a temperature change and a rewritten system prompt look identical in weight.

Grouping the diff by field type — metadata, behaviour, tools, guardrails — is what makes a version history reviewable rather than merely stored. A reviewer scanning a year of changes needs to see which ones touched behaviour, and they need to see it without reading every line.

Rollback is also the incident answer

Rollback should be one action against a version you can point at. That is worth building for operational reasons, and it is worth building for a second reason that only shows up later: during a bad hour you do not debug a live agent, you put back the configuration that was working and investigate afterwards.

Under DORA, that sequence is part of what you tell a financial customer during an incident — what changed, when you reverted it, and what the system did in between. The version history is what turns that account from a recollection into a record.

The field that makes it evidence

Versioning the configuration is necessary and not sufficient. The link that turns it into evidence is the version identifier carried on each run, so a decision points at the exact configuration that produced it.

Without that link you have two separate artefacts — a history of changes and a history of runs — and joining them by timestamp is an inference. Inference is exactly what a reviewer is trying to eliminate. With the link, "which version served this decision" stops being an investigation.

• The version identifier on every run, not only on the change record.
• Who made each change, from an identity system rather than a free-text field.
• What changed, grouped so a reader can tell behaviour changes from cosmetic ones.
• Retention that outlives the period a reviewer will ask about — which is longer than your debug logs.
• Rollbacks recorded as changes in their own right, because a reverted change is still a change that happened.

Where Dobby fits

Dobby reads run telemetry out-of-band — CrewAI, LangChain, OpenAI, Google ADK or a custom SDK — and never sits in the request path. Where the telemetry carries a version identifier, it lands on the run, and change-management controls can be evaluated against the frameworks you have activated.

Where it does not, the per-control verdict returns unverifiable rather than compliant, and the gap report names the missing field. That is the honest outcome for this control in particular: whether a version can be tied to a decision is a property of your instrumentation, and no scanner can infer it from telemetry that never carried it.