I broke my own audit log

I treated serialization as something I could clean up in a refactor. In a hash chain, it's part of the security contract. A postmortem of the Imara 0.1.x to 0.2.0 hash-format change.

Imara is an MCP governance proxy I build and maintain. It sits between an AI agent and its tools, applies policy to every tool call, and writes each decision to a SHA-256 hash-chained audit log. The pitch to a compliance buyer fits in one sentence: if anyone edits history, verification fails. The contrapositive has to hold just as absolutely: if nobody edits history, verification never fails. Tamper evidence is not a feature of the product. It is the product.

In June I shipped Imara 0.2.0 and broke that second half myself. Audit logs written by 0.1.x fail verification under 0.2.0, and nothing in them was tampered with. This is the postmortem, including why the fix I shipped still isn't good enough for the product Imara claims to be.

What the chain commits to

Each audit event is hashed with SHA-256 over a JSON array of its fields in a fixed order: id, timestamp, session, server, tool name, tool arguments, annotations, policy decision, policies evaluated, result status, and the previous event's hash. That last field is the chain link. Rewrite any event and its hash changes; every later event's prevHash then points at nothing, and verification reports the exact index where history diverged.

The 0.1.x hashing code carried a comment: "Explicit key ordering for deterministic hashing across environments." That was true for the top-level fields and false one level down. Tool arguments and annotations are nested JSON objects, and 0.1.x serialized them in whatever key order the JavaScript object happened to have. Key order in a JS object is an accident of insertion: the byte order the client put on the wire, or the order some transform rebuilt the object in. Two semantically identical events could hash differently. Worse, the same stored event, re-serialized through any code path that reconstructs the object (a schema parse, a spread merge, a redaction pass), would fail verification with nothing wrong. A tamper-evidence system that raises false alarms trains its operators to ignore alarms, which is worse than having no alarms at all.

What 0.2.0 changed and why old chains break

The fix is recursive canonicalization before hashing: object keys are sorted at every depth, arrays keep their order because array order carries meaning. Same logical event, same hash, regardless of how the object was built.

The release also fixed verifier bugs that were frankly more embarrassing than the serialization. The 0.1.x verifier only checked an event's stored hash if the event happened to carry one; an event without a stored hash was silently accepted. The genesis event was never anchored, so you could delete the first N rows of the log and the remainder verified clean. 0.2.0 requires the stored hash on every event and recomputes it unconditionally, requires a null prevHash on the genesis event so prefix deletion is detected, and records a head anchor (final hash plus event count) in ~/.imara/anchor.json on each verify, so end-truncation and wholesale rewrite are caught across runs. Those are things a hash chain cannot detect from inside itself, and I say so in the docs now rather than implying the chain does more than it can.

But sorting nested keys changes the hash preimage. Any 0.1.x event whose argument keys weren't already in sorted order recomputes to a different hash under 0.2.0, and the verifier correctly reports it as a mismatch. The verifier is doing its job. The format moved underneath it. An honest history now looks tampered.

What I shipped as the answer

The 0.2.0 CHANGELOG says: treat 0.2.0 as a fresh baseline. Archive any ~/.imara/audit.db you need to keep, then run imara verify to establish a new anchor.

That's a re-baseline: keep the old file if you care about it, start a new chain, record its head. For a v0.1 tool with no production users, it's defensible. For the product Imara claims to be, it isn't. Picture a compliance buyer with a 14-month retention window, which is an ordinary requirement once SOC 2 audit periods and lookback are involved. The instruction "archive the old file" converts fourteen months of cryptographically verifiable history into a folder of bytes the auditor has to take on trust. The moment the old chain can't be verified, its tamper evidence is gone retroactively. You didn't lose the data. You lost the property, and the property was the point.

What the migration should have been

Nothing here requires novel cryptography. Three mechanisms, all boring:

1. Version-tag every record. A hash_version column, or a per-segment format version. The verifier dispatches on it: v1 segments verify under v1 rules, v2 under v2. The old hasher stays frozen in the codebase forever. It's about forty lines. Deleting it saved nothing.
2. Anchor the old chain head into the new one. The first v2 event is a migration record whose payload commits to the v1 head hash, the v1 event count, and the old algorithm identifier. Continuity then spans the format boundary: you can't swap out the archived v1 log without breaking the commitment sitting at the base of the v2 chain.
3. Record an external anchor at cutover, so the migration moment itself is pinned outside the store.

Imara does none of these today. The current design actually forecloses the second one: genesis is required to have a null prevHash, so there's no slot to carry the old head without a schema change. I'm stating that as a known gap, not as a roadmap item dressed up as done. And there will be a next format change. My canonicalization sorts keys but still leans on JavaScript's number formatting; a move to RFC 8785 JCS proper is the likely endpoint, and when it happens it should be the first migration to cross the boundary without breaking verification.

The actual lesson

The serialization of the hash preimage is part of the security contract, and I treated it as an implementation detail I could clean up in a refactor. It's more binding than the database schema. You can migrate a schema with a script. You cannot recompute a commitment, because the whole point of a commitment is that nobody, including you, gets to recompute it. Systems that live with this well (Certificate Transparency logs, TUF metadata) treat verifier behavior for old versions as frozen and version the format from day one. Doing that on day one costs a column and a switch statement. Doing it after the fact costs a breaking release and an essay like this.

Was 0.2.0 still the right call? Yes. A verifier that accepts prefix deletion and skips hash checks on unhashed events is a worse liability than a format break, and shipping the fix fast mattered more than shipping it gracefully. Given the choice again I'd make the same change and ship it behind a version tag.

One more thing, stated plainly: Imara has no production adopters yet. This break inconvenienced nobody but me and my demo database. That's exactly why this is the cheap moment. The migration story costs a design decision today and costs a customer's audit trail later. Every real chain that gets written before record versioning lands is a chain I've promised to carry across the next format change with machinery that doesn't exist yet. Better to build it while the only history at stake is mine.