Auditing Who Changed a Flag and When
This how-to is part of RBAC and Flag Change Governance. A flag flipped at 3 a.m. and an incident followed — and the first question in the retro is “who changed it, and why?” This how-to builds the change record that answers it: every toggle captures actor, before and after value, reason, approvers, and environment, in a form you can query in seconds and hand to an audit trail.
The distinction that matters is between a log and a record. A log is a stream of events your systems happen to emit — verbose, best-effort, and often thrown away after a fortnight. A record is a deliberate, structured artifact you commit as part of the change itself, with a schema you control and a guarantee it cannot be quietly edited. When the pressure is on during an incident and later in front of an auditor, only the record holds up. The rest of this guide treats the change record as a first-class output of every flag write — not a side effect you scrape back together afterward from application logs, chat history, and half-remembered Slack threads.
Prerequisites
Step-by-Step Procedure
Step 1 — Capture the record at the moment of change
Write the full record synchronously with the change, so the write and its provenance are inseparable. Include the previous value read just before applying the new one. The reason you capture at change time rather than reconstruct later is that context evaporates fast — the actor knows exactly why they are flipping the flag in the second they do it, and knows almost nothing about it a week later when someone asks. Capturing the source matters just as much as the actor: the same person acting through the console, a scripted API call, and a CI pipeline are three different provenance stories, and an auditor will care which one it was. Timestamp in UTC with a timezone designator (the trailing Z) so records from services in different regions sort into a single, unambiguous order — a naive local timestamp turns a cross-region incident timeline into a guessing game.
# record.py — build the record from the change context
def record_change(flag, old, new, actor, reason, approvers, env):
return {
"flag": flag, "from": old, "to": new,
"actor": actor, "reason": reason, "approvers": approvers,
"env": env, "ts": current_iso8601(), "source": current_change_source(),
}Step 2 — Persist it append-only alongside the change
Commit the record to append-only storage in the same transaction as the flag write so a change can never exist without its record. The ordering inside the transaction is deliberate: read the old value first, apply the write, then append the record built from both — if any step throws, the whole unit rolls back and neither the flag nor the record moves. When your flag store and your audit store are genuinely separate systems and a single transaction is impossible, fall back to a write-ahead pattern: record the intent first with a pending status, apply the change, then mark the record committed, and treat any record stuck in pending as a reconciliation alarm rather than a silent gap. Append-only does not mean append-once-and-forget — pick storage that rejects updates and deletes at the engine level (an object-lock bucket, an immutable ledger table, or a WORM-configured store) so immutability survives a compromised application credential, not just well-behaved code.
# persist.py — change and record are one atomic unit
def apply_and_audit(change):
with transaction():
old = read_flag(change.flag)
write_flag(change)
append_audit(record_change(change.flag, old, change.to,
change.actor, change.reason, change.approvers, change.env))Step 3 — Make it queryable by flag, actor, and time
Index the records so “every change to this flag,” “everything this actor did,” and “all changes in this window” are fast queries — the three questions a retro actually asks. In practice that means a composite index on (flag, ts) for the per-flag timeline, one on (actor, ts) for the per-actor sweep, and a plain index on ts for the window scan; without them, the query you run under incident pressure does a full table scan across months of history at exactly the moment you need it in under a second. Match the flag LIKE 'ops.payments.%' prefix pattern to your key namespace convention so a single query pulls every flag owned by a service, not just the one you already suspected — the flag that actually caused the incident is often a sibling of the one you went looking for.
-- who changed ops.payments.* in the hour before the incident?
SELECT ts, flag, actor, "from", "to", reason, approvers
FROM flag_change_audit
WHERE flag LIKE 'ops.payments.%'
AND ts BETWEEN '2026-07-22T13:00:00Z' AND '2026-07-22T14:10:00Z'
ORDER BY ts;Step 4 — Surface the record in the incident flow
Link the change record into your incident tooling so the on-call sees “recent flag changes” without leaving the incident channel. The value here is measured in minutes of mean-time-to-resolution: a responder who can see that ops.payments.new-processor flipped four minutes before error rates climbed has a prime suspect immediately, instead of paging three teams to ask “did anyone change anything?” Post the changes automatically when an incident opens rather than making someone remember to run the query — the moments you most need the change history are exactly the moments no one has spare attention to go fetch it. Scope the automatic post to the services the incident touches so the channel shows the two relevant changes, not forty unrelated ones that bury the signal.
# recent-changes.sh — post recent flag changes into the incident channel
flagctl audit query --namespace "ops.payments.*" --since 1h --json \
| jq -r '.[] | "\(.ts) \(.actor) \(.flag): \(.from)->\(.to) (\(.reason))"'Verification
Assert that a change produces a complete record and that the record cannot be altered after the fact. Two properties are worth testing explicitly and neither is obvious from a happy-path check. First, completeness: every one of the six fields is present and non-empty, because a record with a null actor or a blank reason passes a naive “row exists” assertion while failing the only question it needs to answer. Second, immutability at the store level, not the application level — the test below tries to rewrite a committed record and asserts the store itself rejects it, which is the guarantee an auditor actually trusts. Add a third test for the atomicity of failure: inject a fault into the record write and assert the flag write rolls back too, so you have proof the two never drift apart.
# audit.test.py
def test_change_is_recorded_and_immutable():
apply_and_audit(change("web.checkout.x", to=True, actor="alice", reason="GA"))
rec = latest_audit("web.checkout.x")
assert rec["actor"] == "alice" and rec["from"] is False and rec["to"] is True
with raises(ImmutableStore):
rec["actor"] = "bob"; persist(rec) # append-only rejects the rewriteGotchas & Edge Cases
- A change without its record is worse than no audit. If the flag write succeeds but the record write fails, you have a gap precisely where accountability matters. Make the two atomic so a change cannot outlive a failed record.
- Automated changes still need an actor. A rollout controller or a scheduled cleanup is a valid actor, but “system” alone is not enough — record which automation and what triggered it (a pipeline run, a guardrail breach) so the automated path is as traceable as a human one.
- Empty reasons defeat the record. A required reason field that accepts “.” teaches nothing. Require a reason on high-risk changes and, where possible, validate it against a minimum length or a linked ticket.
- Bulk and cascading changes need per-flag records, not one summary. A single “disable everything” action or a namespace-wide kill switch that flips forty flags must write forty records, each with its own before/after value — collapse it into one line and the retro can no longer tell which specific flag was already off and which one you actually turned. Record the batch with a shared correlation ID so the individual entries still group back together when you need the whole story.
- The record schema is itself a governed asset. If you add a field or change what “reason” means six months in, old records predate the new shape and a naive query silently drops them. Version the record schema and keep readers tolerant of older versions, so a query that spans the change boundary returns every relevant record instead of only the recent ones — the gap you introduce is invisible until the one audit that straddles the migration.
Troubleshooting & FAQ
The record shows who but the reason is useless. How do we get better reasons?
Reasons improve when they are required at the point of change on the flags that matter and when they are read in retros. Link the reason field to a change ticket or incident ID so it points at context rather than restating the obvious. Reserve the requirement for high-risk flags so it does not become friction people route around.
How long should we retain flag change records?
At least as long as your compliance regime requires, and long enough to cover a realistic retro window — a change that caused a slow-burn issue may not surface for weeks. The audit trail guide covers retention policy; align flag change records with it.
Can I reconstruct history from the flag system’s built-in log instead?
Sometimes, but vendor logs vary in completeness and retention and may not capture reason or approver. Owning your own change record decouples your audit posture from the vendor’s log format and lets you enforce the fields you need, including the reason and approvers your governance requires.
Should the change record store the full flag config or just the toggled value?
Store enough of the before and after state to reconstruct what actually changed, which for a simple boolean is just the two values but for a targeting rule or rollout percentage is the relevant slice of config. Recording only “changed” without the diff means the retro can see that someone touched the flag but not what they did — and for a rule change, the difference between 5% and 50% rollout is the whole story. Keep the diff scoped to what changed rather than snapshotting the entire config on every write, so records stay small and the meaningful delta is not buried.
How do we audit flag changes that happen through infrastructure-as-code or GitOps?
Treat the merge as the change and the commit as the record’s backbone: the author, reviewers, and commit message already answer who, who-approved, and why, and the deploy that applies the manifest supplies when and to which environment. The gap to close is joining the Git provenance to the runtime apply, so link the record to both the commit SHA and the deploy run — otherwise you can prove the config was authored but not that it actually reached production, or when.
What if the same actor makes many rapid changes during an incident?
Record every one of them individually — rapid changes during firefighting are exactly the history a retro most needs, and deduplicating or rate-limiting the records to save space throws away the sequence that explains what happened. A shared incident correlation ID on each record lets you pull the whole burst back as one story while keeping each step’s before/after value intact, so you can see the order in which mitigations were tried and which one actually moved the metric.