Writing a Feature Flag Incident Response Runbook

Q: We rolled the flag back but errors continued. Why?

Either the change did not propagate to every node — verify the variant split reaches zero — or the flag was not the cause. If propagation is confirmed and errors persist, widen the investigation.

Q: How fast should a kill switch propagate?

With streaming, well under a second to connected nodes; with polling, up to one poll interval. If your time-to-mitigate target is tighter than the poll interval, stream the kill-switch flags specifically.

Q: Should every flag have its own runbook entry?

No — only high-risk flags such as payment, auth, and data-writing paths warrant a bespoke entry. Most are covered by the general roll-back-the-suspect procedure; tag the high-risk ones in metadata.

Q: Who runs the retro, and how soon after the incident?

The commander convenes it with the operator and scribe, within a couple of business days while memory is fresh. Keep it blameless and require at least one concrete action item with an owner and a date.

Q: What belongs in the incident timeline the scribe keeps?

Timestamps for each phase transition — alert, declaration, kill switch, propagation confirmed — plus the exact commands run and who ran them. Capture it live in the incident channel so time-to-mitigate can be computed honestly.

Q: Should the runbook automate the rollback, or keep a human in the loop?

Both, layered. An automated circuit breaker handles fast, unambiguous cases where a guardrail crosses a hard threshold, while the manual runbook covers judgment calls automation cannot make safely, such as stateful flags.

This how-to is part of Operational Safety and Incident Response. At 2 a.m. a rollout is causing errors and the on-call engineer needs to know, without thinking, exactly which flag to flip and how to confirm it worked. This how-to writes that runbook: the detect-decide-rollback-verify-retro sequence, the roles, and the timings — then rehearses it so it works when it counts.

A runbook earns its keep precisely because incidents degrade the people responding to them. Under adrenaline and sleep debt, working memory shrinks and improvisation gets expensive — the engineer who could reason fluently about the rollout at 2 p.m. is guessing at 2 a.m. The runbook exists to move decisions from the incident, where thinking is scarce, to design time, where it is cheap. Everything below is written to be executed line by line, not interpreted — the difference between a document that describes the system and a document that tells you what to type next.

The runbook is these five phases with names, commands, and expected timings attached — so the on-call path is to read and act, not to improvise.

Prerequisites

Alerts wired to the metrics that indicate a bad rollout (error rate, guardrail breach)
A kill switch per high-risk flag, independent of the normal deploy path
A way to confirm a flag change reached every node (propagation check)
Defined incident roles: incident commander, operator, scribe
A retro template and a place incidents are logged

Roles matter as much as tooling — an incident without a named commander becomes several people running conflicting rollbacks.

Step-by-Step Procedure

Step 1 — Detect and name the incident

The alert fires; the on-call engineer confirms it is a flag-driven regression by checking the variant split and error rate for recently changed flags, then declares an incident with a commander. The confirmation step matters — not every error spike is a flag, and rolling back the wrong flag wastes minutes while the real problem grows. The fastest discriminator is timing: if the error curve inflected within seconds of a flag change appearing in the audit log, the flag is the prime suspect. Declaring the incident is not ceremony; it starts the clock, opens the shared channel, and assigns the commander so that every subsequent action has one owner and one timeline.

# triage.sh — which recently changed flag correlates with the error spike?
promtool query instant http://prom:9090 \
  'topk(3, sum by (flag) (rate(flag_eval_total{reason="ERROR"}[5m])))'
# Cross-reference the top flags with the flag change log for the last hour.

Step 2 — Decide: roll back now or investigate live

Default to rolling back a suspected flag first and investigating after — a kill switch is reversible, so a wrong rollback costs little, while a prolonged bad rollout costs users. Record the decision and who made it. This asymmetry is the whole reason flags exist as a safety mechanism: reverting a flag does not require a build, a deploy, or a review, so the cost of being wrong is a few seconds of a slightly worse product, not an outage. That said, “roll back first” has one important exception — flags that write or migrate data. Turning off a half-finished dual-write can leave records inconsistent, so for stateful paths the runbook should name the safe direction explicitly rather than assuming false is always benign. Write the decision rule down as a table the commander reads aloud, because “let’s think about whether to roll back” is exactly the deliberation the runbook is meant to eliminate.

Decision rule (write this in the runbook):
  user-facing errors rising  -> roll back the suspect flag NOW, investigate after
  internal-only / low impact  -> investigate live, roll back if unclear within 10 min

Step 3 — Execute the kill switch

Flip the flag through the out-of-band kill-switch path. This is a single, well-known command per flag, not an ad-hoc edit, so it runs the same at 2 a.m. as in a drill. The --reason and --actor fields are not decoration — they write the incident ID and the human into the audit trail so the retro can reconstruct exactly what changed and when, and so a second responder can see at a glance that a rollback is already in flight. Resist the urge to hand-edit the flag definition in a config file or a provider UI during an incident: those paths have no guardrails, no shared visibility, and a fat-fingered value can turn a one-flag incident into two. The kill-switch command is a contract, and the incident is not the time to renegotiate it.

# rollback.sh — flip the flag to its safe state via the kill-switch path
flagctl set ops.checkout.new-summary --value=false --reason="incident-1421 rollback" --actor="$USER"
# The kill-switch path is independent of the normal deploy pipeline.

Step 4 — Verify propagation to every node

Confirm the change actually reached every replica — a rollback that landed on 90% of nodes still leaves 10% serving the bad path. Check the observed variant split collapses to the safe value. The failure mode this guards against is quiet: the operator runs the command, sees a success response, announces “rolled back,” and moves on — while a stuck poller or a partitioned node keeps a slice of traffic on the bad variant. Watch the observed treatment share, not the command’s exit code, because the exit code tells you the control plane accepted the change, not that every evaluator is now serving it. If the curve plateaus above zero instead of reaching it, you have a propagation gap, and the runbook should escalate to draining or restarting the lagging nodes rather than declaring victory. Verification is where a runbook proves it understands the difference between “I changed the flag” and “the change is in effect everywhere.”

# verify.sh — the treatment share should fall to ~0 within the propagation window
watch -n2 "promtool query instant http://prom:9090 \
  'sum(rate(flag_eval_total{flag=\"ops.checkout.new-summary\",variant=\"true\"}[1m]))'"
# Expect this to reach zero; if it plateaus above zero, a node is not receiving updates.

Verification (step 4) is the step people skip under pressure — declaring "rolled back" before confirming propagation leaves a fraction of traffic still on the bad path.

Verification

Rehearse the whole runbook as a game day and time it. The metric of a good runbook is time-to-mitigate: from alert to a confirmed-propagated rollback. A runbook that has never been executed is a hypothesis, not a procedure — the first real run always surfaces something: a command that needs credentials no one has at 2 a.m., a dashboard link that 404s, a propagation window twice as long as anyone assumed. Run the game day against production or a production-faithful staging environment, because the gaps you care about live in the seams between systems, and those seams do not exist in a toy setup. Schedule it on a cadence — quarterly is a reasonable floor — and rotate who holds the keyboard so the knowledge does not concentrate in one person who might be the one asleep during the real event. Track time-to-mitigate as a trend, not a single number: if it is creeping up, something in the system got slower or more complicated while the runbook stood still.

# gameday.sh — inject a bad flag, run the runbook, record the timeline
flagctl set ops.checkout.new-summary --value=true --reason="gameday-injected-fault"
date +%s > /tmp/gameday_start
# ...on-call executes the runbook; record when propagation hits zero...
echo "time-to-mitigate: $(( $(date +%s) - $(cat /tmp/gameday_start) ))s"

Timing the drill turns "we have a runbook" into a number you can improve — and reveals whether the kill switch really propagates as fast as you assumed.

Gotchas & Edge Cases

The shared-failure-domain case is the nightmare — discovering mid-incident that the kill switch itself is down. Only a game day surfaces it before it matters.

The kill switch may share the outage’s failure domain. If the switch is flipped through the same control plane that just went down, it will not work when you need it. Route the kill-switch path through an independent mechanism and prove it in a game day.
Sticky bucketing survives the rollback. Users pinned to the bad variant by sticky bucketing keep it after you drop the percentage. If you need everyone off immediately, the runbook must clear the assignment, not just lower the rollout.
No commander means chaos. Two engineers rolling back different flags, or one re-enabling what another disabled, prolongs the incident. Name a commander in the first minute; everyone else acts through them. The commander’s job is coordination, not typing — the moment they start editing flags themselves, no one is watching the whole board.
The runbook is stale and no one noticed. Flag keys get renamed, services get split, propagation paths change — and the runbook, being a document, does not fail loudly when it drifts out of sync with reality. A command that references ops.checkout.new-summary after the flag was renamed to ops.checkout.order-summary fails at the worst possible moment. Tie the runbook’s freshness to the game-day cadence so drift is caught on a drill, not on an incident.
The alert that should have fired didn’t. Time-to-mitigate is measured from the alert, so an incident detected by an angry customer instead of a monitor has already burned minutes before the runbook even starts. Every retro should ask whether an alert should have caught this sooner, and treat a missing alert as an action item with the same weight as a code fix.

Troubleshooting & FAQ

We rolled the flag back but errors continued. Why?

Either the change did not propagate to every node (verify the variant split reaches zero), or the flag was not actually the cause. If propagation is confirmed complete and errors persist, widen the investigation — the correlated flag may have been a symptom, not the root cause.

How fast should a kill switch propagate?

With streaming synchronization, well under a second to connected nodes; with polling, up to one poll interval. If your runbook’s time-to-mitigate target is tighter than your poll interval, stream the kill-switch flags specifically. See polling vs streaming.

Should every flag have its own runbook entry?

No — only high-risk flags (payment, auth, data-writing paths) warrant a bespoke entry. Most flags are covered by the general “roll back the suspect flag” procedure. Tag the high-risk ones in metadata so the runbook and the on-call know which get special handling.

Who runs the retro, and how soon after the incident?

The commander convenes it, but they should not be the only voice — the operator and scribe hold details the commander missed while coordinating. Run it within a couple of business days, while memory is fresh but adrenaline has drained, and keep it blameless: the output is fixes to the system and the runbook, not a verdict on a person. Every retro should produce at least one concrete action item with an owner and a date, or it was a debrief, not a retro.

What belongs in the incident timeline the scribe keeps?

Timestamps for each phase transition — alert fired, incident declared, kill switch executed, propagation confirmed — plus the exact commands run and who ran them. The timeline is what lets the retro compute time-to-mitigate honestly and spot where minutes leaked. Capture it in the incident channel as events happen rather than reconstructing it afterward, because human memory of an incident’s timing is unreliable within hours.

Should the runbook automate the rollback, or keep a human in the loop?

Both, layered. An automated circuit breaker handles the fast, unambiguous cases — a guardrail metric blowing past a hard threshold — while the manual runbook covers the judgment calls automation cannot make safely. Automate the mechanics you have proven on game days, but keep a human commander for the decisions where being wrong is expensive, such as stateful flags. See implementing circuit breakers for flag providers.