Designing Approval Workflows for Flag Changes

This how-to is part of RBAC and Flag Change Governance. You want a second pair of eyes on the flag that gates payments, but you do not want every cosmetic toggle stuck behind a review queue. This how-to designs a risk-based approval workflow: high-risk changes require a second approver enforced server-side, low-risk changes flow freely, and a logged break-glass path exists for genuine emergencies.

The failure mode you are designing against is not malice — it is the confident, fast-moving engineer who flips ops.checkout.enable-new-tax-engine to 100% because it looked fine in staging, and takes the checkout page down for twenty minutes. A single-actor mistake on a flag that touches money, auth, or data integrity is the exact class of change where a second reader pays for itself. The trick is scoping that second reader so narrowly that people forget it exists until the moment it matters, rather than letting it metastasize into a queue that gates every string tweak. Get the scope right and approval feels like a seatbelt; get it wrong and it feels like a turnstile, and turnstiles get vaulted.

Prerequisites

• Flags classified by risk (see RBAC and flag change governance)
• Server-side change enforcement, not only a console UI
• Identities for actors and approvers, with distinct people enforceable
• A notification channel to reach approvers quickly
• A change record that captures approver identities (feeds the audit trail)

Step-by-Step Procedure

Step 1 — Gate only the flags that warrant it

Read the required approval count from the flag’s risk class so low-risk toggles need zero approvals and only high-risk flags queue for review. The risk class is metadata that lives with the flag definition, not a judgment the requester makes at change time — otherwise the person in a hurry simply declares their own change low-risk. Classify flags once, when they are created, and treat the class as an attribute the workflow reads rather than an input it accepts.

# gate.py — approvals required scale with risk
APPROVALS_BY_RISK = {"low": 0, "medium": 1, "high": 2}

def approvals_required(flag_meta: dict) -> int:
    return APPROVALS_BY_RISK[flag_meta["risk"]]

A reasonable default is to reserve high for flags that gate payments, authentication, personally identifiable data, or anything that writes to a system of record, and to leave everything cosmetic or purely additive at low. Aim to keep the high-risk set small — a good target is under one flag in ten. If a large fraction of your flags land in high, the classification has drifted and the gate will start feeling like a tax on everything, which is precisely the condition that breeds rubber-stamping. When a flag’s blast radius changes — say a UI toggle grows into a kill switch for a payment provider — reclassify it deliberately, and treat that reclassification as itself a governed change.

Step 2 — Enforce distinct approver identity

Reject an approval from the person who requested the change so a single account cannot satisfy both roles. This is the core of four-eyes review — everything else is plumbing around this one check.

# approve.py — the requester cannot approve their own change
def add_approval(change, approver_id):
    if approver_id == change.actor_id:
        raise Denied("the requester cannot approve their own change")
    if approver_id in change.approvers:
        raise Denied("duplicate approval from the same person")
    change.approvers.append(approver_id)

Two subtleties bite here. First, “distinct” must mean distinct human, not distinct session — if one engineer holds two accounts, or a service account is shared, the identity check passes while four-eyes quietly fails. Bind approver identity to the same SSO subject you use elsewhere, and forbid service accounts from ever counting as approvers. Second, decide up front whether the approver must also be someone who could not have colluded trivially: for the highest-risk flags, some teams require the second approver to sit outside the requester’s immediate team, so the review is a genuine outside read rather than a favor between deskmates. That is a policy dial, not a code change — keep it as configuration so you can tighten it for a handful of flags without rewriting the enforcement path.

Step 3 — Enforce the gate where the change is applied

Check the approval count at the moment the change is written, not merely when it is proposed, so an API call cannot skip the queue. This placement is the whole ballgame. If the gate lives in the console UI — a disabled “Apply” button until two approvals land — then anyone with an API token or a curl command walks straight around it. Enforcement has to sit at the same chokepoint every write passes through: the service that persists flag state, or the admission layer in front of it. A UI check is a convenience for humans, never the control.

# apply.py — final gate at write time
def apply_change(change, flag_meta):
    need = approvals_required(flag_meta)
    if len(change.approvers) < need:
        raise Denied(f"{change.flag}: {len(change.approvers)}/{need} approvals")
    write_flag(change)   # only reached when the gate passes

Re-reading the risk class and re-counting approvals at write time also closes a subtle race: the flag might have been reclassified from medium to high between proposal and apply, or an approver might have withdrawn. Evaluating approvals_required against the current flag_meta rather than a value cached at proposal time means the change is always judged by the rules in force at the instant it takes effect. If your flag store is eventually consistent, do this check inside the same transaction or compare-and-swap that writes the flag, so two approvals racing two writes cannot both slip through.

Step 4 — Provide a logged break-glass path

Allow one authorized actor to bypass the wait in an emergency, but require a reason and emit a high-visibility record so the bypass is reviewed after the fact. The design goal is asymmetry: bypassing must be easy — one command, no waiting for a sleeping approver at 3 a.m. — but never quiet. A break-glass event that does not page anyone or post to the incident channel is indistinguishable from having no gate at all, because the only thing keeping it exceptional is the certainty of being seen.

# breakglass.py — fast path with a loud audit signal
def emergency_apply(change, actor):
    require_role(actor, "incident-responder")
    if not change.reason:
        raise Denied("break-glass requires a reason")
    write_flag(change)
    emit_high_priority_audit("BREAK_GLASS", change, actor)   # paged review afterward

Keep the break-glass role separate from the everyday approver role and grant it narrowly — the on-call rotation, not the whole engineering org. That separation means using the fast path is a conscious act with a named holder, and it keeps your audit trail able to answer “who can bypass” as cleanly as “who did.” Some teams add a time bound: a break-glass write auto-expires or auto-reverts after a set window unless a normal two-approval change ratifies it, so the emergency toggle cannot silently become permanent config. That ratification step turns every bypass into a forcing function for the after-the-fact review rather than relying on someone remembering to look.

Verification

Assert the four-eyes rule directly: the requester cannot self-approve, a high-risk change with one approval is blocked, and a second distinct approval unblocks it. Write this as an executable test rather than a runbook step, because approval logic is exactly the kind of code that gets “simplified” during a refactor by someone who does not realize the self-approval check is load-bearing. A red test is a much louder alarm than a comment.

# approval.test.py
def test_four_eyes():
    c = change("ops.payments.kill", actor="alice", risk="high")
    with raises(Denied): add_approval(c, "alice")     # self-approval blocked
    add_approval(c, "bob")
    with raises(Denied): apply_change(c)              # 1/2 approvals, still blocked
    add_approval(c, "carol"); apply_change(c)         # 2/2 -> applied

Cover the negative cases too, since those are where enforcement quietly rots: a low-risk change should apply with zero approvals, a duplicate approval from bob should not count as the second signature, and a break-glass call from someone without the incident-responder role should be denied. The most valuable assertion of all is the one that proves the gate lives at write time — construct a change with insufficient approvals and confirm apply_change raises even when the proposal step was skipped entirely, so a future API path that forgets to call the proposal flow still cannot write.

Gotchas & Edge Cases

• Over-gating produces rubber-stamps. If routine changes require approval, approvers approve without reading and the review becomes theater. Gate only genuinely high-risk flags so an approval carries real attention. Watch the approval-to-rejection ratio as a health metric: if nothing is ever rejected, either your changes are flawless or your approvers are asleep, and the second is far more likely than the first.
• Approvals can go stale. A change approved on Monday and applied Friday may no longer be appropriate — the surrounding system, the traffic pattern, or the risk class may all have moved. Expire approvals after a short window (an hour for high-risk operational flags, a day at most) so an old sign-off cannot authorize a change in a different context, and re-open the review if the change payload is edited after it was approved.
• Break-glass must be reviewed. An emergency path nobody reviews afterward becomes the normal path. Route every break-glass event to a mandatory post-incident review so the fast lane stays exceptional, and track the count — a break-glass rate that climbs month over month is telling you the normal path is too slow or too broadly scoped.
• The approver needs enough context to actually judge. An approval request that shows only “change flag X” invites reflexive approval. Put the diff, the risk class, the requester’s stated reason, and the current value in the notification so the second reader can form a real opinion in the ten seconds they will actually spend on it — approval quality is bounded by the quality of what you show the approver.
• Do not let the notification channel become the enforcement layer. Reaching approvers over chat is fine for speed, but if the only record of a decision is a Slack reaction, you have no durable, tamper-evident trail. The authoritative approval must be recorded server-side against the change, feeding the audit trail; the chat message is just the doorbell, not the lock.

Troubleshooting & FAQ

Approvers are a bottleneck during business hours. What can we do?

Widen the approver pool (any two of a team’s engineers, not one designated lead) and scope approvals tightly to high-risk flags. If a specific flag needs approval constantly, that is a signal it is being toggled too often — investigate the underlying churn rather than adding approvers.

Can the same person be a valid approver for different changes?

Yes — the constraint is per change, not per person. Anyone with the approver role can approve any change they did not request. The rule only forbids self-approval and duplicate approvals on the same change.

How is this different from a code review on the flag config in Git?

A GitOps flow gives you review through pull requests, which is excellent for planned changes. Runtime approval workflows cover the changes that happen in the console during operations — including urgent ones — where a Git round-trip is too slow. Many teams use both: GitOps for planned config, runtime approval for live toggles.

Should turning a flag off — a rollback — also require approval?

Usually no. The whole point of a flag is to make disabling a bad change instant, so gating the off switch reintroduces the delay you built flags to remove. Treat “return to the last known-safe value” as a low-risk operation even on a high-risk flag, and reserve the two-approval requirement for enabling or widening exposure. If you must record who rolled back, capture it in the audit trail rather than blocking it behind a second signature — the record is cheap, the delay is expensive.

How do we handle approval for a scheduled or automated change?

Attribute the change to the human who scheduled it and capture the approvals at schedule time, then re-validate them at execution time. If the approvals have expired or the flag was reclassified in the interval, the automated apply should fail closed and page rather than proceed on stale authority. A machine actor firing the change does not dissolve the four-eyes requirement — it just moves the human sign-off earlier, so the enforcement at write time still has to see valid approvals or refuse.

What is the right number of approvers — is two always enough?

Two distinct people is the standard for four-eyes and covers the overwhelming majority of high-risk flags; going higher trades safety for latency fast, and each extra required approver roughly multiplies the odds of a stall waiting on availability. Reserve three-approver gates for a tiny set of catastrophic-blast-radius flags — a global payments kill switch, a data-deletion toggle — where the cost of a wrong change dwarfs the cost of waiting. For everything else, one requester plus one distinct approver is the sweet spot.

Where should approval state be stored so it survives a flag-service restart?

In the same durable store that holds the change record, not in the memory of the process handling the request. Approval is a fact about a pending change, and it must outlive deploys, restarts, and failovers — otherwise a rolling deploy mid-review silently drops accumulated approvals and forces re-review, which trains people to rush. Persist approvals alongside the change and read them back at write time; that is also what lets the audit trail reconstruct exactly who signed off on any historical change.