Promoting Flags with GitOps and Pull Requests

Q: The reconciler keeps reverting a change someone made in the console. Is that a bug?

No — git is authoritative, so an out-of-band change is drift and gets reverted. Make legitimate changes via a PR; exempt kill-switch flags from reconciliation so the reconciler does not fight an emergency action.

Q: How do I promote from staging to prod?

Open a PR copying the relevant flag state from the staging overlay into the prod overlay. The diff is the promotion, the review is the gate, and the merge triggers the reconciler.

Q: Does GitOps replace runtime RBAC and approvals?

For planned changes largely yes — PR review is the approval and repo permissions the access control. But runtime RBAC and approvals still cover the break-glass path and console changes, so keep both.

Q: How fast should the reconciler run against production?

Every one to five minutes is common for prod: it bounds the drift window to a single interval while leaving a break-glass change time to be noticed and synced back to git. Pair the schedule with alerting on any non-empty diff so a human learns that live and git disagree.

Q: What happens if the reconciler runs on a stale or bad merge?

It applies whatever is on the watched branch, so a bad merge becomes a bad live state. Require CI and schema validation before merge, pin the reconciler to one branch, and fix a bad state the way you fix any regression — revert the commit and let the reconciler apply the previous known-good state.

Q: Should each environment be its own repo, or overlays in one repo?

Prefer one repo with per-environment overlays: shared base definition, promotion as a readable diff, and one history to blame. Separate repos per environment re-introduce drift because the base gets copied and the copies diverge. Use separate repos only when hard access boundaries require it.

This how-to is part of Multi-Environment Flag Promotion Pipelines. Toggling flags directly in each environment’s console leads to drift and no review trail. This how-to manages flag configuration as code in git and promotes it through environments with pull requests: git is the source of truth, per-environment overlays hold the differences, a PR is the review gate, and a reconciler applies merged changes to the live flag system.

The payoff is that every flag change acquires the same properties your application code already has — a diff, an author, a reviewer, a timestamp, and a revert path. When an incident review asks “who turned this on and when,” the answer is a git blame away rather than a scavenge through console audit logs that may only retain 90 days of history. The cost is latency: a change that used to be a click is now a PR round-trip, typically a few minutes. That trade is almost always worth it for planned changes, and this how-to keeps a break-glass path for the rare case where it is not.

The PR is both the change mechanism and the review gate; the reconciler makes the live system match git, so the repository is always the truth.

Prerequisites

Flag configuration expressible as declarative files (YAML/JSON) in a git repo
Per-environment overlays so dev, staging, and prod can differ intentionally
A reconciler or sync agent that applies git state to the flag system
Branch protection and required reviews on the flag repo
A drift check (see detecting flag configuration drift)

The reconciler is what closes the loop — without it, git is documentation, not a source of truth the live system actually follows.

Step-by-Step Procedure

Step 1 — Model flags as a base plus environment overlays

Keep shared flag definitions in a base file and per-environment differences in overlays, so promotion is a small, reviewable diff rather than a full re-declaration. The base file is where invariants live — the flag’s type, its owner, its default, its description — and those should be identical across every environment. The overlays hold only what legitimately varies: whether the flag is enabled and at what rollout percentage. This split matters because it makes an accidental type change or an owner rename impossible to sneak into a single environment; those edits touch the base file and therefore affect every environment at once, which is exactly the review signal you want. Keep overlays minimal — an overlay that restates the type and default has stopped being a diff and started being a second source of truth, and the two will drift.

# base/flags.yaml — shared definition
web.checkout.new-flow: { type: boolean, default: false, owner: checkout }
# overlays/prod.yaml — prod-specific state
web.checkout.new-flow: { enabled: true, rolloutPct: 25 }
# overlays/staging.yaml
web.checkout.new-flow: { enabled: true, rolloutPct: 100 }

Step 2 — Change flags only through pull requests

Make the flag repo the sole write path, with branch protection requiring review. A flag change is now a diff a reviewer can read and a history git records. Enforce this at the platform level, not by convention: revoke direct write access to the console for humans, and issue the reconciler its own service credential so it is the only identity that mutates live flags. If engineers keep their console edit rights “just in case,” they will use them under pressure and your git history will quietly stop reflecting reality. Require at least one review from someone outside the change author, and consider a CODEOWNERS rule that pulls the owning team onto any PR touching their flags — a rollout percentage on web.checkout.new-flow should not be reviewable by someone who has never seen the checkout funnel. Keep PRs small and single-purpose; a PR that bumps one rollout percentage is trivial to reason about and to revert, whereas a PR that touches twelve flags across three environments forces the reviewer to either scrutinize all of it or rubber-stamp the lot.

# promote.sh — open a PR that advances prod from 25% to 50%
git checkout -b promote/new-flow-50
yq -i '.["web.checkout.new-flow"].rolloutPct = 50' overlays/prod.yaml
git commit -am "checkout new-flow: prod 25% -> 50%" && gh pr create --fill

Step 3 — Reconcile merged state to the flag system

On merge, a reconciler diffs the desired git state against the live flag system and applies only the changes, so the live system converges to git. Make the apply idempotent — reconciling an already-in-sync environment must be a no-op — because the reconciler will run repeatedly on a schedule, and a non-idempotent apply that rewrites unchanged flags floods your audit log and can reset rollout counters. Stamp every write with the commit SHA that authorized it, as the reason field does above; that turns the live system’s own change log into a back-reference to the exact PR, which is invaluable during an incident. Decide deliberately whether the reconciler is authoritative in both directions or only pushes git-to-live. A one-way push is simplest, but it means a flag that exists live and not in git is invisible to it. If you want the reconciler to also delete live flags absent from git, gate that behind an explicit prune flag and never let it run automatically against prod until you trust the desired state completely — an accidental overlay deletion should not silently tear down live flags.

# reconcile.py — apply git desired-state to the live flag system
def reconcile(env: str):
    desired = load_overlay("base", f"overlays/{env}")
    live = flag_api.get_all(env)
    for key, want in desired.items():
        if live.get(key) != want:
            flag_api.set(env, key, want, reason=f"gitops sync {current_commit()}")

Step 4 — Detect and correct out-of-band drift

Run the reconciler (or a drift check) on a schedule so a change made directly in the console is either reverted to match git or surfaced for reconciliation, keeping git authoritative. Run it often enough that drift cannot persist unnoticed — every few minutes is reasonable for prod — but pair the schedule with alerting, because a drift check that only exits non-zero in CI tells you nothing about a Tuesday-afternoon console toggle. The --dry-run mode below is deliberately the same code path as the real apply, just without the write, so the drift report is exactly the set of changes an apply would make; that equivalence is what lets you trust a green dry-run as proof of sync. For deeper coverage of comparing environments and catching schema-shaped divergence, see detecting flag configuration drift across environments.

# drift-check.sh — fail if live differs from git
python reconcile.py --dry-run --env prod | tee drift.txt
[ -s drift.txt ] && { echo "live prod drifted from git:"; cat drift.txt; exit 1; } || true

The drift check (step 4) is what makes git truly authoritative — without it, one console toggle silently breaks the "git is truth" promise.

Verification

Open a PR that changes a flag, merge it, and confirm the live flag system matches the merged git state — then make an out-of-band console change and confirm the drift check catches it.

# After merge, live should equal git; then break it and confirm detection
python reconcile.py --dry-run --env staging   # empty diff => in sync
flagctl set web.checkout.new-flow --env staging --rolloutPct 99   # out-of-band change
python reconcile.py --dry-run --env staging   # now shows the drift

Both directions prove the loop: git changes reach live, and live changes that bypass git are caught rather than silently accepted.

Gotchas & Edge Cases

The reconciler-fights-kill-switch case is the dangerous one — an aggressive reconciler can revert an emergency kill switch back to on within seconds.

Emergencies cannot wait for a PR. A GitOps round-trip is too slow for an incident. Keep a break-glass path (see the kill switch runbook) that changes the live flag directly, then sync the change back into git afterward so history stays complete.
The reconciler can revert a kill switch. If an operator flips a kill switch in the console and the reconciler enforces the old git state, it undoes the emergency action. Exempt kill-switch flags from reconciliation, or have the reconciler treat a live-off as authoritative and sync it back to git.
Do not put secrets in the flag repo. Flag config is fine in git; SDK keys and provider credentials are not. Keep secrets in a secret manager and reference them, so a public or widely-read flag repo never leaks credentials.
A merge is not an apply — watch the gap. Between the moment a PR merges and the moment the reconciler finishes applying there is a window, often tens of seconds to a couple of minutes, where git says one thing and live says another. Do not treat “PR merged” as “change is live.” Have the reconciler report success back to the PR (a status check or a comment with the applied commit SHA) so the person who merged knows the change actually landed, and so a failed apply surfaces loudly instead of leaving git and live silently disagreeing.
Overlays can silently diverge from the base’s contract. If someone changes a flag’s type in the base file — say boolean to string for a multivariate rollout — every overlay that still sets enabled: true may now be applying a value the flag no longer accepts. Validate overlays against the base schema in CI before the reconciler ever sees them, so a type or key mismatch fails the PR rather than the live apply. This pairs naturally with versioning flag configuration schemas safely.

Troubleshooting & FAQ

The reconciler keeps reverting a change someone made in the console. Is that a bug?

No — that is GitOps working as designed: git is authoritative, so an out-of-band change is drift and gets reverted. If the console change was legitimate, make it in git via a PR. If it was an emergency kill switch, exempt that flag from reconciliation so the reconciler does not fight it.

How do I promote from staging to prod?

Open a PR that copies the relevant flag state from the staging overlay into the prod overlay. The diff is the promotion, the review is the gate, and the merge triggers the reconciler to apply it. Because it is a normal PR, it inherits your branch protection and review rules automatically.

Does GitOps replace runtime RBAC and approvals?

For planned changes it largely does — PR review is the approval, and repo permissions are the access control. But runtime RBAC and approval workflows still cover the break-glass path and any console change, so keep both: GitOps for the planned flow, runtime governance for the exceptions.

How fast should the reconciler run against production?

Fast enough that drift cannot live unnoticed for long, but not so fast that it fights every legitimate emergency. Every one to five minutes is a common choice for prod: it bounds the drift window to a single interval while leaving a break-glass change enough time to be noticed and synced back into git. Pair the schedule with alerting on any non-empty diff, because the value of the check is telling a human that live and git disagree, not silently reverting and moving on.

What happens if the reconciler runs on a stale or bad merge?

The reconciler applies whatever is on the branch it watches, so a bad merge becomes a bad live state. Protect against this by treating the flag repo like production code: require CI to pass before merge, validate overlays against the base schema, and keep the reconciler pinned to a specific branch — usually main — rather than any branch. If a bad state does ship, the fix is the same as any code regression: revert the commit and let the reconciler apply the previous known-good state, which is exactly why keeping PRs small and single-purpose pays off.

Should each environment be its own repo, or overlays in one repo?

Prefer one repo with per-environment overlays. A single repo keeps the base definition shared, makes a promotion a diff between two overlay files in the same review, and gives you one history to blame. Separate repos per environment re-introduce the drift problem you are trying to solve — the base definition gets copied, the copies diverge, and a promotion becomes a cross-repo sync rather than a readable diff. Reach for separate repos only when hard access boundaries (for example, a prod repo a smaller group can merge to) genuinely require it.

How do I roll back a flag change made through GitOps?

Revert the merge commit and let the reconciler apply the restored state — that is the whole point of config-as-code. Because the previous value is recorded in git, git revert reproduces it exactly, the revert PR carries its own review trail, and on merge the reconciler converges live back to the known-good state. For an urgent rollback you cannot wait for a PR on, use the break-glass path to flip the flag live first, then open the revert PR so history stays complete.