Designing Safe Fallback Defaults for Flag Outages
This how-to is part of Operational Safety and Incident Response. When the provider is unreachable, every flag resolves to its coded default — and if those defaults were chosen carelessly, an outage can disable the very fallback that would save you or expose a half-finished feature to everyone. This how-to gives you a rule for choosing each flag’s safe default deliberately, encoding it in metadata, and proving it under a simulated outage.
The failure that motivates all of this is quiet: the default value is the one line of flag configuration that almost never runs, so it is almost never tested. A flag can sit at 100% rollout for a year while its coded default says false, and nobody notices the mismatch until the provider drops and every user snaps back to the old code path mid-session. The goal here is to make that default an explicit, owned, and asserted decision rather than a leftover argument someone typed to satisfy a function signature. Treat the default as the behavior your system exhibits during its worst five minutes, because that is exactly when it fires.
Prerequisites
Step-by-Step Procedure
Step 1 — Classify each flag by what it controls
Tag each flag as feature-enabling, fallback-guarding, or routing. The tag determines the direction of the safe default. This classification is the whole trick — once a flag has an honest purpose tag, the default is no longer a judgement call made under pressure but a lookup. The mistake teams make is skipping the tag and reasoning about defaults flag-by-flag from memory, which works until the person who remembers what platform.db.read-replica-fb guards leaves the team. Write the purpose down next to the flag so the reasoning survives the author.
Be honest about the awkward cases. A flag named after a feature can secretly guard a fallback — web.checkout.legacy-summary sounds like a feature but its off state removes the old, reliable summary. Classify by what the off state does to the user, not by what the flag is named after. If you cannot state in one sentence what breaks when the flag is off, you do not yet understand it well enough to pick its default.
# flags.yaml — classify by purpose
web.checkout.new-summary: { purpose: feature } # off is safe
platform.db.read-replica-fb: { purpose: fallback-guard } # on is safe
web.api.route-to-v2: { purpose: routing } # route to the proven targetStep 2 — Derive the safe default from the class
Apply the rule: feature flags default off, fallback-guards default on, routing defaults to the known-good target. Encode the derived value so it is explicit, not implied. The word “known-good” for routing is load-bearing: the safe target is not the newest version or the geographically closest one, it is the one you have the most operational history with. During a provider outage you have already lost your ability to change routing, so the default must point at whatever you would be comfortable serving unattended for an hour.
Deriving rather than hand-writing each value buys you one more thing — the rule itself becomes reviewable. When a reviewer sees a flag whose purpose is feature but whose stored default is true, that contradiction is now visible in the diff instead of buried in a call site three files away. Keep the derivation a pure function of the purpose so it has no hidden inputs; a default that depends on environment or time of day is a default you cannot reason about at 3 a.m.
# derive_default.py
def safe_default(purpose: str) -> bool:
return {
"feature": False, # keep the new thing off
"fallback-guard": True, # keep the safety net up
"routing": False, # False routes to the proven path
}[purpose]Step 3 — Wire the default into evaluation
Pass the metadata-derived default as the fallback argument on every evaluation, and generate a default table so code cannot drift from the documented value. Every SDK’s evaluation call takes a default as a required argument for exactly this reason — OpenFeature’s getBooleanValue(key, default, ctx) will not let you omit it — but a required argument only guarantees a value is present, not that it is the right one. Generating the table closes that gap: the argument is now a reference into a single source of truth, so the compiler and the metadata move together.
Resist the temptation to centralize evaluation behind a helper that silently supplies false for any missing key. That pattern makes the outage path uniform but wrong — a fallback-guard flag that should default on will now default off because the helper never consulted the flag’s purpose. The default belongs to the flag, not to the wrapper; the wrapper’s only job is to apply whatever the flag declared and to catch the provider exception without letting it escape.
// defaults.ts — generated from flags.yaml, single source of truth
export const DEFAULTS: Record<string, boolean> = require('./generated-defaults.json');
// usage: client.getBooleanValue(key, DEFAULTS[key], ctx)Step 4 — Make the default a reviewed, required field
Fail CI if any flag lacks a safe_default, so a new flag cannot ship without someone deciding what happens during an outage. The gate is cheap and the payoff is that the decision moves left, to the pull request where the flag is born and the author still has full context, instead of surfacing during the incident when nobody remembers why the flag exists. A missing default is not a neutral state that resolves to false — it is an undecided state, and undecided states are how outages surprise you.
Make the check enforce the derivation too, not just presence. A flag can carry a safe_default that flatly contradicts its purpose — a feature flag defaulting on — and a presence-only gate waves it through. Have CI recompute the expected default from the purpose and fail on any mismatch, so the two fields can never disagree in main.
# ci-defaults.sh
missing=$(yq '.[] | select(.safe_default == null) | key' flags.yaml)
[ -n "$missing" ] && { echo "flags missing safe_default: $missing"; exit 1; } || trueVerification
Cut the provider in a test and assert every flag resolves to its documented safe default with no exceptions thrown. The two halves of that sentence catch different bugs. Asserting the exact value catches a code default that was set once and never re-synced with the metadata; asserting that no exception escapes catches the more insidious failure where the evaluation wrapper lets the provider’s connection error propagate, so instead of a safe default the caller gets a stack trace and a 500. A “safe default” that only exists on the happy path is not a default at all.
Run this test against a total outage and a slow one. A provider that answers in 30 seconds is often worse than one that is cleanly down, because a naive wrapper will block the request thread until the client’s own timeout fires rather than falling back promptly. Assert that the fallback returns within your evaluation budget — a few milliseconds — not merely that it eventually returns the right value. Pair this with the timeout behavior described in Implementing Circuit Breakers for Flag Providers.
// fallback.test.ts — outage resolves each flag to its safe default
test('all flags resolve to safe_default under provider outage', async () => {
mockProvider.failAll();
for (const [key, cfg] of Object.entries(flags)) {
const v = await safeBool(key, { targetingKey: 'u1' });
expect(v).toBe(cfg.safe_default); // exactly the documented value
}
});Gotchas & Edge Cases
- Hardcoded defaults drift. A default typed into a call site diverges from the metadata the moment the metadata changes. Generate the default table from the source of truth and reference it, never inline literals. The drift is silent because both the literal and the metadata are individually valid — nothing is null, nothing throws — so only a generated table plus the verification test will surface the disagreement before an outage does.
- A kill switch needs a considered default. If the outage disables your kill switch, you lose the ability to stop a bad feature during the very incident that broke the provider. Decide explicitly whether the switch’s safe default keeps it armed. A kill switch that means “off = feature killed” should default to its safe position, which for a switch you might need mid-incident usually means leaving the feature it guards in its conservative state rather than trusting a provider you cannot reach. See the kill switch runbook.
- Partial degradation is common. Real outages often affect some flags and not others (one namespace, one region). Do not assume all-or-nothing; the safe default must be correct in isolation because a single flag can fall back while its neighbors resolve normally. This is where defaults chosen “as a set” bite you — two flags that are only safe when they agree will happily disagree when one falls back and the other stays live, producing a state your code has never been exercised in.
- Non-boolean defaults hide sharp edges. String and JSON flags — a config blob, a rollout percentage, a variant name — need a default that is not just present but structurally complete. A default of
{}for a config flag can fail deeper in the code than a missing flag would, because it passes the null check and then blows up on a missing nested key. Make the default a real, valid instance of the shape the code expects, and put it under the same verification test as the booleans. - The default is not a place to be clever. Resist encoding “smart” fallback logic — reading a second provider, guessing from a cache — into the default path. The default runs precisely when your infrastructure is already impaired; every dependency it touches is another thing that can be down at the same moment. A dumb, constant, locally-computable default is worth more at 3 a.m. than an intelligent one that needs the network.
Troubleshooting & FAQ
Should the safe default match the flag’s current production value?
No — that couples your outage behavior to the current rollout state, which changes constantly. The safe default answers a different question: what is safe when we cannot know the value? Pin it to that, independent of the live setting.
A flag guards an expensive feature. Is off always the safe default?
For cost, usually yes — falling back to off avoids running the expensive path when you cannot confirm it should run. But if off removes a safety behavior (say, extra validation), weigh correctness over cost and default it on. Classify by what off actually does.
How do I keep hundreds of safe defaults correct over time?
Treat them as reviewed metadata with an owner per flag, generate the code table from that metadata, and assert the outage behavior in CI (the verification test). The combination makes an incorrect or missing default a build failure, not a production surprise.
What is the safe default for a string or JSON flag, not just a boolean?
A structurally valid, minimal instance of whatever shape the code consumes — never an empty object or empty string unless the code genuinely treats that as complete. For a config blob, the safe default is the last known-good config baked into the release, not {}; for a variant name, it is the control variant’s exact string. The test is the same as for booleans: cut the provider and assert the flag resolves to that documented value.
Should the safe default ever depend on the environment or region?
Prefer not. A default that varies by environment is a default you cannot reason about globally, and staging will lull you into trusting behavior production never runs. If a genuine regional difference exists — a feature is legally unavailable in one region — encode that as separate flags with separate purposes rather than as a conditional inside one default, so each one stays a pure, reviewable constant.
Where should the fallback default live — the SDK call, a wrapper, or the flag definition?
Conceptually it belongs to the flag definition; mechanically it is applied at the SDK call. Generate the value from the flag’s metadata and pass it into getBooleanValue/getStringValue at the call site through a generated table. The wrapper should apply the declared default and swallow the provider exception, but it must never invent a blanket default of its own — that is how a fallback-guard flag silently ends up defaulting off.
How often should I revisit a flag’s safe default?
Whenever the flag’s purpose changes and at the moment you would otherwise retire the flag. A flag that flips from feature-enabling to permanent-config has a different safe default than it started with, and a flag left at 100% for months is a prime candidate for its default to have quietly drifted. Fold the review into the same cleanup pass you use for stale flags rather than scheduling a separate audit nobody will run.