Operational Safety and Incident Response

This guide is part of the Feature Flag Architecture & Lifecycle Management series. Feature flags are a safety mechanism — until the flag system itself becomes the thing that fails. This guide covers how to keep flag-driven systems safe when the control plane degrades, the provider stalls, or a rollout goes wrong: circuit breakers around evaluation, deliberate fallback defaults, kill-switch discipline, and an incident runbook you have actually rehearsed.

The uncomfortable truth is that a feature-flag layer is a distributed dependency you added to your own request path on purpose, and every dependency you add is a new way for the request to fail. The engineering work here is not “make the provider more reliable” — you rarely control that — it is “make your service correct and fast even when the provider is at its worst.” That reframing changes how you write every evaluation call: you stop asking “what value does this flag have?” and start asking “what does this service do when I cannot find out?” A team that has a crisp, per-flag answer to the second question sails through provider incidents that page other teams at 3am. Everything below is machinery for having that answer ready, wired into the code path, and proven under a deliberately broken provider rather than assumed.

Problem Framing

The promise of feature flags is that risky changes become reversible without a deploy. But that promise inverts when the flag infrastructure is a hard dependency on the request path: a control plane that times out, a provider that throws, or a poisoned rule set can now take down every service that reads a flag. The teams that get burned are the ones that treated the flag system as always-available and wrote evaluation code with no answer for “what happens when this call fails.”

This guide covers the resilience patterns that keep evaluation safe under failure and the human process for responding when something still goes wrong. It does not re-cover the mechanics of a kill switch runbook or the exponential backoff that governs reconnection — those have dedicated guides. Here the focus is making the whole evaluation path fail safe and rehearsing the response.

Three distinct failure shapes drive the design, and conflating them is the most common mistake. The first is latency: the provider is up but slow, so an unbounded read turns provider p99 into your service’s p99 — the quietest and most dangerous mode, because nothing throws and no alert fires until customers do. The second is hard error: the provider throws or refuses the connection, which is easy to catch but useless if the catch block re-raises or returns nothing usable. The third is wrong answer: the provider is fast and healthy but serving a poisoned rule set — a bad targeting expression pushed by an operator, a corrupted cache, a schema mismatch — so evaluation “succeeds” with a value that breaks the feature. Timeouts and breakers handle the first two; only defaults, staged rollout, and a fast kill switch handle the third. Design for all three or you have hardened one door and left two open.

Prerequisites

• Every flag has a documented safe default — the value the service uses when evaluation fails
• A local or cached rule set so evaluation survives a control-plane outage
• A metrics signal for provider health (see observability for flag evaluations)
• A kill switch path that does not depend on the same failing control plane
• An on-call rotation and a place to write the incident runbook

Core Concept & Architecture

The organizing principle is that flag evaluation must never be able to fail the request. Every call is wrapped so that a provider error, a timeout, or a missing flag resolves to a documented default instead of throwing. A circuit breaker sits in front of the provider so that once it is clearly unhealthy, calls stop hitting it and go straight to the local rule set or defaults — protecting both the service (no piled-up timeouts) and the provider (no thundering retries). The kill switch is deliberately built on a path that does not share the failure domain of the normal control plane, so it works precisely when everything else is broken.

// safe-evaluate.ts — evaluation that cannot fail the request
import { OpenFeature } from '@openfeature/server-sdk';
import { breaker } from './provider-breaker';

const DEFAULTS: Record<string, boolean> = {
  'web.checkout.new-summary': false,   // documented safe value on failure
  'ops.payments.kill-switch': false,
};

export async function safeBool(key: string, ctx: object): Promise<boolean> {
  if (breaker.isOpen()) return DEFAULTS[key] ?? false;   // skip an unhealthy provider
  try {
    return await withTimeout(
      OpenFeature.getClient().getBooleanValue(key, DEFAULTS[key] ?? false, ctx), 50);
  } catch (err) {
    breaker.recordFailure();
    return DEFAULTS[key] ?? false;      // never propagate — fail safe
  }
}

The default table is the contract: it states, per flag, the value that is safe when the truth is unavailable. For most flags that is false (the feature stays off), but for a flag that guards a fallback path the safe default may be true. Deciding this per flag, in advance, is the whole point — an undocumented default is a coin flip during an incident.

Notice a subtlety in the code that trips teams up: the SDK’s own getBooleanValue already takes a default as its second argument, and it returns that default on most internal errors. So why wrap it at all? Because the SDK default only covers the errors the SDK chooses to swallow — it does not bound latency, it does not stop a provider that is throwing on every call from being hammered, and it does not give you a single place to record the failure for the breaker. The wrapper is where those three concerns live. Treat the SDK default and your wrapper default as the same constant, sourced from the same metadata, so a provider that returns the default and a wrapper that returns the default produce identical behaviour — otherwise you get a hairline seam where “the flag failed open in one path and closed in the other,” which is precisely the kind of inconsistency that makes an incident hard to reason about.

One more architectural rule earns its keep: the breaker and the default table are process-local state, evaluated with no network hop, so the fallback path has to be self-sufficient. Do not reach for a remote cache or a config service inside the catch block — that just adds a second dependency that can be down at the same moment, for the same reason (a shared network partition, a saturated sidecar). The safe path must be reachable from memory alone. If you need the freshest possible rules under normal operation, keep them warm in the background with a streaming or polling sync that writes into that in-process store, so the read path never blocks on the network even on a cache miss.

Step-by-Step Implementation

Step 1 — Wrap evaluation with a timeout and default

Never call the provider without a timeout and a fallback. A flag read that can block indefinitely is a latent outage.

// with-timeout.ts
export function withTimeout<T>(p: Promise<T>, ms: number): Promise<T> {
  return Promise.race([
    p,
    new Promise<T>((_, rej) => setTimeout(() => rej(new Error('flag_eval_timeout')), ms)),
  ]);
}

Pitfall: a generous timeout (say 2s) on a hot path is worse than no flag at all — under provider slowness every request stalls for the full timeout. Keep it tight (tens of milliseconds) and lean on the local rule set.

Size the timeout against your own latency budget, not the provider’s happy-path number. If a healthy local evaluation resolves in well under a millisecond, a 50ms ceiling is already 50x headroom; there is no reason to grant 500ms. Remember that timeouts compose: a single request that reads eight flags with a 200ms timeout each can, in the worst case, add 1.6 seconds of tail latency before anything returns, so budget the aggregate flag time per request and consider batching reads or evaluating the whole context once. And be honest that Promise.race does not cancel the losing promise — the underlying provider call keeps running and still consumes a connection and a socket. That is fine functionally, but it means a slow provider can pile up in-flight calls faster than they drain; the circuit breaker in the next step is what actually stops that accumulation, which is why the timeout and the breaker are two halves of one defense rather than alternatives.

Step 2 — Add a circuit breaker in front of the provider

Trip the breaker after a threshold of failures so a struggling provider stops receiving traffic and the service serves defaults cleanly until it recovers. See circuit breakers for flag providers for the full implementation.

// provider-breaker.ts — trip after repeated failures, probe to recover
export const breaker = createBreaker({ failureThreshold: 5, windowMs: 10_000, halfOpenAfterMs: 5_000 });

Pitfall: a breaker with no half-open probe stays open forever after one bad minute. Always let it test recovery with occasional trial calls.

Tune the threshold against your provider’s normal error rate, not against zero. If the provider legitimately errors on 0.1% of calls, a breaker that trips on a single failure will flap open and closed constantly, serving stale defaults during healthy operation and adding noise that hides the real incident. Prefer a rolling-window or percentage-based condition — “open when more than 50% of the last 20 calls failed” — over a raw count, so a burst of unrelated timeouts during a traffic spike does not trip you. Make the breaker state per-process, not shared across the fleet through a coordination service: a global breaker adds exactly the cross-service dependency you are trying to remove, and one replica’s local view of provider health is a perfectly good signal for that replica’s own reads. Finally, emit a metric and, ideally, a log line on every state transition. A breaker that trips silently is a breaker you will not know tripped, and “we were serving defaults for forty minutes and nobody noticed” is a worse outcome than the outage that caused it, because it means a stale rollout kept shipping the wrong variant to real users.

Step 3 — Document a safe default for every flag

Make the safe default a required, reviewed field of flag metadata, and generate the default table from it so code and documentation cannot drift. See safe fallback defaults.

# flag-metadata.yaml — safe default is mandatory
web.checkout.new-summary: { safe_default: false, owner: checkout }
ops.db.read-replica-fallback: { safe_default: true, owner: platform }  # true keeps the fallback available

Pitfall: defaulting a fallback-guarding flag to false disables the fallback exactly when the primary is failing. Choose the default by asking “what is safe if we cannot know the real value?” not “what is the flag’s off state?”.

Generating the default table from metadata rather than hand-maintaining a constant in code closes the most common drift: a flag gets renamed or a new one is added, the code default is forgotten, and the SDK falls back to a language default like false that nobody chose. Fail the build if a referenced flag key has no safe_default field, so an undocumented default becomes a compile error instead of a production surprise. Extend the discipline to non-boolean flags, which teams routinely forget: a string flag that selects a payment processor, a number flag that sets a rate limit, and a JSON flag that carries a config blob all need a safe default too, and the safe value for a number is often “the most conservative bound” rather than “zero” — a rate limit that defaults to 0 under provider failure silently blocks all traffic. Record who owns each default and review it when the feature it guards changes behaviour, because a default that was safe when the feature launched can quietly become unsafe two quarters later when the fallback path it assumed gets deleted.

Step 4 — Rehearse the incident runbook

Write and then actually run a game day: cut the provider, confirm the service degrades gracefully, and time how long the kill switch takes to propagate. A runbook that has never been executed is a hypothesis. See the incident response runbook.

# gameday.sh — deliberately fail the provider and observe
iptables -A OUTPUT -p tcp --dport 8013 -j DROP   # block the control plane
# ...confirm requests still succeed on defaults, then restore and time recovery
iptables -D OUTPUT -p tcp --dport 8013 -j DROP

Run the drill against a realistic environment, not a laptop, and inject each of the three failure shapes separately: drop the connection outright (hard error), add latency with tc qdisc to a few hundred milliseconds (the slow-provider case the breaker exists for), and push a deliberately broken rule set to a canary (the wrong-answer case). Each surfaces a different weakness. Assign a scribe whose only job is to write down every manual step someone takes and how long it took — those notes are the next revision of the runbook, and the gap between what the document says and what people actually did is the most valuable output of the exercise. Time three things specifically: detection (how long until an alert fires or a human notices), decision (how long until someone is confident enough to pull the kill switch), and propagation (how long the switch takes to reach every replica). Most teams discover the decision phase dominates — the tooling reverses in seconds, but nobody was sure it was safe to act for twenty minutes. Rehearse the decision, not just the command.

Verification & Testing

Prove the safety properties hold by injecting failure in a test and asserting the request still succeeds on the documented default. The point of these tests is not coverage for its own sake — it is that the safety behaviour is the behaviour that runs least often in production and is therefore least likely to be exercised until the exact moment you depend on it. A default that was correct when written and silently broken by a refactor six months later is the classic way a “fail-safe” system fails closed. Pin each safety property with an assertion so a regression breaks CI, not a customer.

// safety.test.ts
test('provider failure resolves to the documented default, never throws', async () => {
  mockProvider.failWith(new Error('control plane down'));
  const value = await safeBool('web.checkout.new-summary', { targetingKey: 'u1' });
  expect(value).toBe(false);                 // the documented safe default
  expect(breaker.recordedFailure()).toBe(true);
});

test('an open breaker skips the provider entirely', async () => {
  breaker.forceOpen();
  const spy = jest.spyOn(mockProvider, 'resolve');
  await safeBool('web.checkout.new-summary', {});
  expect(spy).not.toHaveBeenCalled();        // straight to default, no provider hit
});

Add three more assertions that catch the subtle regressions. First, assert the timeout actually fires: mock a provider that never resolves and confirm safeBool returns within the timeout budget rather than hanging the test — a broken Promise.race passes the error test but hangs here. Second, assert the breaker recovers: after the failure window, force the half-open probe to succeed and confirm the next call reaches the provider again, so you catch a breaker that trips but never reopens. Third, run a property-style test over your whole default table asserting that every flag key referenced anywhere in the service has a safe_default entry, so a newly added flag with no documented default fails CI on the day it is introduced rather than the day the provider goes down. If you can, also assert that reading a flag never mutates shared request state on the failure path — a surprising number of “safe” defaults are computed inside a code path that also, say, decrements a quota, so the fallback quietly double-counts.

Troubleshooting & FAQ

Our provider had a blip and every service latency spiked. Why, if flags are local?

You were almost certainly calling the provider synchronously per evaluation without a tight timeout, so a slow provider became slow requests. Add a timeout and a circuit breaker, and evaluate against a local rule set kept fresh in the background.

What should the safe default be for a flag that enables a new feature?

Usually false — if you cannot determine the value, keep the new feature off and serve the known-good path. The exception is a flag that guards a fallback; there the safe default is whatever keeps the fallback available.

How often should we run a flag-system game day?

At least quarterly, and after any change to the evaluation path or provider. The value is in discovering that a “safe default” was never wired up, or that the kill switch shares a failure domain with the thing it is supposed to rescue — findings you only get by actually cutting the provider.

Should the circuit breaker be shared across the fleet or per-process?

Per-process, almost always. A shared breaker requires a coordination service, which reintroduces exactly the cross-service dependency you are trying to eliminate — now your safety mechanism can be taken out by a network partition. Each replica’s local view of provider health is a sufficient signal for its own reads, and the small window where different replicas disagree during a transition is harmless because every one of them resolves to the same documented default.

What is the difference between failing open and failing closed for a flag?

Failing open means evaluation resolves to “feature enabled” on error; failing closed means it resolves to “feature disabled.” Neither is universally safe — the correct direction depends entirely on what the flag guards. A flag that turns on an experimental checkout should fail closed, but a flag that guards a database read-replica fallback should fail open so the fallback stays reachable when the primary is down. This is why “safe default” is a per-flag decision recorded in metadata, not a global policy.

The provider recovered but our services kept serving defaults. What happened?

Almost certainly a circuit breaker with no half-open probe, or one whose recovery probe is failing for an unrelated reason. Once open, such a breaker never retests the provider and pins the service to defaults indefinitely. Confirm the breaker logs a state transition back to closed after recovery, and add a test that forces a successful half-open probe and asserts the next call reaches the provider. A breaker stuck open is a silent incident: the service is healthy but shipping stale flag values to real users.

How should safety behave when a flag read is inside a hot loop or fan-out?

Resolve the flag once per request, not once per iteration. A read that is cheap in isolation becomes a latency amplifier when it runs thousands of times inside a loop, and under provider slowness each of those reads can independently wait out the timeout. Hoist the evaluation to the top of the request, pass the resolved value down, and batch or bulk-evaluate the whole context where the SDK supports it so a fan-out that touches many flags makes one round trip rather than many.

Can a healthy provider still cause an incident?

Yes — this is the wrong-answer failure mode. The provider is fast and returns successfully, but it is serving a poisoned rule set: a malformed targeting expression, a corrupted cache entry, or a config push that flips a flag to the wrong variant fleet-wide. Timeouts and breakers do nothing here because nothing is slow or throwing. Your defenses are staged rollout of rule changes, validation of rule sets before they publish, observability that watches variant distribution for sudden shifts, and a kill switch fast enough to reverse a bad push in seconds.

Performance & Scale Considerations

The safety layer is nearly free when the provider is healthy — a timeout wrapper and a breaker check are microseconds — and it earns its cost precisely when the provider is not. The scale consideration is the failure mode: without a breaker, a degraded provider multiplies across every replica and every evaluation into a fleet-wide latency spike, because each request independently waits out its timeout. The breaker collapses that to a single fast path to defaults. Size the timeout to your latency budget, the breaker threshold to your provider’s normal error rate, and keep the local rule set warm so the fallback path is as fast as the happy path. The distributed caching and reducing evaluation latency guides cover keeping that local path fast.