Implementing Circuit Breakers for Flag Providers

Q: The breaker opens but the service still slows during an outage. Why?

The timeout is probably too long and the threshold too high, so many requests fail slowly before the breaker trips. Tighten the timeout and lower the threshold so it trips before slow calls accumulate.

Q: What happens to in-flight evaluations when the breaker opens?

They complete or time out on their own; the breaker only affects new calls, routing them to defaults. In-flight calls are already bounded by their timeout.

Q: Do I need a circuit breaker if my SDK already caches flags locally?

Yes; they solve different problems. A local cache serves the last known value while the provider is unreachable, but without a breaker each read still attempts the provider and waits out the timeout. Put the breaker in front and let the open path fall back to the cache first, then a static default.

Q: How is this different from retrying a failed provider call?

A retry helps a single request survive a transient blip; a breaker helps the whole service survive a sustained outage by refusing to call at all. Retrying every call during a real outage multiplies load on the provider, so scope retries to one attempt and let the breaker handle correlated failures.

Q: What failure threshold and window should I start with?

A reasonable starting point for a low-latency internal provider is about five failures over a two-second window with a five-second cooldown, then tune against reality. Raise the threshold if it flaps in normal operation; lower it or tighten the timeout if the service slows before the breaker trips.

Q: Should the breaker fail open or fail closed?

For a flag provider it should almost always fail open: stop calling the provider and serve reviewed defaults so the service keeps running. A flag read is a soft dependency, so the correct degraded behavior is a sensible default rather than an error, provided the defaults reflect the safe choice.

This how-to is part of Operational Safety and Incident Response. Your flag provider starts timing out and every request that reads a flag now waits out the full timeout before falling back — a slow provider has become a slow service. This how-to builds a circuit breaker that trips after repeated failures so calls go straight to local defaults, then probes for recovery, protecting both your latency and the struggling provider.

The distinction that matters is between a timeout and a breaker. A timeout bounds one call: it caps the damage a single slow evaluation can do at, say, 50ms. But if the provider is down for two minutes and you serve 10,000 requests per second, that is 1.2 million requests each eating a 50ms wall — the tail latency of the whole service climbs even though every individual call is technically bounded. The breaker exists to notice that the failures are correlated and stop paying the 50ms toll on every one of them. Once tripped, an open breaker resolves a flag read in microseconds because it never touches the network at all. That is also a gift to the provider: a degraded control plane recovering from an outage does not need thousands of clients hammering it with retries at the exact moment it is trying to come back.

Closed is normal operation; a burst of failures opens the breaker to serve defaults; after a cooldown it half-opens to probe, closing on success or re-opening on failure.

Prerequisites

A documented safe default per flag to serve while the breaker is open
A timeout already wrapping every provider call (an open breaker complements, not replaces, timeouts)
A provider-health metric to observe breaker state transitions
A local or cached rule set as the fallback source, where available
The operational safety evaluation wrapper in place

The breaker sits on top of timeouts, not instead of them — the timeout bounds a single slow call, the breaker stops a run of them.

Step-by-Step Procedure

Step 1 — Track failures in a rolling window

Count failures within a time window so a slow trickle of unrelated errors does not trip the breaker, but a genuine burst does. The rolling window is what separates a rate from a count — a fixed count of, say, five lifetime failures would eventually trip on a healthy provider that has simply been running long enough to accumulate a few transient errors. By filtering out timestamps older than windowMs on every record, you are really asking “have there been five failures in the last N seconds?”, which tracks the provider’s current health rather than its history.

A word on what counts as a failure. Be deliberate about which errors increment the counter: a timeout, a connection refusal, or a 5xx from the provider are all signals of a degraded control plane and should trip the breaker. A FLAG_NOT_FOUND or a malformed-context error is a bug in your call, not a provider outage — counting it would open the breaker on a mistake that serving defaults will never fix. Filter for the transport-level and availability errors and let application errors surface normally.

// breaker.ts — rolling-window failure tracking
export function createBreaker({ failureThreshold, windowMs, halfOpenAfterMs }: BreakerCfg) {
  let failures: number[] = [];          // timestamps of recent failures
  let openedAt = 0;
  const now = () => Date.now();

  function recordFailure() {
    const t = now();
    failures = failures.filter(ts => t - ts < windowMs);
    failures.push(t);
    if (failures.length >= failureThreshold) openedAt = t;   // trip
  }
  return { recordFailure, /* ...state getters below */ } as Breaker;
}

Step 2 — Expose the tri-state, including half-open

Derive the state from the timers: open right after tripping, half-open once the cooldown elapses, closed otherwise. Half-open is what lets exactly one probe through. Deriving state from timers rather than storing it as a mutable field is a deliberate simplification — there is no background timer to cancel, no setTimeout to leak, and the state is always a pure function of the current clock. A read of state() costs a subtraction and a comparison, which matters when it sits on the hot path of every flag evaluation.

The half-open window has one sharp edge worth naming: you want exactly one probe in flight, not one probe per cooldown. If two requests both observe half-open before either records a result, both will hit a provider you already suspect is fragile. Under low traffic this rarely bites, but a high-QPS service can leak a handful of concurrent probes in the microseconds between the state flip and the first recordSuccess. If that matters, guard the probe with a compare-and-swap flag so the first caller claims the probe slot and the rest fall through to defaults until the result lands.

// breaker.ts (continued)
function state(): 'closed' | 'open' | 'half-open' {
  if (!openedAt) return 'closed';
  return (now() - openedAt >= halfOpenAfterMs) ? 'half-open' : 'open';
}
function isOpen() { return state() === 'open'; }             // skip provider entirely
function allowProbe() { return state() === 'half-open'; }    // let one trial call through

Step 3 — Gate provider calls on breaker state

In the evaluation wrapper, skip the provider when open, allow a single probe when half-open, and on a successful probe reset the breaker to closed. Notice that the open path returns the default before the try block — there is no await, no network, and no timeout to wait out, which is the whole point. The default itself comes from the same DEFAULTS map your safe fallback defaults work produced, so an open breaker degrades to a known, reviewed value rather than to the bare false a naive ?? false would hand back.

One subtlety in recordSuccess: resetting on a single successful probe is optimistic, and for most providers that is the right call — a provider that answers one request cleanly after a cooldown is almost certainly back. If you have seen a provider flap (answer one request, then fail the next ten), require two or three consecutive successes in half-open before closing. The cost is a slightly longer recovery; the benefit is you do not slam the closed breaker back open the instant real traffic resumes.

// safe-evaluate.ts
async function safeBool(key: string, ctx: object): Promise<boolean> {
  if (breaker.isOpen()) return DEFAULTS[key] ?? false;       // open: straight to default
  try {
    const v = await withTimeout(client.getBooleanValue(key, DEFAULTS[key] ?? false, ctx), 50);
    breaker.recordSuccess();                                  // half-open probe succeeded -> close
    return v;
  } catch {
    breaker.recordFailure();
    return DEFAULTS[key] ?? false;
  }
}

Step 4 — Emit state transitions as metrics

Record every open, half-open, and close so you can see the breaker working during an incident and alert on prolonged open states. Encode the state as a numeric gauge — closed: 0, half-open: 1, open: 2 — rather than a string label, because a gauge lets you write range queries and max_over_time alerts directly. Label the metric by provider so a fleet running several providers (flagd for one namespace, a SaaS control plane for another) does not collapse into a single indistinguishable line.

The alert that earns its keep is the prolonged open, not the transient one. A breaker that opens for three seconds during a deploy is working exactly as designed and should page no one; a breaker sitting at 2 for ten minutes means every replica has given up on the provider and the whole service is now running on defaults with no live targeting. That second condition is the incident — the max_over_time(flag_breaker_state[10m]) == 2 expression is deliberately tuned to fire only when the open state persists across the full window. Pair it with a lower-severity signal on transition rate so a breaker that flaps open and closed every few seconds is visible before it turns into a sustained outage.

// on each transition
breakerState.set({ provider: 'flagd' }, { closed: 0, 'half-open': 1, open: 2 }[state()]);
// Alert: max_over_time(flag_breaker_state[10m]) == 2  -> provider down for 10m

The half-open state in step 2 is the whole trick — without it the breaker never learns the provider recovered and stays open indefinitely.

Verification

Drive the breaker through all three states in a test: fail enough calls to open it, wait the cooldown to half-open, then succeed a probe to close it. The load-bearing detail is that time is injected, not read from the real clock — the test uses a fake clock.advance(5_000) to jump past halfOpenAfterMs instantly. If your breaker calls Date.now() directly, this test either sleeps for real (slow, flaky) or cannot be written at all. Inject a now() function the way the createBreaker example does and the whole state machine becomes deterministic to test.

Do not stop at the happy cycle. Add a case where the probe fails and assert the breaker returns to open with the cooldown clock reset — this is the transition the dashed arrow in the diagram represents, and it is the one that a subtly wrong implementation gets wrong by leaving the breaker stuck in half-open forever. A second worthwhile case: record failures spread across a span longer than windowMs and assert the breaker never opens, proving the rolling-window filter actually evicts stale timestamps rather than accumulating them.

// breaker.test.ts
test('opens after threshold, half-opens after cooldown, closes on probe success', async () => {
  for (let i = 0; i < 5; i++) breaker.recordFailure();
  expect(breaker.state()).toBe('open');
  clock.advance(5_000);                       // past halfOpenAfterMs
  expect(breaker.state()).toBe('half-open');
  breaker.recordSuccess();                    // probe succeeds
  expect(breaker.state()).toBe('closed');
});

Testing the full cycle — not just the trip — is what catches a breaker that opens but can never recover.

Gotchas & Edge Cases

The probe storm is the one that can prolong an outage — if every replica probes at the same instant, the recovering provider is hit by a synchronized wave.

Breaker state is per replica. Each process has its own breaker, so a rolling provider outage trips replicas at different times. That is correct and desirable — you do not want one shared breaker to blackout the whole fleet on one process’s bad luck.
Synchronized probes create a storm. If every replica half-opens at exactly the cooldown boundary, the recovering provider gets a simultaneous burst. Add jitter to halfOpenAfterMs so probes spread out, the same principle as exponential backoff with jitter.
Too low a threshold flaps. A threshold of one or two failures trips on normal transient errors and never gives the provider a chance. Set it above the provider’s baseline error rate over the window — if the provider naturally returns one error every few seconds under load, a threshold of three over a two-second window will flap constantly. Base the number on observed error rates, not a round guess.
The cooldown is a latency-versus-load trade-off. A short halfOpenAfterMs recovers fast but probes an unhealthy provider often; a long one shields the provider but keeps you on defaults longer than necessary. A cooldown in the 2–10 second range suits most control planes — long enough that a probe reflects a real recovery, short enough that a brief blip does not strand you on stale defaults for a minute.
An open breaker hides staleness, not just failure. While the breaker is open you are serving defaults, which means any flag change shipped during the outage is invisible until the breaker closes. If an incident responder flips a kill switch expecting it to take effect, a locally-open breaker will happily keep serving the old default. Surface breaker state in your runbook so responders know the control plane is not being consulted right now.
Success must also prune the window. If recordSuccess only closes the breaker without clearing the failure array, stale failure timestamps can linger and re-trip the breaker prematurely on the next single error. Reset the failure count on a successful close so the breaker starts the next window clean.

Troubleshooting & FAQ

The breaker opens but the service still slows during an outage. Why?

The timeout is probably too long, so each request before the breaker trips still waits it out, and the breaker’s threshold is high enough that many requests fail slowly first. Tighten the timeout and lower the threshold so the breaker trips before the slow calls accumulate.

Should the breaker be shared across replicas via a central store?

Usually not. A shared breaker adds a network dependency to the very path you are protecting and can blackout the fleet on one replica’s failures. Keep breaker state local; use aggregate metrics to see fleet-wide health.

What happens to in-flight evaluations when the breaker opens?

Nothing special — they complete or time out on their own. The breaker only affects new calls, routing them to defaults. In-flight calls are already bounded by their timeout.

Do I need a circuit breaker if my SDK already caches flags locally?

They solve different problems and compose well. A local cache serves the last known value even while the provider is unreachable, which is often a better fallback than a static default. The breaker’s job is to stop paying the latency cost of calls that are going to fail — cache or no cache, without a breaker each read still attempts the provider and waits out the timeout. Put the breaker in front, and let the open path fall back to the cache first and the static default only if the cache is empty.

How is this different from retrying a failed provider call?

Retries and breakers pull in opposite directions, which is exactly why you need both. A retry helps a single request survive a transient blip by trying again. A breaker helps the whole service survive a sustained outage by refusing to try at all. Retrying every call during a real outage multiplies load on the struggling provider — the breaker is what caps that, so scope retries to one attempt and let the breaker handle the correlated failures.

What failure threshold and window should I start with?

There is no universal number, but a reasonable starting point for a low-latency internal provider is a threshold of about five failures over a two-second window, with a cooldown around five seconds. Then tune against reality: if the breaker flaps during normal operation, raise the threshold or lengthen the window; if the service still slows noticeably before the breaker trips, lower the threshold or tighten the per-call timeout.

Should the breaker fail open or fail closed?

For a flag provider it should almost always fail open — meaning it stops calling the provider and serves your reviewed defaults, keeping the service running. “Fail closed” (blocking requests entirely when the dependency is down) makes sense for a hard dependency like auth, but a flag read is a soft dependency: the correct degraded behavior is a sensible default, not an error. Make sure your defaults reflect the safe choice so failing open never enables a half-built feature.