Warming the Flag Cache on Deploy

This how-to is part of Distributed Caching for Flag Evaluations. Right after a deploy, every fresh instance has an empty flag cache, so the first requests each pay a cold miss and stampede the provider at once — a latency spike exactly when the new version is most fragile. This how-to warms the cache during startup so the first real user request hits a populated cache, and coordinates warmup across a fleet so the provider is not hammered.

The cost of skipping this is easy to underestimate. A cold miss is not one slow request — it is every distinct flag key evaluated for the first time on a fresh instance, each blocking on a network round trip to the provider while the rule set is absent. On a service that evaluates forty or fifty flags across its endpoints, that is dozens of synchronous provider calls fanning out from a single instance in the first few seconds of traffic, multiplied by however many instances the rollout brought up at once. The blast radius lands precisely when your dashboards are already noisy from the deploy itself, which is why cold-start latency is so often misdiagnosed as “the new build is slow” when the real culprit is an empty cache. Warming turns that unbounded fan-out into one predictable bulk fetch that happens off the critical path, before any user is watching.

Prerequisites

• A local evaluation cache or rule set the instance populates at startup (see distributed caching)
• A readiness probe that gates traffic until warmup completes
• The set of flags the instance will actually evaluate, or a way to fetch the full set
• A deploy strategy (rolling, blue-green) whose ordering you can rely on
• A provider that can serve a bulk fetch of the rule set

Step-by-Step Procedure

Step 1 — Fetch the rule set at startup

On boot, pull the full flag rule set from the provider into the local cache before serving anything. One bulk fetch is far cheaper than N cold misses — it amortizes a single connection setup, TLS handshake, and provider round trip across every flag instead of paying them per key. Prefer an endpoint that returns the entire rule document (flagd’s bulk sync or a provider’s “get all flags” call) over iterating keys one at a time; iterating defeats the purpose because you are back to N calls, just moved earlier in the lifecycle. Make the fetch idempotent and side-effect-free so a warmup that partially fails can simply be retried whole, and validate the payload before you swap it into the cache — loading a truncated or malformed rule set is worse than a cold cache, because it fails evaluations silently instead of loudly.

# warmup.py — populate the local rule set at boot
async def warm_cache(provider, cache) -> bool:
    rules = await provider.fetch_all_rules()   # one bulk call
    cache.load(rules)
    return cache.size() > 0

Step 2 — Gate readiness on warmup completion

Report the instance not-ready until warmup succeeds, so the load balancer withholds traffic until the cache is populated. This is the step that actually moves the first-load cost off the user’s request — everything else is just making that cost small. Keep the readiness signal distinct from liveness: liveness answers “is the process alive?” (yes, from the moment it starts), while readiness answers “should this instance receive traffic yet?” (no, until warm). Conflating them is a classic mistake — if warmup blocks liveness, a slow provider fetch can make the orchestrator conclude the pod is dead and restart it in a crash loop, so warmup never completes. In Kubernetes terms, gate the readinessProbe on warm_done and leave the livenessProbe answering on process health alone, with a startupProbe giving warmup enough grace before either kicks in.

# readiness.py
warm_done = False

async def startup():
    global warm_done
    warm_done = await warm_cache(provider, cache)

def readiness_probe():
    return 200 if warm_done else 503   # 503 keeps traffic away until warm

Step 3 — Stagger warmup across the fleet

Add jitter so a hundred instances rolling out together do not fetch from the provider at the same instant. This prevents the warmup itself from becoming a stampede. The failure mode is subtle: without jitter you have not removed the thundering herd, you have merely relocated it from the first user request to the startup path, where it can knock over the provider just as effectively. Size the jitter window against your rollout velocity — if a rolling deploy brings up ten pods every few seconds, a jitter of two to five seconds is plenty to smear the fetches; a blue-green flip that lights up the entire green fleet at once needs a wider window, or better, a shared cache tier the instances warm from instead of the origin provider (see Redis vs Memcached). Jitter alone is a probabilistic smoothing, not a hard rate limit, so pair it with a bounded concurrency or a request coalescer at the provider if a stray simultaneous burst would still hurt.

# stagger.py — jittered warmup start
import random
async def staggered_startup():
    await asyncio.sleep(random.uniform(0, WARMUP_JITTER_S))  # spread the fetches
    await startup()

Step 4 — Fall back if the provider is unavailable at boot

If the provider cannot be reached during warmup, start with bundled defaults or the last persisted rule set rather than failing to start, and keep retrying in the background. The principle is that a flag system should degrade toward availability, not away from it — an outage in the control plane must never take down the data plane it was meant to make safer. Bundled defaults should encode the conservative choice for every flag (usually “feature off, serve the stable path”), so that a degraded boot is a known-safe boot rather than a coin flip. A persisted snapshot written on each successful sync is strictly better than static defaults because it reflects the last-known real rule set, but it can be arbitrarily stale, so treat any evaluation served from it as provisional and surface that state — a metric like flag_cache_source{source="snapshot"} — so you can see when a fleet is running on fallback rather than live rules. The background retry should use capped exponential backoff and, on success, atomically swap the live rules in without a second cold window.

# fallback.py
async def warm_with_fallback(provider, cache):
    try:
        return await warm_cache(provider, cache)
    except ProviderUnavailable:
        cache.load(load_last_known_rules() or BUNDLED_RULES)  # start degraded, not down
        schedule_background_retry(warm_cache)
        return True   # ready on defaults; refresh will upgrade later

Verification

Deploy and watch the first-request latency and provider request count. A warmed deploy shows flat first-request latency and a single bulk fetch per instance instead of a burst of misses. Do not settle for eyeballing the p99 chart — verify the two signals that actually prove warming worked. First, the provider request count during a rollout should equal roughly one bulk fetch per new instance, not a spike proportional to early traffic; if it scales with request volume, warming is not running or the probe is passing early. Second, the cache-hit ratio measured on the very first requests each instance serves should already be near one — a warmed instance that starts at a low hit ratio and climbs is a sign you warmed only a subset. Run the comparison as a controlled experiment: deploy once with warming disabled to capture the cold baseline, then enable it, so the flat line is measured against a real spike rather than assumed.

# Compare p99 of the first requests after a rolling deploy
promtool query instant http://prom:9090 \
  'histogram_quantile(0.99, rate(flag_eval_duration_seconds_bucket[1m]))'
# Warmed: flat. Cold: a spike in the first minute after each instance joins.

Gotchas & Edge Cases

• Warmup can stall the rollout. If warming fetches a huge rule set slowly, instances take too long to become ready and the deploy drags. Cap warmup time and fall back to defaults (step 4) past the cap rather than blocking the rollout indefinitely. The cap interacts with your orchestrator’s deadline: if the startupProbe grace period expires before warmup finishes, the pod is killed and restarted, so the effective warm budget is the smaller of your cap and that grace window — size them together, not independently.
• Warming a subset still leaves cold misses. If you warm only the “common” flags, the first request for any other flag still misses. Warm the full set the instance can evaluate, or accept and monitor the residual misses explicitly. Subset warming is tempting when the rule set is large, but “common” is measured from yesterday’s traffic and a deploy is often precisely when the traffic mix shifts — the new build may light up flags that were rare before, so the flags you skipped are the ones most likely to miss.
• Readiness must reflect real warm state. A probe that returns 200 on process start, before warmup completes, defeats the whole exercise. Wire the probe to the actual warmup-complete flag, not merely to the server accepting connections. This is the single most common way a “warmed” deploy still spikes, because the HTTP server binds and starts answering long before the rule set is loaded.
• A stale snapshot can serve wrong rules confidently. Falling back to a persisted snapshot keeps you available, but if that snapshot predates a flag change made during an incident, you may re-enable something you just turned off. Stamp snapshots with the provider version or timestamp, refuse ones older than a sane threshold, and prioritize the background refresh so the stale window is measured in seconds, not minutes.
• Warming and cache invalidation must agree on freshness. If warmup loads a snapshot at boot while an invalidation event fires during startup, the instance can come up already stale, having missed the update it was booting through. Subscribe to the invalidation channel before you finish warming, or re-check the rule version at the moment readiness flips, so a change that lands mid-warmup is not silently lost.

Troubleshooting & FAQ

Latency still spikes on deploy even with warming. Why?

The readiness probe is probably passing before warmup finishes, so traffic arrives on a still-cold instance. Confirm the probe returns 503 until warm_done is true, and check that your orchestrator actually respects the probe before routing traffic.

Does warming help with polling or streaming sync?

Yes — warming populates the initial rule set that polling or streaming then keeps fresh. Without warmup, the instance is cold until the first poll completes; warming closes that gap so it starts fresh and stays fresh.

Should I warm from the provider or from a persisted snapshot?

Prefer the provider for correctness (freshest rules), but fall back to a persisted snapshot if the provider is unreachable at boot. The snapshot gets you a warm, if slightly stale, cache immediately, and the background retry upgrades it once the provider is reachable.

How long should warmup take, and when is it too slow?

Warmup is a single bulk fetch plus a cache load, so it should complete in well under a second against a healthy provider — hundreds of milliseconds is typical even for a few thousand flags. If it routinely takes seconds, either the rule set is genuinely enormous or you are iterating keys instead of fetching in bulk. Treat anything approaching your startupProbe grace window as too slow, because past that point the orchestrator will start killing pods mid-warmup and you trade a latency spike for a restart loop.

Does warming matter for serverless or scale-to-zero functions?

It matters more, not less. A function that scales to zero pays a cold start on nearly every burst of traffic, so there is no long-lived instance to amortize the fetch across. The mitigations differ: you cannot rely on a readiness probe, so warm inside the init phase (module scope in the function runtime) so the fetch is reused across invocations on the same warm container, and lean harder on bundled defaults or a shared cache tier because the per-invocation budget is unforgiving.

Can I warm the cache before the deploy instead of during startup?

You can pre-seed a shared cache tier ahead of the rollout — populate Redis with the current rule set before flipping traffic — so each new instance warms from a local hop rather than the origin provider. That shrinks the per-instance warm time and the load on the provider, but it does not remove per-instance warming: each process still needs its in-memory cache populated. Think of pre-seeding as making step 1’s fetch cheaper and safer, not as a replacement for gating readiness on it.

Should warmup block on every flag or just the flags this service reads?

If you can reliably enumerate the flags a service evaluates, warming only those is legitimate and faster — the caveat from the gotchas applies: your enumeration must be exhaustive, including flags reached on rare code paths, or those keys still cold-miss. When you cannot prove the set is complete, warm everything. A missed bulk fetch of the whole rule set costs a little more memory and one slightly larger response; a missed flag costs a live cold miss on real traffic.