Local vs Remote Evaluation Latency Trade-offs
This how-to is part of Choosing a Flag Evaluation Strategy. You have decided a flag is evaluated server-side; now you choose how: locally, against a rule set already in memory, or remotely, by calling the control plane per evaluation. Local is sub-millisecond but only as fresh as its last sync; remote is always fresh but adds a network round-trip to every call. This how-to shows you how to measure both and pick per flag from a real budget.
The temptation is to pick one mode for the whole service and move on. Resist it — the right answer is almost always a mix, because the two costs you are trading, latency and staleness, are not properties of your service but of each individual flag. A pricing experiment read once per checkout can absorb a 25 ms remote call; the same 25 ms multiplied across a flag read in a tight loop inside your rendering path is a catastrophe. Treat this as a per-flag classification exercise backed by numbers you actually measured, not a platform-wide religion. The steps below give you those numbers and a decision rule you can defend in a design review.
Prerequisites
Step-by-Step Procedure
Step 1 — Benchmark local evaluation in isolation
Measure a warm local evaluation with the rule set already in memory. This is your floor — a data-structure lookup with no I/O. Run the loop long enough that the JIT (or the interpreter’s inline caches) has settled and the CPU has stopped scaling frequency; a hundred thousand iterations is usually enough to get a stable microsecond figure. Watch for a hidden allocation on the hot path: some SDKs build a fresh evaluation-context object or clone the targeting rules on every call, and that garbage-collection pressure, not the lookup itself, is what shows up as jitter at p99. If your local number is not comfortably under 100 microseconds, profile before you trust it — the point of local evaluation is that it disappears into the noise floor of the request, and anything slower means you are paying for work that should have happened once at sync time.
# bench_local.py — measure warm in-process evaluation
import time
from openfeature import api
client = api.get_client() # backed by a local file provider
N = 100_000
start = time.perf_counter()
for _ in range(N):
client.get_boolean_value("web.checkout.express-pay", False)
per_call_us = (time.perf_counter() - start) / N * 1e6
print(f"local: {per_call_us:.2f} us/call") # typically well under 100 usStep 2 — Benchmark remote evaluation under realistic placement
Measure the same read against a remote provider from where the service actually runs — same region and network as production. A benchmark from your laptop understates real remote latency badly: a home connection with a fat, idle pipe can look faster than a production pod that shares a NAT gateway and competes for connection-pool slots under load. The number that matters is not the median of a quiet box but the p99 of a saturated one, so drive your benchmark concurrently — spin up as many worker threads as the service runs in production and measure while they all hammer the provider at once. Pay attention to whether the provider keeps a persistent connection: the first call after an idle period may pay a fresh TLS handshake and a DNS lookup, and if your traffic is bursty enough to let the pool go cold between spikes, that handshake cost is not an outlier you can ignore — it is the latency a real user sees.
# bench_remote.py — measure per-call remote evaluation, warmed connection
import time
from openfeature import api
client = api.get_client() # backed by a remote provider
client.get_boolean_value("web.checkout.express-pay", False) # warm the connection
samples = []
for _ in range(1000):
t = time.perf_counter()
client.get_boolean_value("web.checkout.express-pay", False)
samples.append((time.perf_counter() - t) * 1e3)
samples.sort()
print(f"remote p50={samples[500]:.1f}ms p99={samples[990]:.1f}ms")Step 3 — Multiply by evaluations per request
The per-call number is not the whole cost — a request that reads twelve flags pays the remote latency twelve times unless you batch. Multiply the p99 by the real fan-out to see the request-level impact. And be honest about the fan-out: it is rarely the number a developer guesses, because flag reads hide inside shared components, middleware, and library code that the request author never sees. Instrument a real production request end to end and count the actual evaluate() calls before you plug a number into the formula. The multiplication is also why batching changes the shape of the decision entirely — a bulk evaluation API that resolves all twelve flags in one round-trip collapses twelve sequential p99s into one, but only if your SDK supports it and only if every flag on the path is known up front. If flags are read lazily as the request branches, you cannot batch what you do not yet know you need, and the serial cost is real.
# impact.py — request-level cost from per-call latency and fan-out
def request_impact_ms(per_call_p99_ms: float, flags_per_request: int, batched: bool) -> float:
return per_call_p99_ms if batched else per_call_p99_ms * flags_per_request
# e.g. 25ms remote p99 x 12 flags = 300ms unbatched, vs 25ms batched.Step 4 — Decide per flag against the budget
Compare the request-level impact to the path budget. If local fits and remote does not, choose local and add background sync. If both fit and the flag needs maximum freshness, remote is acceptable. The budget you compare against is not the whole request SLO — flag evaluation is one line item competing with database queries, downstream service calls, and serialization for the same milliseconds, so allocate it a slice and hold the decision to that slice, not to the generous total. A useful discipline is to give flag evaluation a hard sub-budget (say, five percent of the path’s p99 target) and treat any flag that cannot fit local into that slice as a design smell worth fixing at sync time rather than tolerating on the request path. When in doubt, default to local: freshness can almost always be tightened later by streaming a specific flag, but latency spent on the hot path is spent on every request forever, and you rarely get it back without a migration.
# decide.py
def choose(local_us: float, remote_impact_ms: float, budget_ms: float, needs_fresh: bool) -> str:
if remote_impact_ms <= budget_ms and needs_fresh:
return "remote" # freshness worth the affordable round-trip
return "local" # default: sub-ms, sync in the backgroundVerification
Confirm the deployed choice matches the benchmark. Emit an evaluation-duration metric in production and check the percentiles against what you measured, so a regression (a mis-configured remote provider on a hot path) surfaces immediately. Label the metric by flag key and by mode so you can tell at a glance which flags are local and which are remote — an aggregate histogram hides exactly the outlier you care about, the one flag someone quietly switched to a remote provider in a hotfix. The benchmark you took before deploy is your baseline; wire it into an alert that fires when a flag’s measured p99 drifts more than, say, two or three times above it, because that drift is almost always a configuration change nobody flagged, not a gradual degradation. This closes the loop: the decision you made from measured numbers stays true only if production keeps confirming those numbers, and the histogram is what keeps it honest.
# Query the evaluation-duration histogram — local flags should sit near zero
curl -s localhost:9090/api/v1/query \
--data-urlencode 'query=histogram_quantile(0.99, rate(flag_eval_duration_seconds_bucket[5m]))'
# Expect sub-millisecond for local flags; tens of ms only for the remote ones.Gotchas & Edge Cases
- Cold start hides in the average. The first local evaluation after boot pays the initial sync, and the first remote call pays connection setup. Benchmark warm, but budget for the cold first request separately — see lazy initialization. This bites hardest on autoscaled fleets and serverless, where fresh instances start constantly and a meaningful fraction of production traffic is always hitting a cold process. If your platform recycles workers aggressively, the cold path is not a boot-time curiosity — it is a steady-state percentile, and it belongs in your p99, not in a footnote.
- Remote tail latency is the real risk. A remote provider’s p50 may be fine while its p99 spikes on timeouts. Always cap remote evaluations with a short timeout and a safe default so one slow call cannot stall a request. Set that timeout deliberately: too tight and you fall back to defaults during normal jitter, silently serving the wrong variant; too loose and it stops protecting you at all. The default you fall back to must be a genuinely safe value for that flag, not just
false— a fail-closed default on a flag that gates a critical feature can be its own outage. - Local staleness is silent. If background sync stalls, local evaluation keeps returning old values with no error. Alert on the age of the last successful sync, not just on sync failures — a connection that hangs without erroring is worse than one that cleanly fails, because retry logic never triggers and the age just keeps climbing. Expose the last-sync timestamp on a health endpoint so an operator can confirm freshness during an incident without redeploying.
- A single provider outage fails both modes differently. When the control plane is unreachable, local evaluation degrades gracefully — it keeps serving the last synced rule set — while remote evaluation degrades to your timeout-and-default behavior on every single call. That asymmetry is itself a reason to prefer local for anything on a critical path: local’s failure mode is stale, remote’s is unavailable, and stale is almost always the safer of the two.
- Batched remote reads trade tail risk for blast radius. Collapsing twelve calls into one round-trip fixes the fan-out multiplication, but now a single slow or failed batch call takes out all twelve flags at once instead of one. Make sure your batch path has the same timeout-and-default discipline as a single call, and that a partial failure returns safe defaults for the flags it could not resolve rather than throwing the whole request.
Troubleshooting & FAQ
My local p99 is fine but occasional requests are slow. Why?
Those are almost certainly cold evaluations right after a deploy or a rule-set reload, when the in-memory structure is being rebuilt. Warm the rule set during startup readiness checks so the first user request does not pay that cost.
Can I get remote freshness without remote latency on the hot path?
Yes — that is exactly what local-plus-streaming gives you. Evaluate locally for sub-millisecond reads and keep the local rule set current with a streaming connection so changes land in well under a second without a per-call round-trip.
How stale is “too stale” for local evaluation?
It depends on the flag. A cosmetic toggle can tolerate minutes; a kill switch cannot tolerate seconds. Set the acceptable staleness per flag and drive the sync interval from the tightest one, or stream the kill-switch flags specifically.
Does local evaluation increase memory usage I should worry about?
The whole rule set lives in every process, so memory scales with flag count and rule complexity times replica count, not with request volume. For a few thousand flags this is negligible — kilobytes to low megabytes per process — but a runaway namespace with tens of thousands of flags and large targeting lists can become noticeable on memory-constrained pods. Measure the resident set after a full sync, and if it matters, scope the local rule set to only the flags a given service actually reads rather than syncing the entire catalog everywhere.
Should I ever mix local and remote for the same flag?
Occasionally, yes — evaluate locally on the hot request path for speed, and use a remote call only in a low-frequency control path where you need a guaranteed-fresh read, such as an admin screen confirming a rollout state. Keep the local value as the source of truth for user-facing behavior so the round-trip never touches the request budget, and treat the remote read as a diagnostic, not a dependency. Mixing per call within one flag is rare and usually a sign the flag is doing two jobs that should be split.
How does local evaluation affect experiment and analytics accuracy?
It does not change which variant a user gets — the assignment is deterministic from the flag rules and the evaluation context, so a locally evaluated flag returns the same variant a remote call would for the same inputs. What it changes is timing: after a rule update, local processes converge only as fast as their sync, so for a brief window different replicas may serve different variants. For most experiments that skew is immaterial, but if you are measuring a metric sensitive to the exact assignment boundary, tighten the sync interval or stream that flag so the whole fleet flips together.
Is remote evaluation ever genuinely faster than local?
Not for the evaluation itself — an in-memory lookup will always beat a network round-trip. But remote can win on freshness-critical paths where local’s alternative is an aggressive sync that itself costs CPU and network, and it can simplify a client that would otherwise struggle to hold a large rule set in memory. The speed comparison is never in remote’s favor; the trade you sometimes make for remote is operational simplicity or guaranteed freshness, paid for in latency.