Observability for Flag Evaluations
This guide is part of the Backend Evaluation & Server-Side SDKs series. A flag evaluation that you cannot observe is a decision you cannot debug: when a user reports the wrong experience, “which variant did they get, and why?” should be a query, not an archaeology project. This guide shows how to instrument every evaluation with three signals — metrics, traces, and logs — using OpenFeature hooks so the instrumentation is vendor-agnostic and lives in one place.
The reason to treat evaluation observability as its own discipline — separate from the metrics and traces your services already emit — is that a flag decision is the one runtime input to your code that has no representation in the code itself. A database row you can query later; a request header you can log at the edge; but the variant a targeting rule resolved to for one user at one instant vanishes the moment the if statement runs unless something captures it in flight. The hook is that something. Get it right once and every flag your team ships from now on is observable by default, with no diff to any feature’s call site and no checklist item that a rushed pull request can skip. That “observable by default” property is worth more than any single dashboard, because the flag that causes your next incident is almost always one nobody thought to instrument by hand.
Problem Framing
Feature flags multiply the number of code paths in production without leaving a trace in the code itself — the branch that runs depends on a value resolved at runtime from data outside the repository. Without instrumentation, a support ticket that says “the new checkout looks broken for me” is unanswerable: you cannot tell which variant the user received, whether the flag even resolved, or how the targeting decided. Teams end up adding console.log around suspect flags reactively, which never covers the flag that breaks next.
This guide covers the three observability signals for flag evaluations and how to emit them from a single hook. It does not cover general application observability, nor the flag change audit trail — who toggled a flag and when lives in building audit trails for compliance. Here the subject is the evaluation event, not the configuration change.
The distinction matters more than it sounds. An audit trail answers “what did an operator do to the flag?” — a low-volume, high-value stream measured in events per day. Evaluation telemetry answers “what did the flag do to a request?” — a high-volume stream measured in events per second, sometimes per microsecond on a hot path. They have opposite cost profiles, opposite retention needs, and opposite consumers: the audit trail is read by compliance and incident reviewers weeks later, while evaluation telemetry is read by on-call engineers minutes into an incident. Conflating them — writing every evaluation to the audit log, or trying to reconstruct which user saw what from configuration-change records — produces a pipeline that is simultaneously too expensive and unable to answer the question you actually have. Keep the two streams physically separate, with separate retention and separate access controls, and each stays fit for its purpose.
There is also a subtler trap worth naming up front: the temptation to instrument reactively. When a flag misbehaves, the instinct is to add logging around that flag and redeploy. That fix arrives too late for the current incident and does nothing for the next flag, so the pattern repeats indefinitely and your codebase accretes ad-hoc console.log statements that nobody ever removes. Instrumenting at the hook seam inverts this: coverage is universal and retroactive-by-design, so the flag you have not written yet is already observed the day it ships.
Prerequisites
Core Concept & Architecture
OpenFeature hooks are the right seam because they fire around every evaluation regardless of flag, provider, or call site. A single hook implementation emits all three signals, so you instrument once and cover every flag automatically — including the ones added next quarter. The hook has access to the flag key, the resolved variant, the evaluation reason (TARGETING_MATCH, DEFAULT, ERROR), and timing, which is exactly the payload the three signals need.
// telemetry-hook.ts — one hook, three signals, emitted for every evaluation
import { Hook, HookContext, EvaluationDetails, FlagValue } from '@openfeature/server-sdk';
import { evalCounter, evalDuration } from './metrics';
import { tracer } from './tracing';
import { logger } from './logging';
import { hashContext } from './context-hash';
export class TelemetryHook implements Hook {
private started = new WeakMap<HookContext, number>();
before(hc: HookContext) { this.started.set(hc, performance.now()); }
after(hc: HookContext, details: EvaluationDetails<FlagValue>) {
const ms = performance.now() - (this.started.get(hc) ?? performance.now());
// 1. Metric — aggregate counts and latency, low-cardinality labels only
evalCounter.inc({ flag: hc.flagKey, variant: String(details.variant), reason: details.reason ?? 'UNKNOWN' });
evalDuration.observe({ flag: hc.flagKey }, ms / 1000);
// 2. Trace — attach the decision to the active request span
tracer.getActiveSpan()?.addEvent('flag.evaluated', {
'flag.key': hc.flagKey, 'flag.variant': String(details.variant), 'flag.reason': details.reason ?? '',
});
// 3. Log — structured, PII-safe, replayable
logger.debug({ evt: 'flag_eval', flag: hc.flagKey, variant: details.variant,
reason: details.reason, ctx: hashContext(hc.context), ms: Math.round(ms) });
}
error(hc: HookContext, err: Error) {
evalCounter.inc({ flag: hc.flagKey, variant: 'ERROR', reason: 'ERROR' });
logger.warn({ evt: 'flag_eval_error', flag: hc.flagKey, err: err.message });
}
}The key design rule is cardinality discipline: metrics use only low-cardinality labels (flag key, variant, reason) so the time series does not explode, while high-cardinality detail (the context hash, the user correlation) goes to logs and traces where it is queryable but not aggregated.
Each of the three signals answers a different shape of question, and understanding the division of labor is what keeps you from misusing one to do another’s job. Metrics answer how much and how often — “what fraction of web.checkout.express-pay evaluations returned the treatment in the last hour, and did the error reason spike after the 14:03 deploy?” They are cheap to keep forever and fast to aggregate, but they carry no per-request identity, so they can never tell you which request. Traces answer where in a request — they place the evaluation inside the causal chain of one operation, so you can see that the flag resolved to the treatment before the downstream call that timed out, establishing order and blame. Logs answer exactly what happened to one decision — they are the forensic record you replay when a specific user files a ticket, carrying the context hash, the reason, and the resolved variant for that one evaluation. A mature setup uses all three in sequence during an incident: the metric shows the anomaly, the trace localizes it to a request path, and the log reproduces the precise decision. Skipping any one of them leaves a gap that the other two cannot fill.
Notice that the hook’s after and error methods are the only place these three emissions live. That single-seam design is deliberate: it means a field-naming decision (say, standardizing on flag.key rather than flagKey to match OpenTelemetry semantic conventions) is a one-line change that propagates to every flag at once, and it means there is exactly one code path to test rather than one per feature. The WeakMap keyed on the hook context, rather than a member field, is what makes the timing correct under concurrency — two evaluations in flight on the same hook instance each get their own start timestamp, and the entry is garbage-collected with the context so the map never leaks.
Step-by-Step Implementation
Step 1 — Register the telemetry hook globally
Attach the hook at the OpenFeature API level so it applies to every client and every evaluation without per-call-site changes.
// bootstrap.ts
import { OpenFeature } from '@openfeature/server-sdk';
import { TelemetryHook } from './telemetry-hook';
OpenFeature.addHooks(new TelemetryHook()); // covers all flags, all clientsPitfall: registering the hook per client instead of globally means a client created elsewhere (a background job, a test harness) evaluates flags with no telemetry. Register once at the API level.
Registration order also matters more than it first appears. OpenFeature runs before hooks in the order they were added and after hooks in reverse, so if you rely on the telemetry hook seeing the final context after an enrichment hook has run, add the telemetry hook first — its after then fires last, once every mutation is settled. Conversely, if a validation hook can short-circuit an evaluation by throwing, only the error path of hooks registered before it will see that failure, so a telemetry hook registered late can silently miss the very errors it exists to count. When in doubt, register telemetry as the outermost hook so it brackets everything else and observes the true, final outcome of each evaluation.
Step 2 — Define low-cardinality metrics
Declare a counter labeled by flag, variant, and reason, and a duration histogram labeled only by flag. Resist adding a user or request label — that is what logs are for.
// metrics.ts
import { Counter, Histogram } from 'prom-client';
export const evalCounter = new Counter({
name: 'flag_eval_total', help: 'Flag evaluations by variant and reason',
labelNames: ['flag', 'variant', 'reason'], // all low-cardinality
});
export const evalDuration = new Histogram({
name: 'flag_eval_duration_seconds', help: 'Flag evaluation duration',
labelNames: ['flag'], buckets: [0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05],
});Pitfall: an unbounded variant label (e.g. a free-form string flag) can still explode cardinality. For string flags with open-ended values, bucket the variant into known classes before labeling.
The histogram buckets deserve a deliberate choice rather than a copy-paste default. The example boundaries — 0.1ms through 50ms — are tuned for local evaluation, where a resolved-in-memory flag should land in the first two buckets and anything past 5ms signals a cold cache or a rule engine doing real work. If your provider evaluates remotely, those buckets are useless: every observation lands in the overflow bucket and the histogram tells you nothing. Match the buckets to the latency regime you actually run in, and re-check them after you change evaluation strategy, because a switch from remote to local evaluation shifts the entire distribution by two orders of magnitude and silently invalidates every percentile alert built on the old buckets. A second subtlety: the reason label is bounded by the OpenFeature spec’s enumerated set (STATIC, DEFAULT, TARGETING_MATCH, SPLIT, CACHED, DISABLED, ERROR, UNKNOWN), so it is always safe — but only if you map unrecognized provider-specific reasons back into that set rather than passing them through raw.
Step 3 — Emit a PII-safe context hash
Log a stable hash of the evaluation context, not its contents, so you can group all evaluations for one context without ever storing the attributes.
// context-hash.ts — deterministic, salted, non-reversible
import { createHash } from 'crypto';
export function hashContext(ctx: Record<string, unknown> = {}): string {
const canonical = JSON.stringify(ctx, Object.keys(ctx).sort());
return createHash('sha256').update(process.env.CTX_SALT + canonical).digest('hex').slice(0, 16);
}Pitfall: hashing without a salt lets anyone with the log and a guess at the attributes brute-force the input. Salt the hash and keep the salt out of the log. See masking PII in evaluation context.
Two properties make this hash useful rather than merely safe. First, it must be stable across the fields that identify a user and insensitive to the fields that do not — if you hash the entire context including a request timestamp or a per-call trace ID, every evaluation for the same user produces a different hash and the correlation you were building collapses. Canonicalize to the identity-bearing subset (typically the targetingKey plus the durable attributes that drive targeting) before hashing, and deliberately exclude volatile fields. Second, the salt is a rotation liability you should plan for: rotating CTX_SALT breaks correlation across the rotation boundary, so treat it like a key with a documented rotation window and accept that queries spanning a rotation need both salts. Truncating to 16 hex characters (64 bits) keeps log lines compact while leaving collision probability negligible at realistic user counts — but if you operate at hundreds of millions of distinct contexts, widen it, because the birthday bound is closer than intuition suggests.
Step 4 — Add the evaluation to the request trace
Attach the flag decision as an event on the active span so a single trace shows every flag that shaped the request. This is the fastest path from “this request behaved oddly” to “here are the flags it read.”
// already in the hook — the span event carries key, variant, reason
tracer.getActiveSpan()?.addEvent('flag.evaluated', {
'flag.key': hc.flagKey, 'flag.variant': String(details.variant),
});Pitfall: creating a new span per evaluation rather than an event on the existing request span floods the tracer with tiny spans. Use span events for flag decisions; reserve spans for operations with their own duration.
The span-event choice has a downstream consequence for querying that is easy to overlook. Because the decision is an event attribute set rather than a span, you query it with your tracing backend’s event or attribute filters — “show me traces containing a flag.evaluated event where flag.reason = ERROR” — not with span-duration filters. That is exactly the query you want during an incident, but it only works if the attribute keys are consistent, which is another argument for emitting them from the single hook. There is also a sampling interaction to respect: if your tracing is head-sampled at, say, 5%, then 95% of evaluations leave no trace event at all, even though every one of them still produces a metric and a log. That asymmetry is by design — traces are for the sampled examples, metrics for the aggregate truth — but it surprises engineers who go looking for a specific user’s evaluation in traces and find nothing. For guaranteed per-user recall, reach for the log’s context hash, not the trace. When you need the flag decision to influence sampling — keeping every trace where a flag errored — set the span status or a sampling-relevant attribute so a tail sampler can retain it.
Verification & Testing
Confirm the three signals actually fire. Evaluate a flag in a test with the hook attached and assert the counter incremented, a span event was recorded, and a log line was written with the expected fields. The reason to assert all three in one test, rather than three separate tests, is that the failure mode you most want to catch is a refactor that drops one signal while leaving the others intact — a single combined test makes that regression impossible to merge, whereas three isolated tests can rot independently and give a false sense of coverage.
Beyond the happy path, add a test that exercises the error method directly by forcing a provider failure, and assert that the error counter increments with reason: ERROR and that no success log is emitted. The error path is the one that runs least often in normal operation and therefore the one most likely to have quietly broken — and it is precisely the path you depend on during an incident, when a provider is timing out and you need the error rate to be truthful. It is also worth asserting the negative: that a raw context attribute never appears in any emitted field. A cheap way to enforce this is to seed the evaluation context with a sentinel value and assert that the sentinel appears nowhere in the captured metrics labels, span attributes, or log fields — only its hash should survive. That single assertion turns “we masked PII” from a claim into a test the build enforces.
// telemetry-hook.test.ts
test('one evaluation emits all three signals', async () => {
const client = OpenFeature.getClient();
await client.getBooleanValue('web.checkout.express-pay', false, { targetingKey: 'u1' });
expect(await counterValue('flag_eval_total', { flag: 'web.checkout.express-pay' })).toBe(1);
expect(recordedSpanEvents()).toContainEqual(expect.objectContaining({ name: 'flag.evaluated' }));
expect(capturedLogs()).toContainEqual(expect.objectContaining({ evt: 'flag_eval', flag: 'web.checkout.express-pay' }));
});Troubleshooting & FAQ
My metrics cardinality exploded after adding flag telemetry. What happened?
A high-cardinality label slipped into a metric — usually a user ID, a request ID, or an unbounded string variant. Move that field to logs or traces and keep metric labels to flag key, variant, and reason. Audit the label set with your metrics backend’s cardinality report.
How do I find which variant a specific user saw last Tuesday?
Query the structured logs by the context hash for that user and the time window. Because the hash is stable, all of that user’s evaluations share it, and the log line carries the flag key, variant, and reason. Never rely on metrics for this — they are aggregates.
Does wrapping every evaluation in a hook add meaningful latency?
For local evaluation the hook is a few microseconds of in-process work — negligible next to the evaluation itself. The one cost to watch is synchronous log or metric I/O; keep emission non-blocking by buffering. See reducing evaluation latency.
Should the telemetry hook ever throw, and what happens if it does?
No — a telemetry hook must never throw, because in OpenFeature a hook error can surface as an evaluation error and flip an otherwise healthy flag into its default value. Wrap the emission logic so any failure in metrics, tracing, or logging is swallowed and, at most, counted as an internal error metric. Observability is supposed to watch the evaluation, not endanger it; a monitoring layer that can take down the thing it monitors is worse than none.
How should I instrument a flag whose value is an object or JSON, not a boolean?
Do not label a metric with the object itself — serialized objects are effectively unbounded cardinality. Instead, emit a metric labeled only by flag key and a derived low-cardinality classification (for example a schema version or a named preset), and put the full resolved object, or its hash, in the log line and the span event where high cardinality is affordable. This keeps the aggregate counts meaningful while preserving the exact value for forensic replay. The same rule that governs string variants — bucket before you label — applies with more force to structured values.
The metric shows an evaluation but there is no matching log line. Why?
Almost always sampling or buffering. Debug-level evaluation logs are typically gated behind a sampling rate to survive hot paths, so most evaluations are counted but not logged — the metric is exhaustive, the log is a sample. A buffered log writer can also drop or delay lines on flush failure or process exit. If you need a guaranteed record for a specific evaluation, raise the log level or sampling rate for that flag temporarily, or rely on the metric plus the trace event rather than expecting a log for every single evaluation.
How long should I retain each of the three signals?
They warrant different retention because they answer different questions. Metrics are cheap and worth keeping for months so you can compare a rollout’s variant distribution against the same flag a quarter ago. Traces are expensive per byte and usually retained for days — long enough to investigate a recent incident, not to build history on. Structured evaluation logs sit in between and are frequently the shortest-lived of the three, since they are the highest-volume and often the most privacy-sensitive; retain them just long enough to answer support tickets, and let the audit trail — not the evaluation log — be your long-term system of record.
Performance & Scale Considerations
The instrumentation must be cheaper than the thing it measures. Metric increments and span events are in-memory operations costing microseconds, but the log line is the risk: a synchronous write per evaluation can dominate a sub-millisecond evaluation by orders of magnitude. Buffer log emission and flush asynchronously, and gate debug-level evaluation logs behind a sampling rate so a hot path evaluated millions of times per minute does not drown the pipeline. At scale, sample traces rather than recording every request’s flag events, and rely on the always-on metrics for aggregate truth. The distributed caching guide covers keeping the evaluation itself fast so the telemetry overhead stays proportionally small.
There is a second-order cost that only shows up under load: memory and allocation pressure. Every evaluation that builds a log object, a labels map, and a span-event attribute set allocates short-lived objects, and at millions of evaluations per minute that allocation rate becomes visible as garbage-collection pauses even when each individual operation is cheap. The mitigations are the ordinary ones — reuse label objects for the common cases, avoid string concatenation in the hot path, and let sampling thin the log allocations — but the point is to measure the hook’s overhead the same way you measure any hot code, with a benchmark that evaluates a flag in a tight loop with and without the hook attached. If the delta is not comfortably under the evaluation’s own budget, your telemetry has become part of the problem it was meant to observe.
Finally, treat cardinality as a running budget, not a one-time review. Every new flag adds a row to the metric’s label space, and a flag with many variants multiplies it; a system with thousands of active flags can accumulate a surprising number of active time series even with perfect label discipline. Put a ceiling on it: alert when the flag telemetry’s series count crosses a threshold, and reclaim series aggressively when a flag is retired — a stale flag that no longer evaluates should stop producing series, which is one more reason cleaning up stale flags pays for itself in observability cost as well as code clarity. Sampling, buffering, bounded labels, and retirement together keep the cost of watching every evaluation proportional to the value of being able to answer, at any moment, which variant a user saw and why.