Correlating Flag Evaluations with Distributed Traces
This how-to is part of Observability for Flag Evaluations. A request behaves oddly in production and you suspect a flag, but the trace shows the slow span without telling you which variant the code took. This how-to wires every flag decision into the request’s OpenTelemetry trace as a span event, so one trace answers “which flags ran, what did they resolve to, and why” without a separate log hunt.
The payoff is that flag state stops being invisible context you have to reconstruct after the fact. When a support ticket says “checkout was broken for this one user at 14:32,” you paste their trace ID into Jaeger and read the flag decisions directly off the request span — no guessing which experiment they were bucketed into, no cross-referencing a separate exposure log whose clocks drift from your trace backend. Because the events live on the trace itself, they inherit the trace’s sampling, retention, and access controls for free; you are not standing up a second pipeline that can silently fall out of sync with the first. The whole technique fits in one OpenFeature hook of roughly thirty lines, and it works the same whether you evaluate against flagd, a hosted provider, or an in-process store.
Prerequisites
Step-by-Step Procedure
Step 1 — Read the active span inside the hook
In the hook’s after and error methods, fetch the current span from the OpenTelemetry context. If there is no active span, skip silently rather than creating a detached one. A detached span — one you start yourself because none was active — attaches to no trace, so it exports as an orphan root that clutters the backend and correlates with nothing; a silent skip is strictly better than a lie. The after method is deliberate: it runs only when the evaluation resolved cleanly, so d.variant and d.reason are populated, whereas an after recorded during error handling would carry the default value and mislead whoever reads the trace later.
Read the span once and reuse it; trace.getActiveSpan() is cheap but the discipline of a single early null-check keeps both methods honest. Note that the event only becomes part of the trace if the span is still recording — a span that has already ended, or one on a non-sampled trace, will silently drop the addEvent call, which is exactly the behaviour you want on the hot path.
// trace-hook.ts
import { trace } from '@opentelemetry/api';
import { Hook, HookContext, EvaluationDetails, FlagValue } from '@openfeature/server-sdk';
export const traceHook: Hook = {
after(hc: HookContext, d: EvaluationDetails<FlagValue>) {
const span = trace.getActiveSpan();
if (!span) return; // no request span here — nothing to correlate
span.addEvent('flag.evaluated', {
'feature_flag.key': hc.flagKey,
'feature_flag.variant': String(d.variant),
'feature_flag.provider_name': hc.providerMetadata.name,
'feature_flag.reason': d.reason ?? 'UNKNOWN',
});
},
};Step 2 — Use the OpenTelemetry semantic conventions
Name the attributes with the feature_flag.* semantic convention so any OTel-aware backend recognizes and can filter on them, rather than inventing bespoke keys. The convention matters more than it looks: Grafana, Honeycomb, and Datadog all ship pre-built flag views keyed off these exact attribute names, so a bespoke myflag_key leaves you writing custom queries forever while feature_flag.key lights up the built-in tooling immediately. The OpenTelemetry convention also standardises the set — key, provider name, variant, and evaluation reason — which is enough to answer most incident questions without adding your own attributes.
One judgement call is whether to attach the context key (the targeting key that decided the bucket). It is invaluable for reproducing a single user’s decision, but it is frequently PII, so treat it the way you would any identifier on a span: hash it, or gate it behind the same masking rule you apply elsewhere. If you already mask targeting context upstream, keep that consistent here rather than leaking the raw value onto the trace.
// The semantic-convention keys backends understand out of the box
const FF = {
key: 'feature_flag.key',
variant: 'feature_flag.variant',
provider: 'feature_flag.provider_name',
};Step 3 — Record evaluation errors as span exceptions
When an evaluation errors, record it on the span so the trace flags the failing request. This turns a silent fallback into a visible signal. The distinction that trips people up is that an evaluation error is not the same as a request error — the flag SDK returns the default value and the request usually completes with a 200, so nothing else in the trace hints that a decision quietly fell back. Recording the event, and optionally calling span.setStatus or span.recordException on genuinely unexpected provider failures, is the only thing that makes that fallback searchable later.
Keep the error event lean: the flag key and the exception message are enough to find the request. Resist stuffing a full stack trace onto every failed evaluation — during a provider outage that fires thousands of times a minute, and the event payload is what your exporter has to serialise on the request path.
// trace-hook.ts (error path)
error(hc: HookContext, err: Error) {
const span = trace.getActiveSpan();
if (!span) return;
span.addEvent('flag.evaluation_error', { 'feature_flag.key': hc.flagKey, 'exception.message': err.message });
}Step 4 — Filter traces by flag in your backend
With the semantic-convention attributes in place, query the trace backend for all traces where a given flag resolved to a given variant — the fast path from “the discount cohort sees errors” to the exact requests. This is where the earlier work pays off: because the attributes are indexed by name, the query returns in the same time as any other trace search, and you can pivot from it straight to the latency distribution of just that variant. That comparison — p99 for variant = "on" versus variant = "off" on the same endpoint — is often the fastest way to prove or disprove that a rollout caused a regression before you touch the code.
Save the useful queries as backend-side searches or dashboard panels rather than retyping them under pressure. When you are mid-incident, “show me every trace in the last ten minutes where any flag event carries an error” should be one click, not a TraceQL expression you are debugging while the pager is going off.
# Tempo/Grafana example: traces where regional-discount resolved to "on"
curl -s "$TEMPO/api/search" --data-urlencode \
'q={ event.name = "flag.evaluated" && feature_flag.key = "web.pricing.regional-discount" && feature_flag.variant = "true" }'Verification
Fire a request through a service with the hook attached and inspect the exported trace. The request span should carry a flag.evaluated event per flag read, each with the semantic-convention attributes. Verify against the exported trace, not a unit test that mocks the span — a passing unit test only proves the hook calls addEvent, while the exported trace proves the span was real, the context propagated, and the exporter serialised the event all the way to the backend. Those are three separate failure points, and only end-to-end verification exercises all of them at once.
Count the events, too. If a request reads four flags but the trace shows three events, one evaluation is running outside the request context — usually a flag read at module load, in a startup path, or on a thread that lost the propagated context. A missing event is a more useful signal than it looks, because it points straight at the evaluation that is not correlated.
# Confirm the exported trace has flag events with the right attributes
otel-cli span list --service checkout --last 1m \
| jq '.[].events[] | select(.name=="flag.evaluated") | .attributes["feature_flag.key"]'
# Expect the flag keys read during that request.Gotchas & Edge Cases
- Background work has no request span. Cron jobs and queue consumers evaluate flags outside an HTTP request, so
getActiveSpan()returns nothing. Create a job-level span for those paths, or accept that their flag reads land only in logs and metrics. If a job processes a batch, wrap each item in its own span rather than the whole batch — otherwise every item’s flag events pile onto one span and you lose the per-item correlation that made tracing worth it. - A span per evaluation floods the tracer. Emitting a child span for each flag turns a 3-flag request into a noisy tree. Use span events on the existing request span, which are cheap and keep the trace readable. Events are also the right primitive semantically: a flag read is a point-in-time decision, not a unit of work with its own duration, so modelling it as a zero-duration event matches what actually happened.
- Sampling can discard the trace you need. With head-based sampling, the decision to keep a trace is made before the flag error occurs. Use tail-based sampling that retains traces containing a
flag.evaluation_errorevent. Remember that tail sampling runs in the collector, so the rule lives in your collector config, not your app — and it can only keep what reached it, so the app must still emit the event unconditionally regardless of the sampling decision. - The same flag read twice records two events. If your code reads a flag in a guard and again where it uses the value, the trace shows two identical
flag.evaluatedevents, which can look like a bug during review. Either read once and pass the value down, or accept the duplication and remember it when counting events during verification. - Attribute cardinality still matters on traces. Span-event attributes are cheaper than span attributes for indexing, but a backend that indexes event attributes will still choke if you attach a high-cardinality value like a raw user ID to millions of events. Keep the attribute set to the semantic-convention keys unless you have measured that your backend can afford more.
Troubleshooting & FAQ
The hook runs but no flag events appear in my traces. Why?
Almost always a context-propagation gap: the hook executes on a different async context than the request span, so getActiveSpan() returns undefined. Confirm your runtime’s context propagation (async hooks in Node, contextvars in Python) is enabled and that the evaluation happens within the request’s active context.
Should flag attributes go on the span itself or as an event?
As an event. A span has one set of attributes for its whole lifetime, but a request reads several flags at different moments; events capture each decision with its own timestamp and attributes without overwriting the others.
How do I correlate a flag decision across services?
The trace context already propagates across service boundaries, so a flag evaluated in a downstream service lands on the same trace. Ensure each service attaches its flag events using the same feature_flag.* convention, and the full trace shows the decision path across all of them.
Does adding flag events measurably slow down evaluation?
Not in any way you can measure against the flag read itself. getActiveSpan() and addEvent() are in-process calls that append to a buffer already held in memory; the real cost is at export time, in the collector, not on the request path. If evaluation latency is your concern, the flag lookup and any network round-trip to the provider dominate by orders of magnitude — the span event is noise beside them. Keep the attribute payload small and you will not notice the hook at all.
Should I trace flag reads at debug level or always?
Always, at the same sampling rate as the rest of the request. The value of correlation is that it is there when an incident happens, and incidents do not announce themselves in advance — a debug-only path means the one trace you need was recorded without its flag events. Because events ride the trace’s existing sampling decision, “always on” costs you nothing beyond the sampled traces you were already keeping.
How is this different from logging the flag decision?
A log line is correlated to the trace only if you remember to inject the trace and span IDs into it, and even then you are joining two systems across a timestamp and an ID. A span event is part of the trace — same retention, same sampling, same view — so there is nothing to join. Logs still earn their place for high-volume or long-retention decision records; the span event is for the in-context question “what did this specific request decide,” which the trace answers directly.
Can I put the evaluation reason to work in queries, not just display?
Yes, and it is one of the most useful filters you have. Searching for feature_flag.reason = "ERROR" or "DEFAULT" across a service surfaces every request that fell back — often before users report anything — because a spike in DEFAULT reasons usually means a provider or config problem rather than deliberate targeting. Treat the reason attribute as a first-class query dimension, not a label you only read after you have already found the trace.