Emitting Flag Evaluation Metrics to Prometheus
This how-to is part of Observability for Flag Evaluations. You are rolling a flag from 5% to 50% and want a graph that shows the variant split shifting and the error rate staying flat — in real time, not after the incident. This how-to sets up a Prometheus counter and a duration histogram from an OpenFeature hook, keeps label cardinality safe, and writes the alert that pages you when a rollout goes wrong.
The reason to reach for aggregate metrics rather than logs is volume and cost. A busy service evaluates the same flag millions of times a minute; you do not want a log line per evaluation, but you do want to know the running variant split and error rate at all times. Two Prometheus series — a counter and a histogram — compress that firehose into something you can graph, alert on, and retain for months at negligible cost. The discipline that makes this work is keeping the label set small and closed: every distinct combination of label values is a separate time series that Prometheus must store and index, so an unbounded label turns a cheap counter into an operational hazard. Get the labels right once and the same two series serve every dashboard, alert, and post-incident query you will ever write against flag traffic.
Prerequisites
Step-by-Step Procedure
Step 1 — Declare the counter and histogram with bounded labels
Create a counter labeled by flag, variant, and reason, and a histogram labeled only by flag with buckets tuned to local-evaluation latency. The three counter labels are deliberate. flag lets you slice by the rollout you care about; variant gives you the split; reason — the OpenFeature evaluation reason such as TARGETING_MATCH, DEFAULT, SPLIT, or ERROR — tells you why a value was served, which is the difference between “the flag returned false because the user missed the rollout” and “the flag returned false because the SDK fell back to a default after a fetch failed.” Those look identical in the variant column and completely different in the reason column.
Note that the histogram carries only the flag label, not variant or reason. Latency does not meaningfully differ by variant in local evaluation, so adding those labels would triple the histogram’s series count — and a histogram is already expensive, emitting one series per bucket plus a sum and a count. With six buckets that is eight series per flag; multiply that by variant and reason and a handful of flags balloons into hundreds of series for no analytic gain. Choose bucket boundaries that straddle your expected p50 and p99: the boundaries above assume sub-millisecond in-process evaluation, so most of the resolution sits below one millisecond with a single coarse bucket at 20ms to catch the pathological slow path.
// metrics.ts
import { Counter, Histogram, register } from 'prom-client';
export const evalTotal = new Counter({
name: 'flag_eval_total',
help: 'Feature flag evaluations',
labelNames: ['flag', 'variant', 'reason'] as const,
});
export const evalSeconds = new Histogram({
name: 'flag_eval_duration_seconds',
help: 'Feature flag evaluation duration',
labelNames: ['flag'] as const,
buckets: [0.00005, 0.0001, 0.0005, 0.001, 0.005, 0.02],
});Step 2 — Increment from the hook
In the hook’s after, increment the counter and observe the duration; in error, increment with an ERROR variant so failures show in the same series. Emitting from a hook rather than sprinkling evalTotal.inc() at every call site is what keeps the instrumentation honest — the hook fires on every evaluation the SDK performs, including the ones buried in library code you did not write, so there is no evaluation path that silently escapes the counter. Capture the start time in the hook’s before stage and stash it on the hook context; after reads it back to compute elapsedSeconds. Do not call Date.now() inside after alone, because by then the evaluation is already finished and you have lost the start reference.
The error stage matters more than it looks. When an evaluation throws — a malformed rule, a provider timeout, a deserialization failure — the SDK returns your code-supplied default and the after stage never runs, so without an explicit error handler those evaluations vanish from the counter entirely and your “total” is quietly undercounting. Folding failures back in as variant: 'ERROR', reason: 'ERROR' keeps the denominator honest and makes the error-rate alert in Step 4 possible. One subtlety: the default value a user actually received during an error is not the ERROR variant — it is whatever your code passed as the fallback — so treat the ERROR series as a health signal for the evaluation path, not as a record of what value the user saw.
// metrics-hook.ts
after(hc, details, ) {
evalTotal.inc({ flag: hc.flagKey, variant: String(details.variant), reason: details.reason ?? 'UNKNOWN' });
evalSeconds.observe({ flag: hc.flagKey }, hc.elapsedSeconds); // elapsed captured in before()
},
error(hc, err) {
evalTotal.inc({ flag: hc.flagKey, variant: 'ERROR', reason: 'ERROR' });
},Step 3 — Bound cardinality for open-ended variants
A boolean flag has two variants; a string flag can have unbounded values. Map open-ended variants into a small set of known classes before labeling so the series count stays finite. The danger is not hypothetical: if a string flag returns a per-user value, or a JSON flag whose stringified form varies by request, every distinct value becomes a permanent time series. Prometheus never forgets a series it has seen within the retention window, so a single evaluation with a unique variant leaves a series behind for hours. Ten thousand unique variants means ten thousand series for one flag, and cardinality explosions like that are the most common way a metrics pipeline takes down its own Prometheus — ingestion slows, the head block bloats, and queries across the whole instance degrade, not just queries for the offending flag.
Keep the KNOWN set close to the flag’s actual variant schema and treat other as a tripwire: if the other series starts carrying real traffic, either a new variant shipped without updating the map or something is returning values you did not anticipate. The same collapse applies to reason if you ever see a provider emit non-standard reason strings, though the OpenFeature reason vocabulary is already a closed, small set. For flags whose variants genuinely change often, drive the KNOWN set from the same configuration that defines the flag so the map cannot drift out of sync with the rollout.
// bucket-variant.ts — collapse open-ended string variants
const KNOWN = new Set(['control', 'treatment-a', 'treatment-b']);
export const bucketVariant = (v: string) => (KNOWN.has(v) ? v : 'other');
// evalTotal.inc({ flag, variant: bucketVariant(String(details.variant)), reason })Step 4 — Write the rollout-safety alert
Alert when a flag’s error reason rises above a threshold, so a rollout that starts erroring pages you before users pile up in support. The expression divides the ERROR-reason rate by the total rate per flag, so it measures a proportion rather than a raw count — a service that does ten million evaluations a minute and one that does ten thousand both page at the same 1% error fraction, without per-service tuning. The for: 5m clause is what keeps this alert survivable: a brief blip during a provider reconnect will clear before the timer elapses, so you are paged for sustained failure, not transient noise. Set the threshold against your own baseline — 1% is a reasonable default for a flag that should essentially never error, but a flag that intentionally falls back under load may sit higher, and alerting below its normal floor just trains you to ignore the page.
Consider pairing this with a second alert on a sudden shift in the variant split, which catches a different failure: a rollout that is erroring stays visible in the ERROR series, but a rollout that is silently serving the wrong variant to everyone — a misconfigured percentage, a targeting rule that matches far more users than intended — shows up only as the split diverging from what you configured. The split-ratio query in the FAQ below is the raw material for that alert.
# alert.rules.yml
groups:
- name: feature-flags
rules:
- alert: FlagEvaluationErrorRateHigh
expr: |
sum(rate(flag_eval_total{reason="ERROR"}[5m])) by (flag)
/ sum(rate(flag_eval_total[5m])) by (flag) > 0.01
for: 5m
labels: { severity: page }
annotations:
summary: "Flag {{ $labels.flag }} error rate above 1%"Verification
Scrape the endpoint after exercising a flag and confirm the series exist with the expected labels and a plausible variant split. Do this against a single pod, not through a load balancer, so you are reading one process’s counters rather than a random replica per curl — otherwise the numbers appear to jump around and you will chase a phantom bug. The two things to check are that the labels carry the values you expect (a reason of TARGETING_MATCH on the treatment arm, DEFAULT on control is typical) and that the ratio between variants is in the neighborhood of your configured percentage once enough traffic has accumulated. Small samples are noisy; a 9% rollout will not read exactly 9% after fifty evaluations, so drive a few thousand before you trust the split. If the reason column shows ERROR on a flag you did not expect to fail, you have found a real problem before shipping the dashboard.
# After driving traffic through a flag, check the series appear
curl -s localhost:8080/metrics | grep 'flag_eval_total'
# flag_eval_total{flag="web.checkout.express-pay",variant="true",reason="TARGETING_MATCH"} 4123
# flag_eval_total{flag="web.checkout.express-pay",variant="false",reason="DEFAULT"} 39544Gotchas & Edge Cases
- Counters reset on process restart. A raw counter value is meaningless across deploys; always graph and alert on
rate()orincrease(), which handle resets correctly. - Metrics are per-pod. In a horizontally scaled service each replica exposes its own counter. Sum across pods (
sum by (flag, variant)) before computing a variant ratio, or a rolling deploy will skew the split. - Deleted flags leave stale series. After a flag is removed, its series linger until the retention TTL. That is normal; do not alert on the absence of new samples for a decommissioned flag. Cross-reference with flag cleanup.
- The error-rate alert divides by zero when a flag goes idle. If a flag stops being evaluated entirely, both the numerator and denominator of the ratio drop to zero and the expression yields
NaN, which most Prometheus comparisons treat as not firing — so the alert silently stops working rather than paging. If a flag must always see traffic, add a companion alert onabsent()or on the total rate falling to zero, so “no evaluations at all” is itself a condition you hear about. - Histogram buckets are cumulative and immutable once chosen. Each Prometheus bucket counts every observation less than or equal to its boundary, so
le="0.001"includes everything in the smaller buckets too — remember that when reading raw samples. More importantly, you cannot retroactively re-bucket old data: if your boundaries turn out to be wrong, changing them only affects samples recorded after the change, and the two ranges do not align cleanly on ahistogram_quantile()graph. Pick boundaries deliberately the first time. observe()andinc()must see identical label sets per call. If a code path increments the counter but skips the histogram (or vice versa), the two series drift apart and any query that joins them — say, latency per variant — returns gaps. Emit both from the same hook stage so they cannot diverge.
Troubleshooting & FAQ
My histogram shows almost everything in the smallest bucket. Is that wrong?
No — that is the expected shape for local in-process evaluation, which is sub-millisecond. If you want more resolution at the low end, add finer buckets below 0.001s. If evaluations are landing in high buckets, you are probably making a remote call; see local vs remote latency.
Should I put the rollout percentage in a metric label?
No. The rollout percentage changes over time and is better read from the observed variant split than from a label. Adding it as a label also multiplies cardinality every time you adjust the rollout.
How do I graph the variant split as a percentage?
Divide each variant’s rate by the total rate for that flag: sum(rate(flag_eval_total{flag="X"}[5m])) by (variant) / ignoring(variant) group_left sum(rate(flag_eval_total{flag="X"}[5m])). That yields each variant’s share, which should track your configured rollout.
Counter or histogram — which one should I add first if I can only do one?
The counter. The variant-split and error-rate signals it carries are what actually protect a rollout, and they are the ones an on-call engineer reaches for during an incident. Latency from the histogram is valuable, but for local in-process evaluation it is almost always sub-millisecond and rarely the thing that goes wrong. Ship the counter, wire the error-rate alert, and add the histogram once the counter is proving its worth.
Should I emit these metrics from a hook or directly at each call site?
From a hook. An OpenFeature hook fires on every evaluation the SDK performs, including evaluations inside dependencies you did not write, so no path escapes instrumentation. Manual inc() calls at each site inevitably miss some evaluations and drift as the code changes, leaving blind spots exactly where a new feature is being rolled out. The hook also gives you a single place to enforce the cardinality-bounding logic from Step 3.
Why is my summed variant ratio slightly above 100%?
That is usually a boundary artifact of rate() over a rolling window across pods that restarted or scaled mid-window, combined with floating-point division. Small overshoots of a fraction of a percent are normal; if the sum is consistently and substantially off, check that you are summing across all pods with sum by (flag, variant) before dividing, and that no stray label (like an instance or pod label) is splitting what should be one series.
Can I use a Prometheus summary instead of a histogram for latency?
Prefer the histogram. A summary computes quantiles inside each pod, and those per-pod quantiles cannot be aggregated across replicas — averaging p99s from ten pods does not give you the fleet p99. A histogram exposes raw bucket counts that Prometheus aggregates server-side with histogram_quantile(), so you get a correct fleet-wide percentile. The cost is choosing bucket boundaries up front, which for flag evaluation is straightforward.