Debugging Unexpected Flag Values in Production

This how-to is part of Observability for Flag Evaluations. A user says they should see the new experience and do not, or a cohort got a variant they should not have. This how-to is the systematic path from that report to a definitive answer: read the evaluation reason, reproduce the exact context, and confirm against a provider dry-run — so you fix the real cause instead of toggling flags hopefully.

The discipline here matters because a flag value is never wrong at random — it is the deterministic output of a rule set applied to a context. If you can name the rule set and reconstruct the context, the value becomes fully explainable, and “unexpected” collapses into “expected, given inputs I had not seen.” Almost every wasted hour in flag debugging comes from skipping straight to the rule and editing it on a hunch, when the reason field and one reconstructed evaluation would have told you the rule was fine and the context was the surprise. Treat every report as a claim to be reproduced, not a bug to be patched — the reproduction is what separates a real fix from a coincidence that quiets the ticket for a day.

Prerequisites

• Evaluations logged with their reason and errorCode (see observability for flag evaluations)
• The context hash recorded so you can find one user’s evaluations
• Access to the flag’s current targeting rules
• A way to run a provider dry-run with an arbitrary context (staging or a local provider)
• The reported user’s identifying attribute (the targeting key or an equivalent)

Step-by-Step Procedure

Step 1 — Retrieve the evaluation and read its reason

Find the user’s evaluation by context hash and flag key, and read the reason. This single field eliminates most hypotheses immediately. The reason is not decoration — it is the provider’s own account of which branch of its decision tree produced the value, and it is far more reliable than reasoning backward from the variant. A false could mean the rule matched and returned false, the rule did not match and the default is false, or the provider errored and fell back to false; only the reason distinguishes those three, and each demands a different fix. Pull the single most recent evaluation rather than an aggregate, because you are debugging one user’s one decision at one moment, and an averaged view hides the very outlier you are chasing.

# Find the user's most recent evaluation of the flag and read its reason
logcli query '{app="checkout"} | json | evt="flag_eval"
  | flag="web.checkout.new-summary" | ctx="a1b2c3d4e5f6a7b8"' --limit 1 \
  | jq '{variant, reason, errorCode}'
# -> { "variant": false, "reason": "DEFAULT", "errorCode": null }

Step 2 — Branch on the reason

A DEFAULT with no error means no targeting rule matched — the context did not qualify. A DEFAULT with PROVIDER_NOT_READY means a bootstrap race. An ERROR points at the errorCode. Handle each with its own next move rather than assuming the rule is wrong. The distinction between a clean DEFAULT and an error-driven DEFAULT is the one people most often collapse, and it sends them in opposite directions: the first is a targeting question (“why did this context not qualify?”), the second is a lifecycle question (“why did the provider hand back the code default?”). Note too that DEFAULT returns the in-code fallback you passed at the call site, not the flag’s configured off-variant — if those two values differ, the number you see in production is coming from your source, not from the flag service, which is a useful tell on its own. When the reason is ERROR, resist reading the value at all; a fallback value carries no information about the rule, and the only signal worth acting on is the errorCode.

# triage.py — map reason to the next diagnostic move
def next_move(reason: str, error_code: str | None) -> str:
    if reason == "ERROR":            return f"inspect errorCode={error_code}"
    if reason == "DEFAULT" and error_code == "PROVIDER_NOT_READY":
        return "bootstrap race — check init ordering"
    if reason == "DEFAULT":          return "no rule matched — inspect the context"
    if reason == "TARGETING_MATCH":  return "rule DID match — expectation is wrong or context differs"
    return "check flag enabled state"

Step 3 — Reproduce with the exact context in a dry-run

Rebuild the user’s evaluation context and run it against the provider offline. If the dry-run reproduces the value, the rule and context are the cause; if it does not, the production context differed from what you assumed. Pin the dry-run to the same flag configuration version production served — a local flagd instance loaded from the exact rule snapshot, not today’s staging config — or you are comparing two different rule sets and any disagreement is meaningless. The dry-run is a controlled experiment, and its whole value is that only one variable moves at a time. When the offline run refuses to reproduce the reported value even with a faithful context, that is itself a finding: it points away from targeting and toward per-node state — a stale local rule set, a warm cache serving a superseded decision, or a sticky assignment recorded before the rule changed.

# dryrun.py — reproduce the decision deterministically
from openfeature import api

ctx = {"targetingKey": "user-5571", "plan": "free", "country": "PT"}  # reconstructed
details = api.get_client().get_boolean_details("web.checkout.new-summary", False, ctx)
print(details.value, details.reason, details.variant)
# Matches production? -> context/rule is the cause. Differs? -> the live context was different.

Step 4 — Confirm the fix against the same dry-run

Once you have a hypothesis — a rule that should match, an attribute that was missing — apply it in staging and re-run the identical dry-run. A green reproduction is proof before you touch production. The word “identical” is load-bearing: reuse the same reconstructed context object, not a fresh hand-typed one, so the only thing that changed between the failing run and the passing run is the fix itself. If you rebuild the context by hand for the confirmation, you risk quietly correcting the very attribute that was the bug and declaring victory over a problem you never actually reproduced. Keep the failing context checked in next to the fix — a two-line fixture that resolves DEFAULT false on the old config and TARGETING_MATCH true on the new one is the cheapest regression test you will ever write, and it stops the same report from reopening after a later rule edit.

# After adjusting the rule in staging, the same context should now resolve true
python dryrun.py --env staging   # -> True TARGETING_MATCH treatment

Verification

Prove the reported case is fixed by replaying the exact production context and asserting the new value with a matching reason. Assert on the reason as well as the value, not the value alone — a run that returns true for the wrong reason (say, a fallback true from an error path) looks green but proves nothing about your targeting change. The pairing of expected value and expected reason is what makes the assertion trustworthy, because it confirms the decision arrived through the branch you intended to fix.

# The reconstructed context now resolves to the expected variant and reason
python dryrun.py --env staging --assert value=true reason=TARGETING_MATCH
# exit 0 = fixed; non-zero prints the actual value/reason for further digging

Gotchas & Edge Cases

• The live context was not what you rebuilt. The most common false lead: a reconstructed context is missing an attribute the real request had (or vice versa). Log enough of the context — safely hashed — to rebuild it faithfully, and diff the two if the dry-run disagrees. Watch the boundary cases that silently flip a rule: a plan that arrived as the string "free" in one path and null in another, a country that was lowercase in the request but uppercase in the rule, or an attribute that was present but empty. Targeting comparisons are exact, so a type or case mismatch reads as “did not qualify” with no error to warn you.
• One node had a stale rule set. With local evaluation, a single pod whose background sync stalled serves old values while its peers serve new ones. Check the sync age on the specific node the request hit before blaming the rule. This is why the context hash alone is not enough — you also want the node identity and the config version stamped on each evaluation, so “which pod, running which rules” is answerable from the log rather than reconstructed after the fact.
• Sticky bucketing pins an old variant on purpose. A user assigned a variant by sticky bucketing keeps it even after the percentage or rule changes. That is correct behavior for consistent experiences, not a bug to fix. Before you spend an afternoon on a “wrong” variant, confirm whether the flag uses a sticky assignment and when this user was first bucketed — the timestamp usually predates the rule change everyone is blaming.
• The clock and the report disagree. A user reports a value they saw an hour ago, but you are reading the rule set as it stands now. If the flag was edited between the observed evaluation and your investigation, you are debugging a decision that no longer exists. Anchor every investigation to the evaluation’s timestamp and the config version live at that moment, not to the present state.
• A default-value mismatch masquerades as a targeting bug. When the in-code fallback passed at the call site differs from the flag’s configured off-variant, a DEFAULT reason returns your source-code value — so two call sites for the same flag can legitimately return different values on a DEFAULT. Grep for the flag key and reconcile every fallback before concluding the rule is inconsistent.

Troubleshooting & FAQ

The reason is TARGETING_MATCH but the user says the value is wrong. What now?

The rule did match — so either the expectation is wrong or the context differs from what you assume. Reproduce with the exact reconstructed context; if the dry-run agrees with production, the rule is doing what it was configured to do, and the conversation moves to what the rule should be.

I get PROVIDER_NOT_READY intermittently. Is that a flag bug?

No — it is an initialization race: code evaluated a flag before the provider finished connecting. Gate first reads behind a readiness signal, or await setProviderAndWait. See server-side SDK integration patterns.

How do I debug a value I cannot reproduce in staging?

The context or the rule set differs between environments. Compare the flag’s targeting rules across environments for configuration drift, and confirm the reconstructed context carries every attribute the production request had.

The value looks wrong but the reason is DISABLED. What does that mean?

The flag is turned off, so the provider is short-circuiting the rules entirely and serving the off-variant — targeting never ran. Do not debug the rule; check who disabled the flag and when, since a DISABLED reason almost always traces to a kill-switch toggle or an expired schedule rather than a targeting mistake.

Two identical-looking requests returned different variants. Is that a bug?

Usually not. Compare the reason and the config version stamped on each evaluation first: if one hit a node with a stale rule set, or one user was pinned by an earlier sticky assignment, the variants differ for legitimate reasons. Only after the reason, node, and config version all match should you treat divergent values from equal contexts as a genuine inconsistency worth escalating.

Should I edit the rule directly in production once I have a hypothesis?

No — confirm the fix against the reconstructed context in staging first. A production edit made on a hunch can shift every user in the cohort, not just the reported one, and if your hypothesis was wrong you have now created a second incident. The staging dry-run costs a minute and turns a guess into proof before anything user-facing moves.

How much of the evaluation context is safe to log for later reconstruction?

Log a stable context hash plus the non-sensitive attributes that drive targeting — plan, country, cohort — and never the raw identifiers or PII themselves. The goal is to rebuild the decision faithfully without turning your log store into a copy of your user table; hash what identifies a person, keep in clear only what a rule actually reads.