Measuring Flag Debt with a Staleness Score

This how-to is part of Managing Flag Deprecation and Cleanup. “We should clean up old flags” is a good intention that never gets prioritized because debt is invisible. This how-to makes it visible: a staleness score computed from a flag’s age, last change, evaluation skew, and code references, so you can rank flags for cleanup, target the worst offenders, and track total flag debt as a number that goes down.

The reason flag debt festers is that no individual flag ever feels urgent — each stale toggle costs you a few extra branches, a slightly harder-to-reason-about rollout, one more entry in the dashboard — and none of that shows up on a burndown chart. A score changes the economics: it converts a diffuse, deferrable liability into a ranked list with a number attached, which is the form work has to take before anyone will schedule it. Aim for a score in the range 0 to 1 per flag so it reads like a probability of “this flag is dead,” and keep the inputs cheap enough to recompute nightly — the whole approach only works if the number stays fresh without a human babysitting it.

Prerequisites

• Flag metadata with creation and last-modified timestamps
• Per-flag evaluation metrics (see observability for flag evaluations)
• A way to count code references to each flag key (a repo grep or index)
• Owners assigned per flag so cleanup work can be routed
• The deprecation and cleanup process to act on the ranking

Step-by-Step Procedure

Step 1 — Gather the four signals per flag

Collect age, days since last change, evaluation skew (how one-sided the variant split is), and the count of code references for each flag. Age and last-change both come from flag-management metadata — most SDKs and control planes stamp a created and modified timestamp on every flag, and if yours does not, an audit log gives you the same thing. Skew comes from your evaluation telemetry: over a trailing window (28 days is a reasonable default), what fraction of resolutions returned the single most-common variant. Code references come from a repo grep or, better, a pre-built symbol index — grepping every repo on every run is slow and misses monorepo edges, so cache the count and refresh it on merge to keep the nightly job fast. Pull all four for a flag in one pass; a flag you cannot get all four signals for is itself a signal that something is misconfigured.

# signals.py — the four inputs
def signals(flag) -> dict:
    return {
        "age_days": days_since(flag.created),
        "since_change_days": days_since(flag.modified),
        "skew": max(flag.variant_share.values()),   # 1.0 = always one variant
        "refs": count_code_references(flag.key),
    }

Step 2 — Normalize and weight into a score

Scale each signal to 0–1 and combine with weights that reflect how strongly each predicts a dead flag — skew and time-since-change usually dominate. Normalization matters as much as the weights: an unbounded raw value (a flag 900 days old, a flag referenced 40 times) would otherwise swamp everything else, so each input is clamped with min(...) to a ceiling past which “more” tells you nothing. The ceilings encode judgment — 365 days for age, 180 days without a change, 5 references as “well-embedded” — and they are the first thing to adjust for your codebase, not the weights. Note the skew term maps a 50/50 split to 0 and a fully one-sided split to 1, so an even experiment contributes nothing to staleness no matter how old it is; that is deliberate, because a flag people are still actively split-testing is not debt. Keep the weights summing to 1 so the output stays in [0, 1] and stays comparable across runs.

# score.py — weighted staleness score in [0, 1]
W = {"age": 0.15, "since_change": 0.35, "skew": 0.4, "refs": 0.1}
def staleness(s: dict) -> float:
    age = min(s["age_days"] / 365, 1)
    stale = min(s["since_change_days"] / 180, 1)
    skew = (s["skew"] - 0.5) / 0.5            # 0.5 split -> 0, one-sided -> 1
    refs = 1 - min(s["refs"] / 5, 1)          # fewer refs -> more stale
    return W["age"]*age + W["since_change"]*stale + W["skew"]*skew + W["refs"]*refs

Step 3 — Rank and route to owners

Sort flags by score and route the worst to their owners as cleanup candidates, so the work lands with the people who can safely remove each flag. Routing to an owner is what turns a report nobody reads into work someone does — a ticket assigned to the team that shipped the flag carries the context needed to delete it safely, whereas a central “please clean up” list gets ignored. Cap the batch (the example takes the top 20) so you create a steady, absorbable trickle of cleanup tickets rather than a 300-item dump that demoralizes everyone and gets closed en masse as “won’t fix.” Stamp the score onto the ticket so the owner can see why the flag surfaced, and re-run the ranking after each cleanup cycle so newly-promoted flags take the place of the ones just removed.

# route.py — top staleness -> owner queue
ranked = sorted(all_flags, key=lambda f: staleness(signals(f)), reverse=True)
for f in ranked[:20]:
    open_cleanup_ticket(owner=f.owner, flag=f.key, score=round(staleness(signals(f)), 2))

Step 4 — Track total debt over time

Sum the scores into a single fleet-wide debt number and chart it, so cleanup effort shows up as a downward trend and new debt is visible as it accrues. A sum rather than an average is the right aggregate here: an average hides fleet growth, because adding ten pristine new flags drags the mean down and makes debt look like it fell when it actually rose. The total, by contrast, moves up when you accumulate debt and down only when you retire it, which is the behavior you want a chart to reward. Emit it on a fixed cadence — weekly is enough to see the trend without chasing daily noise — and consider splitting the total by team so each group can see its own line, which turns a shared abstraction into a scoreboard people act on. When the line trends up for several weeks running, that is your signal to tighten the deprecation and cleanup process, not to fiddle with the score.

# debt.py — one number to track
total_debt = sum(staleness(signals(f)) for f in all_flags)
emit_metric("flag_debt_total", total_debt)   # chart this weekly

Verification

Sanity-check the ranking against known-dead flags: a flag you know is stale (old, one-sided, few references) should score near the top, and a freshly-changed, evenly-split flag near the bottom. This is calibration, not just a passing test — pick five or six flags whose real status you already know cold, score them, and check the ordering matches your gut before you trust the number to route anyone’s work. If a flag you know is dead lands in the middle of the pack, one signal is fighting the others (often a stray dynamic reference inflating the ref count), and that tells you where to look. Keep these assertions in CI so a future weight change that quietly breaks the ranking fails loudly instead of silently mis-prioritizing cleanup for months.

# score.test.py
assert staleness(signals(known_dead_flag)) > 0.8   # old, skewed, unreferenced
assert staleness(signals(active_experiment)) < 0.3 # recent, ~50/50, many refs

Gotchas & Edge Cases

• Permanent flags score high but must not be removed. Kill switches and long-lived operational toggles are old, one-sided, and lightly referenced — exactly the stale profile — yet they are intentional. Tag such flags permanent so the score excludes them or flags them as review-only, never auto-delete.
• Low traffic makes skew unreliable. A flag evaluated ten times a week has a noisy variant split that can look one-sided by chance. Down-weight skew when the sample is small, or require a minimum evaluation count before trusting it.
• Dynamic references evade a grep. A flag key built at runtime from a variable is not found by a literal code search, so the reference count reads zero when the flag is actually used. Never auto-delete on the score alone; the score prioritizes review, and a human confirms before removal — as the cleanup process requires.
• Recently touched but functionally dead flags hide from the last-change signal. A flag whose rollout finished months ago can still show a fresh modified timestamp because someone renamed its description or an automated tool bumped its metadata. Cosmetic edits reset the clock without changing what the flag does, so a flag can read “active” on the time-since-change axis while being fully baked. If you can distinguish targeting changes from metadata changes in your audit log, key the signal off the last targeting change, not the last write of any kind.
• New flags start at zero debt and mask fleet growth. A flag created yesterday scores near zero on every axis, which is correct, but it also means a burst of new flags temporarily makes the fleet look healthy even as the surface area you have to maintain grows. Watch the count of flags alongside the debt total; a flat debt number over a rising flag count means you are only breaking even, and today’s fresh flags are next quarter’s cleanup queue.

Troubleshooting & FAQ

The score flags things that turn out to still be in use. Why?

Usually the code-reference count missed a dynamic or indirect reference, so a live flag looked unreferenced. Treat the score as a prioritized review queue, not a delete list — a human confirms each removal. Improve the reference detection over time, but never let the score delete on its own.

What weights should I use?

Start with skew and time-since-change carrying the most weight, since a flag that never changes and always resolves the same way is the clearest dead-flag signal. Then tune against a set of flags whose real status you know (verification), adjusting until the ranking matches your judgment.

How is the staleness score different from just finding stale flags?

Binary “stale or not” thresholds miss the nuance and produce a flat list you cannot prioritize. A continuous score ranks flags so you attack the worst first, and summing it gives a single debt number you can track down over time — turning cleanup from a one-off purge into a managed, measurable process.

How often should I recompute the score?

Nightly is a good default, with the weekly total charted for trend. The signals barely move day to day, so a nightly job keeps the ranking current without churn, and it runs cheaply if you cache the code-reference count and refresh it on merge rather than grepping every repo on every run. The only thing that needs to be timely is the total-debt metric you chart, and weekly resolution is enough to see whether cleanup is winning.

Should a permanent flag be excluded from the debt total, or just from cleanup routing?

Exclude it from both. A kill switch or long-lived operational toggle scores high by design — old, one-sided, lightly referenced — so leaving it in the total inflates a number you can never bring down, which teaches people to ignore the chart. Tag permanent flags explicitly and filter them out before summing, so the debt total only reflects flags that are genuinely removable.

Can I use the score to gate new flag creation?

Not directly, and you should not try to. The score measures an existing flag’s staleness, not the wisdom of adding a new one, and a hard cap on flag count tends to push teams into reusing one flag for two purposes — which is worse debt than an extra toggle. Use the per-team debt trend as a soft signal instead: a team whose line keeps climbing owes cleanup before it opens new rollouts, enforced socially rather than by a gate that blocks a legitimate launch.