Calculating Sample Size for Flag Experiments

This how-to is part of Experimentation and A/B Testing Guardrails. You are about to flag-gate an experiment and someone asks “how long do we run it?” — and the honest answer requires a sample-size calculation, not a guess. This how-to computes the sample size an experiment needs from four inputs — baseline rate, minimum detectable effect, power, and significance — and turns that number into a realistic decision date given your traffic.

The calculation matters because the two failure modes it prevents are both invisible until it is too late. Run an underpowered experiment and you will call a real improvement “no significant difference,” shipping nothing and quietly training the team to distrust experimentation. Run one with no fixed endpoint and you will stop the day the numbers happen to look good, banking a false positive that a monitored rollout would never have hidden. A sample size is the number that turns “we tested it” into a claim you can defend six months later. Do the arithmetic once, before a single user is bucketed, and the rest of the experiment becomes a matter of waiting rather than negotiating.

Prerequisites

• A single primary metric with a known baseline (conversion rate, click rate)
• The minimum detectable effect you actually care about, in relative or absolute terms
• Target statistical power (commonly 0.8) and significance level (commonly 0.05)
• A reliable estimate of eligible traffic per day
• A cohort assignment that splits traffic evenly

Step-by-Step Procedure

Step 1 — Fix the minimum detectable effect

Decide the smallest change worth detecting — the effect below which you would not act anyway. Setting it too small demands enormous traffic for a difference you would ignore. The MDE is a product decision disguised as a statistical one: if shipping the variant costs two weeks of engineering and a 0.3% conversion lift would not pay that back, then 0.3% is below your MDE and there is no point powering for it. Sample size scales with roughly the inverse square of the MDE, so this is the single most expensive knob on the whole calculation — halving the effect you want to catch roughly quadruples the traffic and time you must spend to catch it. Write the MDE down as a number with a business justification attached, not as “whatever we can detect,” because “whatever we can detect” is how experiments quietly balloon to eight-week runs nobody agreed to.

# mde.py — express MDE relative to the baseline
baseline = 0.12          # 12% conversion
mde_rel = 0.05           # care about a 5% relative lift
target = baseline * (1 + mde_rel)   # 0.126

Step 2 — Compute sample size per variant

Use the standard two-proportion sample-size formula with your power and significance to get the number of subjects each variant needs. Power (conventionally 0.8) is the probability you detect the effect when it is genuinely there; significance alpha (conventionally 0.05) is the probability you claim an effect when there is none. Both feed the formula through their critical z-values, so nudging power from 0.8 to 0.9 or alpha from 0.05 to 0.01 is not free — each pushes the required sample up by tens of percent. Treat proportion_effectsize as computing Cohen’s h, the arcsine-transformed distance between the two rates, which is what keeps the formula honest at very low or very high baselines where a raw percentage-point difference misleads.

# sample.py — two-proportion sample size
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

effect = proportion_effectsize(0.126, 0.12)
n_per_variant = NormalIndPower().solve_power(
    effect_size=effect, alpha=0.05, power=0.8, alternative="two-sided")
print(round(n_per_variant))   # subjects needed in EACH of canary and control

Step 3 — Convert sample size to a decision date

Divide the total required subjects by the eligible traffic per day to get a realistic run length, so you commit to a date before starting, not after. The denominator is where most estimates go wrong: eligible traffic is not total traffic. Only users who actually reach the flagged surface, pass its targeting rules, and get bucketed into the experiment count toward the sample, so a flag scoped to logged-in checkout users on namespace.checkout.new-flow may see a fraction of the daily site visitors you first pictured. Round the run length up to whole weeks, not days — conversion behavior swings hard between weekdays and weekends, and a seven-day multiple keeps each variant balanced across the same mix of day-of-week traffic rather than ending mid-week on an unrepresentative slice.

# duration.py
total_needed = 2 * round(n_per_variant)     # both variants
per_day = eligible_traffic_per_day * fraction_in_experiment
days = math.ceil(total_needed / per_day)
print(f"run for ~{days} days to reach the required sample")

Step 4 — Pre-register the stopping rule

Commit to the sample size and decision date in advance, so you stop when the plan says — not when the numbers first look good, which is the peeking trap. Pre-registration is not ceremony; it is the thing that converts your alpha of 0.05 from a hopeful label into an actual guarantee. Store the plan somewhere immutable and timestamped — a committed file, a ticket, an experiment-platform record — so that when the result comes in nobody can quietly relitigate the metric or the threshold after seeing the data. Include the guardrail metrics and their halt thresholds in the same record, because a stopping rule that only governs the primary metric still leaves you free to bail out early on a scary-looking secondary, which is peeking wearing a safety vest.

# plan.py — the pre-registered rule
experiment_plan = {
    "primary_metric": "checkout_conversion",
    "n_per_variant": round(n_per_variant),
    "decision_date": start + timedelta(days=days),   # do not stop before this
    "alpha": 0.05, "power": 0.8,
}

Verification

Sanity-check the number against a power simulation: run many simulated experiments at the computed sample size and confirm the true positive rate matches the target power. The closed-form formula rests on a normal approximation that frays at small n or extreme rates, and a simulation costs seconds while an underpowered experiment costs weeks — so pay the seconds. The check is also the cleanest way to catch a subtle bug in your own effect-size call: if the simulation reports 0.62 where you expected 0.80, your MDE and your formula have drifted apart, and it is far cheaper to find that here than in the post-mortem. Run the same loop with both variants set to the baseline rate and confirm the false-positive rate lands near your alpha — that second simulation validates the other half of the guarantee, that you are not crying wolf when nothing changed.

# power_check.py — simulate to confirm power
hits = 0
for _ in range(5000):
    a = bernoulli(0.126, round(n_per_variant))
    b = bernoulli(0.12,  round(n_per_variant))
    if two_prop_pvalue(a, b) < 0.05: hits += 1
print(hits / 5000)   # should be ~0.8, matching target power

Gotchas & Edge Cases

• Multiple metrics inflate the error rate. Testing five metrics at alpha 0.05 makes a false positive on at least one likely. Correct the significance level (Bonferroni or similar) or designate one primary metric that decides, with the rest as guardrails.
• A novelty effect skews early data. Users often react to any change at first, so the first days overstate the effect. Run the full pre-registered window rather than reading the early spike, and consider excluding a warm-up period.
• Early stopping voids the guarantee. Stopping the moment the p-value dips below 0.05 inflates false positives exactly like peeking in canary analysis. Honor the pre-registered decision date, or use a sequential design built for continuous looks.
• The randomization unit must match the analysis unit. If you bucket by user but count events — sessions, clicks, page views — the observations are correlated and your effective sample size is smaller than the raw event count, so the naive formula overstates your power. Either analyze per-user aggregates or apply a variance correction for the clustering; otherwise the number you computed is optimistic fiction.
• Assignment imbalance is a symptom, not a rounding error. A 50/50 split that arrives as 48/52 usually means a targeting rule, a caching layer, or a bot filter is quietly dropping subjects from one arm. Run a sample-ratio-mismatch check before you trust any result — a failed ratio test invalidates the experiment no matter how clean the p-value looks, because the two arms are no longer comparable populations.
• Dilution by ineligible traffic wastes power. If users who can never convert on the flagged surface still land in the experiment, they add noise to both arms and drag the observed effect toward zero, so you need more subjects to clear the same MDE. Scope the flag’s targeting so only genuinely eligible users are bucketed, and keep the eligibility filter identical across variants.

Troubleshooting & FAQ

The required sample size is larger than our total traffic. Now what?

Your minimum detectable effect is too ambitious for your traffic. Either accept a larger MDE (detect only bigger effects), run longer to accumulate more subjects, or reconsider whether an experiment is the right tool — a low-traffic surface may be better served by a monitored rollout with guardrails than a powered A/B test.

Can I just run until it looks significant?

No — that is early stopping, and it inflates your false-positive rate well above the nominal 0.05. Commit to the sample size up front and read the result at the decision date. If you genuinely need to monitor continuously, use a sequential test with the appropriate correction.

Does sticky bucketing affect the sample size?

It affects who is in the sample, not the size. Sticky bucketing ensures each user stays in one variant, which is required for a valid experiment. The count you need per variant is unchanged; sticky assignment just guarantees the counts are clean.

Should I use a relative or an absolute minimum detectable effect?

Use whichever the business actually reasons in, but be explicit about it. An absolute MDE (“a 1 percentage-point lift”) is a fixed distance regardless of baseline; a relative MDE (“a 5% lift”) scales with the baseline, so 5% on a 2% conversion rate is a much smaller absolute move than 5% on a 40% rate. The two produce very different sample sizes, and the most common mistake is computing one while everyone in the room is picturing the other. Pin the definition next to the number in the pre-registered plan.

How many variants can I add without recomputing?

Each additional variant is another comparison against control, and every comparison spends alpha, so more arms means either a stricter per-comparison significance level or a higher false-positive rate. A four-arm test does not just split traffic four ways — after correcting alpha it needs meaningfully more subjects per arm than a two-arm test at the same MDE. Recompute for the corrected alpha, and remember the traffic is now divided across all arms, so the decision date moves out on both counts.

Do I need a sample size if I’m only rolling out, not experimenting?

Not the same way. A monitored rollout asks “is this safe?” and can lean on guardrail thresholds and anomaly detection rather than a powered comparison, so it does not need a pre-computed n to proceed. The moment you want to make a causal claim — “the new flow lifted conversion” — you are experimenting, and the claim is only as trustworthy as the power behind it. If the surface has too little traffic to power a real test, favor a guardrailed rollout and resist reporting its outcome as a measured effect.

What baseline rate should I use if I don’t have one yet?

Measure it first, from a week or two of production data on the same eligible population, rather than guessing. The baseline enters the effect-size calculation directly, so a baseline off by a few points can swing the required sample by a large margin in either direction. If you truly cannot measure it, compute the sample size across a plausible range of baselines and size for the worst case in that range — an underestimate of the baseline that leaves you underpowered is the expensive direction to be wrong in.