Benchmarking Rule Engine Throughput

This how-to is part of Optimizing Rule Engine Performance. Before you can claim a rule engine change made evaluation faster, you need a benchmark you trust — one that warms the engine, measures only the evaluation, drives a realistic rule mix, and reports percentiles instead of a misleading average. This how-to builds that benchmark so your optimization work rests on real numbers, not a hot-loop artifact.

The stakes are higher than they look. A flag evaluation sits on the request path of every feature it guards, so an engine that spends 50 microseconds instead of 5 quietly taxes every endpoint that reads a flag — and at hundreds of evaluations per request, that tax compounds. The trouble is that flag evaluation is fast enough that naive timing is dominated by measurement error: at sub-microsecond speeds, the clock resolution, the loop overhead, and the JIT warmup can each be larger than the thing you are trying to measure. A benchmark that ignores those effects will happily report a 3x improvement that vanishes in production, or hide a regression behind noise. The four disciplines below — warm, isolate, realistic mix, percentiles — exist precisely because each one removes a class of measurement lie that would otherwise let you ship the wrong conclusion.

Prerequisites

• The rule engine callable in isolation, without a live provider fetch
• A representative sample of production rules (simple booleans through nested targeting)
• A representative sample of evaluation contexts
• A benchmark harness that records per-call timings, not just totals
• A quiet machine or pinned CPU to reduce measurement noise

Step-by-Step Procedure

Step 1 — Warm the engine before measuring

Run several thousand untimed evaluations first so the JIT has compiled the hot path and any caches are populated, then measure. Cold numbers describe startup, not steady state. On a JIT runtime — the JVM’s C2 compiler, V8’s TurboFan, or PyPy — the first few thousand calls run interpreted or lightly optimized, and the tiered compiler only promotes the method to fully optimized native code after it crosses an invocation threshold. Measure before that promotion and you record the interpreter’s speed, which can be an order of magnitude slower than the code you actually ship. Even on a purely interpreted runtime like CPython, warmup still matters: the first pass populates rule caches, resolves attribute lookups, and pulls the working set into the L1/L2 cache, so the cold call pays for memory traffic that every later call avoids.

How many warmup iterations is enough depends on the runtime’s compilation thresholds — 20,000 is a safe default that clears CPython’s cache-fill and comfortably exceeds the JVM’s default -XX:CompileThreshold of 10,000. The reliable test is empirical: plot per-call latency against iteration number and warm until the curve flattens. If it never flattens, you have a deeper problem — an unbounded cache, a growing allocation, or a slow memory leak — and that is worth finding before you benchmark anything.

# warmup.py
for _ in range(20_000):
    engine.evaluate(sample_rule, sample_ctx)   # untimed — warm the JIT/caches

Step 2 — Isolate the evaluation from setup and I/O

Build the context and fetch the rule once, outside the timed loop, so you measure the evaluation itself and not JSON parsing or a network call. This is where most homegrown benchmarks go wrong: they wrap a timer around client.getBooleanValue(...), which internally deserializes the flag config, constructs an evaluation context, and only then runs the rule. You end up timing the SDK’s plumbing — often the dominant cost — instead of the rule engine you set out to measure. Separating setup from the measured region is what lets you attribute a change to the right layer.

Keep the timed region as small as the language allows. Read the clock immediately before the call and immediately after, and do nothing else inside — no formatting, no logging, no list append that triggers a resize mid-loop. Preallocate the timings buffer to length N so the append never reallocates while you are measuring, and prefer a monotonic, high-resolution clock (time.perf_counter_ns in Python, System.nanoTime on the JVM) over wall-clock time, which can jump backward when NTP adjusts. If the evaluation itself performs I/O — a remote lookup, a segment fetch from a store — that I/O belongs in a different benchmark; a rule engine microbenchmark should touch only in-memory state, or the network jitter will swamp the signal you care about.

# isolate.py — time only engine.evaluate()
rule = engine.compile(load_rule())     # setup outside the loop
ctx = build_context(sample)            # setup outside the loop
timings = []
for _ in range(N):
    t = time.perf_counter_ns()
    engine.evaluate(rule, ctx)         # the ONLY thing timed
    timings.append(time.perf_counter_ns() - t)

Step 3 — Drive a realistic rule-complexity mix

Weight the benchmark toward the distribution of rule complexity you actually run — mostly simple, some nested — so the aggregate reflects production, not a single worst or best case. A rule engine’s cost is not uniform across rules: a bare boolean returns in a handful of nanoseconds, a segment match walks a list and may hash an attribute, and a deeply nested AND/OR tree with regex or semantic-version comparators can cost ten or a hundred times more. If your benchmark is 100% trivial booleans, you will optimize the cheap path and miss the nested rules that dominate your real p99. If it is 100% worst-case, you will chase a tail that almost never fires and possibly regress the common path to help it.

Derive the mix from real data, not intuition. Sample the flag keys your service actually reads over a representative window — a day that spans your traffic patterns — and bucket them by the shape of their targeting rules, then set the weights to match that histogram. The 70/25/5 split above is only an illustration; your service might be 95% simple booleans because most flags are kill switches, or skewed toward segment matches if you do heavy cohort targeting. Getting the weights wrong tilts every downstream percentile, so treat the mix as a measured input, not a guess.

# mix.py — sample rules by production frequency
RULE_MIX = [(simple_bool, 0.7), (segment_match, 0.25), (nested_and_or, 0.05)]
def next_rule():
    r = random.random(); acc = 0
    for rule, weight in RULE_MIX:
        acc += weight
        if r < acc: return rule
    return RULE_MIX[-1][0]

Step 4 — Report percentiles and throughput

Compute p50, p99, and evaluations per second. The average hides the tail that determines user-facing latency; p99 is the number that matters on a hot path. A distribution with a great mean and an ugly tail is common in rule engines: garbage-collection pauses, occasional cache misses, and the rare nested rule all live in the upper percentiles, and a single request that reads dozens of flags will hit that tail with near certainty. If you serve a million requests a day and each reads twenty flags, you evaluate twenty million times daily — your p99 fires two hundred thousand times, so it is not an edge case, it is a routine event you are on the hook for.

Report the throughput figure with care, because “evaluations per second” is ambiguous. The single-threaded figure — one core, one loop — is a clean measure of per-call cost and the right number for comparing engine changes. The aggregate figure your service actually delivers depends on core count, lock contention, and allocator behavior under concurrency, and it does not scale linearly with cores. State which one you are quoting. And beware the mean-based throughput in the snippet: dividing by the average latency inflates the number whenever the distribution is skewed, so a throughput derived from the median is often the more honest headline. Whichever you pick, publish p50 and p99 alongside it — a throughput number with no percentiles is a marketing figure, not an engineering one.

# report.py
timings.sort()
p50 = timings[len(timings)//2] / 1000       # us
p99 = timings[int(len(timings)*0.99)] / 1000
throughput = 1e9 / (sum(timings)/len(timings))  # eval/s
print(f"p50={p50:.2f}us p99={p99:.2f}us throughput={throughput:,.0f}/s")

Verification

Run the benchmark twice unchanged and confirm the p99 is stable between runs; a benchmark whose numbers swing wildly is measuring noise, not the engine. Stability is the precondition for every conclusion you will draw: if the run-to-run variance is 15% and the optimization you are evaluating buys 8%, the benchmark literally cannot see your change, and any number it prints is a coin flip dressed up as data. Establish the noise floor first — run the identical binary several times, note the spread in p99 — and only trust a measured improvement once it clears that spread by a comfortable margin.

If the numbers refuse to settle, the machine is usually the culprit before the code is. Disable CPU frequency scaling and turbo boost so the clock speed does not drift under thermal load, pin the benchmark to an isolated core with taskset or the equivalent, close the noisy neighbors — browsers, sync daemons, container siblings — and prefer bare metal over a burstable cloud instance whose CPU credits throttle mid-run. On a shared CI runner you will rarely get clean numbers, which is why an absolute-latency gate in CI tends to flap; compare relative deltas against a baseline measured on the same host in the same run instead.

python bench.py; python bench.py
# p50=0.41us p99=1.9us throughput=2,300,000/s
# p50=0.42us p99=2.0us throughput=2,280,000/s   <- stable => trustworthy

Gotchas & Edge Cases

• The compiler may delete your benchmark. If the evaluation result is never used, an optimizer can eliminate the call entirely and report near-zero time. Consume the result (accumulate it, assert on it) so the work cannot be optimized away.
• A single context flatters the branch predictor. Evaluating the same context repeatedly lets the CPU predict every branch perfectly, understating real cost. Rotate through a set of varied contexts that exercise different rule branches.
• Microbenchmarks miss contention. A single-threaded microbenchmark says nothing about behavior under concurrent load, where cache contention and allocation pressure appear. Pair the microbenchmark with a concurrent load test before trusting the throughput number in production terms.
• Garbage collection hides in the tail. If the engine allocates per evaluation — a fresh context map, an intermediate result list — those allocations do not cost much on the median call, but they accumulate until a GC pause lands squarely in your p99 or p999. A benchmark long enough to trigger several collections will show the pauses; one that finishes before the first GC will not. Run for enough iterations to cross multiple collection cycles, and watch allocation rate, not just wall time, so a zero-allocation optimization gets the credit it deserves.
• The average is not additive the way you think. It is tempting to benchmark each rule type separately and reconstruct the mixed p99 by weighting the individual percentiles — but percentiles do not compose linearly, and the mixed distribution’s p99 is not the weighted sum of the per-type p99s. If you need the aggregate tail, benchmark the actual weighted mix in one run and read the percentile off the combined sample; do not synthesize it from parts.
• Comparing across machines invalidates the delta. A p99 of 1.9us on your laptop and 3.1us on the CI box does not mean the engine regressed — different CPUs, cache sizes, and memory speeds move the absolute number. Only compare before/after measured on the same host, in the same session, ideally interleaved, so a background disturbance hits both arms equally.

Troubleshooting & FAQ

My benchmark says millions of evals per second but production is slower. Why?

The microbenchmark measures a warm, single-threaded, contention-free path; production adds concurrency, cache pressure, and real context construction. Use the microbenchmark to compare engine changes relatively, and a load test to predict absolute production throughput.

Should I benchmark the compiled rule or the raw rule?

Benchmark what production evaluates. If you precompile rules into an AST, benchmark the compiled form, with compilation done once outside the timed loop — because that is exactly what happens at runtime.

How many iterations are enough?

Enough that the p99 stabilizes between runs — typically hundreds of thousands to millions for a sub-microsecond operation. Increase N until two consecutive runs agree to within a few percent; that convergence is your signal the sample is large enough.

Should I use a benchmarking framework or a hand-rolled loop?

Prefer an established framework — JMH on the JVM, pytest-benchmark or timeit in Python, criterion in Rust, Go’s built-in testing.B. They solve the hard parts you would otherwise get wrong: they defeat dead-code elimination, auto-tune iteration counts until the measurement stabilizes, run isolated forks to avoid cross-contamination, and report percentiles rather than a bare mean. A hand-rolled loop is fine for a quick relative check, but for a number you will cite in a decision, lean on a tool that has already been hardened against the traps in the gotchas above.

Does the benchmark belong in CI, and will it catch regressions?

A microbenchmark on a shared CI runner is too noisy for an absolute-latency gate — CPU credits, neighbors, and frequency scaling make the number flap. What works is a relative gate: measure a fixed baseline and your candidate in the same job, on the same host, back to back, and fail only when the candidate is slower by a margin that comfortably exceeds the run-to-run variance. That catches gross regressions without crying wolf on noise.

How do I benchmark the rule engine without the SDK wrapper around it?

Call the engine’s evaluation function directly rather than going through the client’s getBooleanValue-style API, which bundles context construction, type coercion, hook execution, and telemetry into the measured region. Most engines expose the core evaluate(rule, context) entry point; compile the rule and build the context once outside the loop, then time only that call. If you must go through the SDK, at least disable hooks and telemetry so you are not benchmarking your own instrumentation.

Why is my p99 much worse than my p50, even for simple rules?

A wide gap between median and tail almost always points to something periodic rather than the rule itself — a GC pause, a background thread stealing the core, a cache line bouncing between CPUs, or occasional context reconstruction. Profile the slow samples specifically: capture the timestamps of the calls above p99 and correlate them with GC logs or scheduler events. Nine times out of ten the tail is an artifact of the environment or allocation pattern, not the evaluation logic, and fixing it means reducing per-call allocation or quieting the machine rather than touching the engine.