Recovering from SSE Reconnection Storms
This how-to is part of Polling vs Streaming Flag Synchronization. Your flag stream endpoint restarts, and thousands of SDK clients reconnect in the same instant — a self-inflicted denial of service that keeps the endpoint down. This how-to breaks that storm: jittered backoff so clients do not reconnect in lockstep, a connection cap on the server, a resumable event cursor so reconnects are cheap, and a poll fallback that absorbs the surge while the stream recovers.
The mechanism worth internalizing before you touch any code is that a reconnection storm is not a capacity problem you can buy your way out of — it is a correlation problem. Ten thousand clients each opening one connection is trivial; ten thousand clients opening a connection inside the same 50ms window is a spike an order of magnitude beyond your steady-state provisioning, and it recurs on every restart, deploy, or upstream blip. Doubling the endpoint’s capacity only moves the cliff; the storm scales with your fleet size, and your fleet only grows. The durable fix is to decorrelate the reconnects and to give the server the authority to refuse work it cannot do, so a restart drains and refills over a window instead of arriving as a wall. Every step below either spreads the arrivals in time or hands the server a way to say “not yet” without going dark.
Prerequisites
Step-by-Step Procedure
Step 1 — Jitter the reconnect backoff
Add full jitter to the reconnect delay so clients that dropped together reconnect at randomly spread times rather than in lockstep.
// reconnect.ts — exponential backoff with full jitter
let attempt = 0;
function nextDelayMs(): number {
const capped = Math.min(2 ** attempt * 500, 30_000);
attempt++;
return Math.random() * capped; // full jitter — spread across [0, capped]
}The word “full” is load-bearing here. A common half-measure is equal jitter — capped/2 + Math.random() * capped/2 — which still clusters every client into the back half of the window and leaves the front half empty; under a storm that half-empty spread is often still enough to trip the endpoint. Full jitter draws uniformly across [0, capped], which is the flattest arrival distribution you can produce with a single random draw, and it is what you want when the correlation is total. The exponential base matters on the second wave: if the first reconnect attempt itself fails because the endpoint is still saturated, attempt climbs and the window widens to seconds, then tens of seconds, so a fleet that could not converge in the first window naturally thins out over the next few. Cap the growth — 30s here — so a genuinely recovered endpoint is not left waiting on clients that backed off too far, and reset attempt to zero the moment a stream stays open past a few seconds, or a flapping endpoint will ratchet every client into its maximum delay and freshness will crater fleet-wide.
Step 2 — Resume with Last-Event-ID
Send the last event ID on reconnect so the server replays only what was missed, making each reconnect a cheap catch-up instead of a full state resync.
// resume.ts
const es = new EventSource('/flags/stream', {
headers: { 'Last-Event-ID': lastEventId }, // server resumes from here
});
es.onmessage = (e) => { lastEventId = e.lastEventId; applyDelta(e.data); };Resumption is what turns the reconnect from a thundering herd into a manageable trickle, because it collapses the cost per reconnect, not just the timing of reconnects. Without it, every one of those ten thousand clients asks the server to serialize and ship the full flag set — for a large project that can be hundreds of kilobytes each, and the storm becomes a bandwidth and CPU event on top of a connection event. With a cursor, a client that was offline for two seconds during a restart asks only for the handful of deltas it missed, which is often zero. Make the event ID monotonic and server-authoritative — a database log sequence number or a per-namespace version counter works well — so Last-Event-ID is a precise cursor rather than a timestamp you have to fuzzy-match against. Note the browser EventSource sets Last-Event-ID for you automatically on its own reconnects; the explicit header above matters for server-side SDKs and any client that manages its own reconnect loop, where you must persist lastEventId across the socket teardown yourself or the resume silently degrades to a full resync.
Step 3 — Cap concurrent connections server-side
Limit accepted stream connections and shed excess with a 503 and a Retry-After, so the endpoint protects itself instead of collapsing under the storm.
# server.py — cap connections, shed with Retry-After
async def stream(request):
if active_streams() >= MAX_STREAMS:
return Response(status=503, headers={"Retry-After": str(random.randint(5, 30))})
return await open_stream(request) # within capClient-side jitter is cooperative — it only helps if every client is well-behaved, and you will never control every client in a fleet that includes old SDK versions, misconfigured services, and the one team that hard-coded a one-second reconnect. The server cap is the backstop that does not depend on anyone else behaving: it is a hard ceiling the endpoint enforces on itself. Set MAX_STREAMS from what a single instance can hold, not what it can accept — an SSE connection is cheap to open and then lives for hours, so the binding constraint is usually file descriptors, memory per idle connection, and the fan-out cost of pushing each update to every open socket, not request throughput. Randomize the Retry-After across a range (5–30s here) rather than returning a constant, because a fixed value re-synchronizes the very clients you just shed into a second storm one Retry-After later. Shed at the cheapest possible layer: reject before you allocate the stream buffers, run the flag serializer, or touch the database, so a shed request costs microseconds and the act of refusing load never becomes its own load.
Step 4 — Fall back to polling under shed
When a client is shed (503) or cannot hold a stream, have it poll the fallback endpoint on a jittered interval until a stream slot frees up. Freshness degrades gracefully rather than the client going dark.
// fallback.ts
async function onShed() {
while (!streamAvailable()) {
await sleep(pollInterval() + Math.random() * 5000); // jittered poll
applyDelta(await fetch('/flags/poll').then(r => r.json()));
}
reconnectStream(); // slot freed — resume streaming
}The fallback is what makes the cap safe to enforce. Without it, shedding a client means that client is running on stale or default flag values until it wins a stream slot back, and under a long recovery that window can be minutes — long enough for a bad default to matter. Polling converts “no stream slot” from an availability failure into a freshness degradation: the client is still converging on the truth, just at 10–30 second granularity instead of sub-second, which is exactly the trade you want during an incident. Point the poll at a route backed by cache or a read replica rather than the same hot path the stream serves, so the surge you diverted does not simply reappear on a different endpoint — the whole point is to move load off the resource under pressure. Keep the poll on the same Last-Event-ID cursor so a client can hand its position straight to the stream when a slot frees, and probe for stream availability cheaply — a HEAD request or a slot-count header on the poll response — rather than blindly attempting a full reconnect on every loop, which would itself become storm traffic against the endpoint you are trying to protect.
Verification
Simulate the storm — drop all connections at once — and confirm the endpoint stays up: reconnections spread over time, shed clients poll, and every client converges to the current flag state.
# Kill the stream server, watch reconnect arrival times spread, then check convergence
kill -HUP $(pgrep flag-stream)
# reconnect timestamps should span the backoff window, not cluster at t=0
# and every client's flag hash should equal the server's within one windowDo this test at realistic scale, because the storm is a property that only appears in aggregate — dropping ten connections proves nothing, since ten clients cannot overwhelm anything even in perfect lockstep. Load-test with a client count in the same order of magnitude as production, then plot a histogram of reconnect arrival times: a healthy run shows arrivals spread roughly uniformly across the backoff window, while a broken jitter implementation shows a sharp spike you can see by eye. Watch two server-side numbers during the drain: the 503 shed rate, which should rise then fall to zero as the fleet reconverges, and the peak concurrent stream count, which should plateau at your cap rather than blow past it. The convergence check is the one people skip and regret — flip a flag during the storm and confirm every client, including the ones that spent the incident on the poll fallback, lands on the new value once the dust settles. A recovery that leaves even one percent of the fleet on a stale flag is not a recovery; it is a silent split-brain that surfaces later as an inexplicable behavior difference between two identical-looking hosts.
Gotchas & Edge Cases
- A load balancer idle timeout synchronizes drops. If the LB kills idle streams every 60s, every client reconnects on the same 60s boundary — a recurring storm. Send periodic heartbeat comments to keep the stream non-idle, and confirm the LB timeout exceeds your heartbeat interval.
- The SDK may ignore Retry-After. Shedding with a
503only helps if clients honor theRetry-After. Verify your SDK respects it; if not, the server cap must also drop the connection so the client backs off on its own reconnect policy. - The resume cursor can expire. If the server only retains recent events, a client offline longer than the retention window cannot resume from its
Last-Event-ID. Detect the expired-cursor response and fall back to a full poll resync rather than looping on a resume that can never succeed. - A restart storm can trigger an autoscaler storm. The
503surge and connection spike look like real demand to a naive autoscaler, which spins up instances that are ready just as the storm subsides — then scales back down, dropping those connections and seeding the next wave. Scale streaming tiers on held connection count and a smoothed window, not on request rate or short-lived CPU spikes, so the autoscaler does not amplify the very event you are absorbing. - The full-resync fallback can restampede the origin. If every shed or expired-cursor client resolves to the same full-state poll at once, you have simply relocated the storm from the stream endpoint to the poll endpoint. Serve the full-state poll from a cache with a short TTL so a thousand simultaneous resyncs collapse into one origin read, and jitter the resync just as you jitter the reconnect.
- Health checks and the cap can deadlock. If your readiness probe opens a real stream and the cap is already full of storm traffic, the probe gets shed with a
503, the orchestrator marks the instance unhealthy and restarts it — dropping every connection it held and worsening the storm. Exempt health-check paths from the cap, or probe a lightweight liveness route that never counts againstMAX_STREAMS.
Troubleshooting & FAQ
Our stream endpoint melts down every time it restarts. Is streaming just too fragile?
No — the meltdown is the synchronized reconnect, not streaming itself. Add full jitter to the client backoff and a connection cap with shedding on the server. Those two changes convert a restart from an outage into a brief, absorbed reconnection wave.
Should clients reconnect immediately for freshness?
No. Immediate reconnect is exactly what creates the storm. A small jittered delay costs a second or two of freshness and prevents the fleet from converging on the endpoint at once. For flags that need instant updates, the poll fallback keeps them current while the stream recovers.
How is this different from polling entirely?
Polling avoids the persistent-connection storm but trades away sub-second update latency and adds constant request volume. The hybrid here keeps streaming’s low latency in the normal case and uses polling only as a fallback during recovery — the best of both under stress.
What should MAX_STREAMS actually be set to?
Derive it from what one instance can hold, not what it can accept per second. Measure the memory footprint of an idle SSE connection under your framework, account for the per-update fan-out cost of pushing to every open socket, and leave headroom below your file-descriptor limit. Then divide your total fleet size by instance count to confirm the caps across all instances sum to comfortably more than your steady-state connection count — otherwise you will shed even when there is no storm.
Does HTTP/2 or HTTP/3 multiplexing prevent reconnection storms?
No. Multiplexing lets many streams share one TCP or QUIC connection, which reduces socket and handshake overhead, but a storm is about the application-level reconnect of the flag stream itself, not the transport. When the endpoint restarts, every multiplexed flag stream still tears down and re-establishes at once. You still need jitter, a cap, and a fallback — multiplexing lowers the per-connection cost but does nothing about the correlation in time.
How do I keep the storm from cascading into my flag database?
Insert a cache between the stream and its source of truth, and serve both the resume replay and the poll fallback from it. During a storm the traffic that gets past the cap must not each translate into a database read; a short-TTL cache with request coalescing collapses a thousand simultaneous full-state fetches into a single origin query. Treat the database as the thing you are ultimately protecting — the connection cap guards the endpoint, and the cache guards everything behind it.
Can I just warm connections gradually after a deploy instead of handling the storm?
Staggered startup helps on the deploy you control, but it does not cover the drops you do not — an upstream network blip, a load balancer failover, or a client-side wake-from-sleep all produce storms no rollout schedule can pre-stagger. Graceful connection draining on deploy and jittered reconnect are complementary: the drain spreads the load you cause, and the jitter plus cap absorb the load events cause. Build both; relying on staggered warmup alone leaves you exposed to every drop that is not a deploy.