Retry storms: when recovery logic multiplies the outage

Count attempts across layers

Suppose an export request crosses three components and each permits three attempts. In a worst-case nested failure, downstream attempts can multiply to 27. The system sees much more traffic while producing no more successful exports.

The model below deliberately shows this worst-case amplification. Real request distributions and retry scheduling vary. The point is to inspect every layer rather than assume a small retry count is harmless.

Spread retries and bound the budget

Exponential backoff increases delay between successive retries. Jitter spreads clients in time so they do not all return together. Neither creates downstream capacity; requests still need deadlines and an attempt budget.

Choose the layer that has enough information to retry safely. Propagate deadlines and stop retrying permanent failures. A rate-limited or unavailable service can require a different response from invalid input or denied access.

A timeout does not prove nothing happened

The server may have committed the operation before the caller lost the response. Retrying a payment or export request without a stable idempotency key can duplicate its side effect.

A useful key identifies the intended operation, and the service must remember its outcome for an appropriate window. A random new key on every retry defeats deduplication. The failure model needs to include both caller uncertainty and server state.

Practice a repair

Increase retry layers and attempts, then compare the estimated downstream load with capacity. Reduce nested retries and inspect the difference. Backoff and jitter should be treated as scheduling strategies, while this calculator focuses on the maximum number of attempts.

In production, monitor useful success rate, request attempts and tail latency together. A rising request count during an outage can mean amplification rather than increased customer demand.

Sources and scope

Reviewed against these official references. The model’s supported scope appears alongside its controls.

• AWS Builders Library: timeouts, retries, backoff and jitter