Retries, idempotency and backpressure · Chapter 2 of 2
How idempotency handles duplicates
Since network failures make duplicate messages inevitable, applications must use idempotency keys and conditional writes to ensure safe execution.
FoundationsBuilds the idea from nothing. No prior AWS assumed.
The inevitability of duplicate messages
Why is it impossible to guarantee exactly-once delivery across a network?
A client initiates a payment request. The network connection drops before the client receives the response.
- 01
The three delivery guarantees
A message broker can deliver messages in three ways: at-most-once (messages can be lost but never duplicated), at-least-once (messages are never lost but can be duplicated), and exactly-once. In a distributed system with network failures, exactly-once delivery is physically impossible without a shared coordinator.
- 02
Why duplicates happen
A client sends a payment request to a server. The server processes the payment successfully and sends an acknowledgment back. However, the network drops the acknowledgment. The client times out, assumes the request failed, and sends the payment request again. The server now sees a duplicate.
- 03
Idempotency is the only cure
An operation is idempotent if running it multiple times produces the exact same state and response as running it once. Since networks cannot guarantee that a message is delivered only once, the receiving application must be designed to handle duplicate messages safely without duplicating the side effects.
Check yourself
Why does a network failure during acknowledgment delivery result in duplicate requests?
Under the hoodThe same thing from underneath: limits, failure modes, numbers.
Implementing deduplication mechanics
How do AWS services and databases identify duplicate requests?
A client initiates a payment request. The network connection drops before the client receives the response.
- 01
Idempotency keys track progress
An idempotency key is a unique identifier generated by the client for a specific operation. Before processing a request, the server stores this key in a database. If a second request arrives with the same key, the server returns the cached response instead of processing it again.
- 02
Conditional writes in DynamoDB
DynamoDB conditional writes allow updates only if specific attributes match your criteria. For example, you can update a bank account balance only if the transaction ID has not already been added to a historical list. If the condition fails, DynamoDB rejects the write.
- 03
FIFO queues handle deduplication
Amazon SQS FIFO queues use a deduplication ID to identify duplicate messages. If a message is sent with an identical deduplication ID within the deduplication window, SQS accepts the message but does not deliver it to consumers.
- 04
SNS and EventBridge deliver at-least-once
Amazon SNS and Amazon EventBridge guarantee at-least-once delivery to their targets. They retry delivery if the target is slow or unreachable, which means your subscriber endpoints (such as Lambda functions) must expect duplicate events.
The numbers
- SQS FIFO deduplication window
- 5 minutesMessages sent with the same deduplication ID within this interval are discarded as duplicates.
- EventBridge retry duration
- Up to 24 hoursEventBridge retries failed deliveries for 24 hours with backoff before sending to a dead-letter queue.
- DynamoDB conditional write exception
- ConditionalCheckFailedExceptionThe specific error returned when a conditional write fails because the target condition was not met.
- SNS HTTP delivery retries
- Up to 100,000 timesSNS retries delivery to HTTP endpoints over several hours using a backoff policy.
Check yourself
What happens if a message is sent to an SQS FIFO queue with a deduplication ID that matches a message processed 10 minutes ago?
In practiceThe judgement call you actually have to make.
Designing idempotency stores
How do you build an idempotency layer without degrading database performance?
- 01
Store keys in a fast database
Do not run expensive table scans to check for duplicate keys. Store idempotency keys in a fast key-value store like DynamoDB or ElastiCache, with a Time to Live (TTL) configured. Set the TTL to match your expected retry window, such as 24 hours.
- 02
Save the response, not just the status
When implementing an idempotency store, you must save the original response body and status code. If a client retries a successful operation, they expect to see the original payload, not a generic success message or a resource-exists error.
- 03
Handle concurrent identical requests
If a client sends two identical requests at the same millisecond, both might check the database, see that the key does not exist, and attempt to process the transaction. Use database unique constraints or conditional writes to ensure only one thread proceeds.
Check yourself
Why should you store the response body along with the idempotency key in your store?
Official references & further reading
These lessons simplify selected behaviors for learning. Verify current service limits, Region support and production requirements with the official references. Experiments describe their own assumptions.
Report an error or suggest a clearer explanation →