SimAWSby ShahriarLabs
Search

Decoupling with queues and events · Chapter 2 of 3

How fanout pattern distributes messages

Publishing an event to a topic allows multiple downstream queues to receive the message independently, isolating failures and scaling systems.

14 min2 sectionsBack to Decoupling with queues and events

FoundationsBuilds the idea from nothing. No prior AWS assumed.

The publish subscribe architecture

Why should you send events to a topic instead of calling multiple queues directly?

Order ServicePublisherSNS Topicorders-topicInventory SQSSubscriber 1Shipping SQSSubscriber 2Analytics APISubscriber 3SNS fanout routes a single published message to multiple subscribers in parallel, isolating delivery failures so a single failing endpoint does not block others.

The order service publishes a message to SNS, which fans out the event to all registered subscribers.

1/5
  1. 01

    The integration spaghetti problem

    If an order service must notify the inventory, shipping, and email services, hardcoding three separate queue writes inside the order code makes it difficult to add a fourth service later.

  2. 02

    Publish subscribe with SNS

    Using Simple Notification Service, the order service publishes a single event to an SNS topic. Downstream services subscribe their own SQS queues to this topic to receive the event.

  3. 03

    Failure isolation between consumers

    Because each subscriber has its own queue, if the shipping service goes down, its queue buffers the messages. The inventory and email services continue processing events without interruption.

Check yourself

How does the SNS+SQS fanout pattern isolate failures between multiple consumers of the same event?

Under the hoodThe same thing from underneath: limits, failure modes, numbers.

Subscriber limits and delivery retry shapes

How does SNS deliver messages and retry failed endpoints under the hood?

Order ServicePublisherSNS Topicorders-topicInventory SQSSubscriber 1Shipping SQSSubscriber 2Analytics APISubscriber 3SNS fanout routes a single published message to multiple subscribers in parallel, isolating delivery failures so a single failing endpoint does not block others.

The order service publishes a message to SNS, which fans out the event to all registered subscribers.

1/5
  1. 01

    Filtering messages at the subscription level

    Downstream subscribers can define subscription filter policies. SNS evaluates these policies against message attributes, delivering the message only if the attributes match.

  2. 02

    The SNS retry policy shape

    When delivering to HTTP endpoints, SNS uses a multi-phase retry policy. It executes immediate retries, followed by exponential backoff, before giving up and dropping the message.

  3. 03

    The standard fanout payload ceiling

    Because SNS delivers payloads to SQS, the payload must fit within the SQS size limits. This means large payloads should pass object references, like an S3 path, rather than raw data.

The numbers

Maximum subscribers per topic
12,500,000 subscribersThe default limit for both FIFO and standard SNS topics, allowing massive scaling.
SNS message size limit
256 KBSNS supports messages up to 256 KiB; do not assume the newer larger SQS limit applies to SNS.
Default HTTP retry attempts
100,000 timesThe retry limit for HTTP endpoints over 23 hours, using backoff and jitter.

Check yourself

You want to ensure that a subscriber only receives order events where the total cost is greater than $100. How do you implement this in a serverless design?

Official references & further reading

These lessons simplify selected behaviors for learning. Verify current service limits, Region support and production requirements with the official references. Experiments describe their own assumptions.

Report an error or suggest a clearer explanation →