Publish one event to EventBridge or SNS, give every consumer its own SQS queue, retry policy, and dead-letter queue, and let each consumer scale, fail, and deploy without touching the others.
Official AWS sources reviewed 2026-08-29.
Architecture flow
Produce the event: An object-created event or an application event is published once, with no knowledge of who consumes it.
Route or broadcast: EventBridge matches on event content and routes to selected targets; SNS pushes every matching message to all subscriptions.
Buffer per consumer: Each consumer owns a queue, a redrive policy, a dead-letter queue, and its own permissions.
Consume independently: Consumers scale on their own queue depth, and a slow one grows its own backlog instead of blocking the others.
Watch each lane: Per-queue age, depth, and dead-letter counts show which consumer is behind, not just that something is.
Text alternative: An Amazon S3 or application event is published once to Amazon EventBridge or an Amazon SNS topic. The event is delivered to one Amazon SQS queue per consumer, each with its own dead-letter queue. Independent consumers on AWS Lambda and Amazon ECS read only their own queue, and Amazon CloudWatch reports queue age, depth, and dead-letter counts for each lane separately.
How the services connect
The queue per consumer is the point of the pattern. Without it, a shared subscription makes every consumer share a retry policy, a failure, and a deployment window; with it, each consumer gets its own backlog, its own poison messages, and its own permissions on the message body.
SNS pushes to every subscription that passes its filter policy, which suits broadcast to a known set of endpoints. EventBridge matches on event content, integrates directly with AWS services and partners, and supports archive and replay. S3 can notify SNS, SQS, or Lambda directly, but only one destination per notification configuration, and S3 cannot notify a FIFO queue without routing through EventBridge.
Direct S3 notifications are enough when one consumer needs one event type from one prefix. Move to EventBridge when several consumers want different subsets of the same events, when you want content-based filtering beyond prefix and suffix, or when replay after an outage matters.
Independent consumers are independently timed, and that is the failure mode teams underestimate. Two consumers of the same event reach their conclusions at different moments, so anything that must agree across consumers needs a version in the event or a single owner of the final state.
Version the event contract from the first release. Consumers deploy on their own schedules, so a producer change lands against a mix of old and new consumers; additive fields and an explicit version field cost little at the start and remove the coordinated release later.
Tradeoffs and caveats
One queue per consumer isolates retries, poison messages, permissions, and deployment risk, at the cost of more queues, alarms, and infrastructure code.
EventBridge adds content routing, service integrations, and archive and replay; SNS is simpler and pushes rather than requiring a poller.
Fan-out multiplies delivery: the same event is charged and processed once per consumer, so an unused subscription is a permanent line item.
Decision points
SNS versus EventBridge versus direct SQS
Option
Choose when
Filtering
What you give up
Direct S3 notification to SQS
One consumer, one prefix, no routing decisions.
Prefix and suffix only.
Additional consumers later mean reconfiguring the producer; FIFO queues are not supported as a direct destination.
SNS topic to per-consumer SQS queues
A known set of consumers all wanting the same messages, with per-subscription filters.
Filter policies on message attributes or body.
Content routing across many event types, and replay of what was already delivered.
EventBridge to per-consumer SQS queues
Several consumers wanting different subsets, or you need replay and service integrations.
Content-based rules over the whole event.
One more service to reason about, plus rule and delivery charges.
Shared queue versus queue per consumer
Layout
Failure isolation
Scaling
Use it when
One shared queue
A poison message and a retry storm hit everyone.
One concurrency setting for all work.
One consumer exists, and you expect that to stay true.
One queue per consumer
Each consumer has its own backlog, dead-letter queue, and redrive.
Each consumer scales on its own depth and age.
Two or more consumers, or one consumer whose failure must not stall the others.
How companies use this
Outcomes below are attributed to their sources, not independently measured. Sources reviewed 2026-08-29.
Slack distributes Chef cookbooks to a fleet the post describes as tens of thousands of EC2 instances, and needed consistent cookbook versions across Chef stacks and environments.
Before
S3 bucket notifications sent messages to a single SQS queue, which an application called DishPig consumed hourly to upload cookbooks to one Chef stack across sandbox, dev, and prod.
After
Moving to multiple Chef stacks, S3 published notifications to an SNS topic that fanned out to several SQS queues, one per DishPig deployment. Slack later replaced that with a centralized Chef Librarian service that builds versioned artifacts, uploads them to every stack without activating them, exposes an API to promote a version per environment, and keeps deployment state in DynamoDB.
Service bundle
Amazon S3 event notifications, Amazon SNS fan-out, one Amazon SQS queue per consumer deployment, and DynamoDB for deployment state after the redesign.
Disclosed scale
The post cites tens of thousands of EC2 instances under Chef management.
Failure modes
Slack reports that artifacts built close to the top of the hour, when DishPig ran, produced SQS messages that reached some queues on time and not others, leaving Chef stacks on different cookbook versions.
Reported outcome
Slack reports that promoting explicit versions through a central service, rather than relying on independently timed consumers to converge, gave staged rollouts and visibility into what each environment runs.
What generalizes
The failure is generic: fan-out gives each consumer its own delivery timing, so consumers that must agree on a version need the version carried in the event or decided by one owner. The isolation the per-queue layout provides is real; it just does not produce agreement.
What does not generalize
The specifics are Chef cookbook distribution, where an hourly schedule and a promotion step are acceptable. A low-latency fan-out cannot borrow the scheduled promotion, only the idea that shared state needs a single owner.
Delivery semantics and failure handling
S3 event notifications are at-least-once and not ordered, and delivery to different queues is independently timed.
Every consumer must be idempotent on its own, because a duplicate delivered to one queue says nothing about the others.
Give each queue its own dead-letter queue and redrive policy; a shared dead-letter queue reintroduces the coupling the fan-out removed.
Set the SNS or EventBridge target permissions and the KMS key policy per queue, so a consumer cannot read the messages of another lane.
Use EventBridge archive and replay for recovery rather than asking producers to resend, and make consumers safe to replay.
S3 notification messages can reach 64 KB, so set the queue MaximumMessageSize to at least that or the notification is rejected.
Operational signals to watch
Per-queue ApproximateAgeOfOldestMessage, which identifies the slow consumer by name.
SNS NumberOfNotificationsFailed and EventBridge FailedInvocations for delivery failures before the queue.
Consumer-side Lambda errors and throttles, or ECS task health, per lane.
Divergence checks between consumers that are supposed to converge on the same state.
Cost drivers and quota pressure
The same event is delivered, stored, and processed once per consumer, so cost scales with the number of subscriptions, not the number of events.
EventBridge rule matching and target delivery, or SNS publish and delivery charges, on top of SQS requests.
Archive and replay retention in EventBridge, which is worth paying for deliberately rather than by default.
Quota pressure: EventBridge target and rule limits per bus, SNS subscription limits per topic, and consumer-side concurrency shared with the rest of the account.
AWS Periodic Table recommendations
Give every consumer its own queue on day one; retrofitting isolation after the first shared-queue incident costs more than the extra infrastructure code.
Put a version field in the event contract before you have a second consumer.
When two consumers must agree, give one of them ownership of the decision and let the other read it, rather than hoping both converge.
Delete unused subscriptions. A consumer nobody reads still costs delivery, storage, and an alarm nobody acknowledges.