Buffer background work in Amazon SQS, process it with Lambda or ECS workers under a concurrency ceiling, keep state in DynamoDB, and use dead-letter queues, redrive, and CloudWatch alarms to keep the system honest.
Official AWS sources reviewed 2026-08-29.
Architecture flow
Enqueue the work: An API or domain event writes a small message holding identifiers and an idempotency key, not the payload itself.
Process under a ceiling: Competing consumers on Lambda or ECS read batches; maximum concurrency protects the databases and APIs behind them.
Own the state: The durable store, not the queue, holds job state, results, and the record that makes retries idempotent.
Isolate failures: Partial batch responses return only the failed messages, and a dead-letter queue collects poison messages for redrive.
Scale on the right signal: Queue age and depth drive scaling decisions and alarms; message count alone hides a stalled consumer.
Escalate when it is a workflow: Multi-step business processes with compensation, long waits, or human tasks belong in Step Functions rather than in a chain of queues.
Text alternative: An API or domain event places a small message on an Amazon SQS queue. Competing consumers running on AWS Lambda or Amazon ECS process batches under a maximum concurrency setting and write results to Amazon DynamoDB. Failed messages return through partial batch responses and end in a dead-letter queue for redrive, while Amazon CloudWatch tracks queue age and depth. Multi-step processes graduate to AWS Step Functions.
How the services connect
A queue is a work distribution mechanism, not a database and not an event history. Put identifiers and an idempotency key in the message and keep the payload in S3 or a table; SQS messages are capped at 256 KB and a large message body makes every redelivery more expensive.
Processing is at least once. Lambda event source mappings can deliver the same record more than once, and the visibility timeout controls how long a consumer holds a message before it becomes visible again. Set the timeout above the worst-case processing time, and make the side effects idempotent rather than assuming the timeout is enough.
Concurrency is a protection mechanism. Lambda starts an SQS event source mapping at five concurrent invocations and adds up to 300 more per minute, to a maximum of 1,250 concurrent invocations for one mapping, so an unbounded worker will find the limit of whatever database sits behind it before it finds its own. Use maximum concurrency on the event source, and keep reserved concurrency at or above the sum of those ceilings.
Choose standard queues unless ordering is a requirement. FIFO queues preserve order within a message group, but concurrency is then capped by the number of message groups, so a single group serializes the whole workload.
Return partial batch failures so a single bad message does not send its entire batch back. Give every queue a dead-letter queue, alarm on it above zero, and treat redrive as a deliberate operation after the cause is fixed.
Scale on queue age rather than depth alone. A queue with a hundred messages that are ten minutes old is a different problem from a queue with a hundred messages that arrived a second ago.
Tradeoffs and caveats
Queues absorb bursts, isolate producers from consumer failures, and make retry policy explicit at the cost of eventual consistency and duplicate delivery you have to design for.
Competing consumers scale throughput easily but give no ordering, no view of a multi-step process, and no natural place for compensation logic.
A queue per workload isolates failures and quotas; a shared queue is cheaper to operate and lets one noisy workload starve everything behind it.
Decision points
SQS worker queues versus Step Functions versus streams
Option
Fits
State ownership
Breaks down when
SQS worker queue
Independent units of work: notifications, report generation, media jobs, quota-limited outbound calls.
The consumer and its datastore own all state.
The unit of work becomes a multi-step process with compensation, long waits, or human approval.
Step Functions
Business processes with explicit steps, branches, retries, callbacks, and audit requirements.
The execution holds state and history.
The work is a single high-volume step, where per-transition cost and orchestration add nothing.
Kinesis or Amazon MSK
An event history multiple consumers replay independently, with ordering per partition.
The log is the state; consumers own offsets.
You need per-message retry, per-message visibility, and a dead-letter queue for one bad record.
Standard versus FIFO queues
Queue type
Ordering
Duplicate handling
Concurrency limit
Standard
Best effort, not guaranteed.
At-least-once delivery; your consumer must be idempotent.
Scales to the event source and account concurrency ceiling.
FIFO
Strict order inside a message group.
Deduplication inside the deduplication window, on top of idempotent consumers.
Capped by the number of message group IDs, or the maximum concurrency setting, whichever is lower.
How companies use this
Outcomes below are attributed to their sources, not independently measured. Sources reviewed 2026-08-29.
McGraw-Hill's Open Learning Solutions generated performance reports for instructors and needed more throughput and stability without adding clusters.
Before
A watchdog process polled a PostgreSQL database for pending report requests and started Spark clusters on Databricks, each handling up to ten reports before shutting down. The post reports a five-minute cluster start-up, so cluster time was spent on start-up rather than reports, and the database absorbed the polling.
After
Job identifiers go onto SQS queues, SQS consumers run inside the Spark clusters, and the watchdog uses queue depth to decide when to start a cluster. Task distribution moves off the database.
Service bundle
Amazon SQS, Spark on Databricks running on AWS, and PostgreSQL as the system of record.
Disclosed scale
The post describes a test of 2,030 simulated reports inside a four-hour window.
Failure modes
In the original design the authors report 142 of those reports did not complete inside the window.
Reported outcome
The authors, an AWS product manager and a McGraw-Hill principal data engineer, report that the SQS design completed all 2,030 reports within the window using fewer concurrent clusters, with lower database connection pressure.
What generalizes
Two ideas transfer directly: queue depth is a better scaling signal than database polling, and moving task distribution off the transactional database removes load you were paying for twice.
What does not generalize
The reported numbers come from a simulated workload published by the vendor and the customer together. The post also explains why they chose a task queue over Kafka for this job, which is a scoping decision, not a general ranking of the two.
Grab needed one platform to award loyalty points across many products and external partners without double-awarding or losing awards when a dependency failed.
Before
Point issuance logic sat with individual products rather than in a shared ledger path.
After
A calculation module publishes to a point-awarding SQS queue, a separate retry queue holds work whose dependencies were unavailable, and every message carries an idempotency key that Grab describes as ensuring points are rewarded only once per transaction. A ledger, a notification service, and a Kafka stream sit downstream.
Service bundle
Two SQS queues, a calculation service, an awarding service, a ledger, and Kafka for downstream consumers.
Disclosed scale
Grab describes the platform handling millions of points transactions per day.
Failure modes
The post names dependency outages and missing acknowledgements causing redelivery as the two failure modes the design absorbs.
Reported outcome
Grab reports supporting products, membership tier multipliers, and partner integrations on the same path without significant downtime.
What generalizes
Separating a retry queue from the primary work queue, and carrying an idempotency key that maps to the business transaction rather than to the message, are portable design choices for any ledger-like consumer.
What does not generalize
The scale figure and the availability claim are Grab's own, without independent measurement. A two-queue split also adds ordering questions between the primary and retry paths that the post does not resolve for you.
GrabUnlimited memberships are long-lived state machines with purchases, renewals, and cancellations, and Grab reports subscriber growth of more than 1000 percent between January 2022 and June 2023.
Before
SQS acted as the task manager, delegating state handlers to workers, with Redis locks on a five-minute TTL preventing duplicate processing and a nightly cron job batching renewals.
After
Grab moved the membership lifecycle to Temporal, a durable workflow engine, keeping the queue out of the role of workflow state.
Service bundle
SQS, Redis locks, and cron before; a workflow engine with timers, retries, and resumable executions after.
Disclosed scale
Grab reports the renewal cron caused database surges that persisted even after splitting the work into batches across several jobs.
Failure modes
The post describes race conditions producing memberships that were cancelled and renewed at once, retries without exponential backoff bounded only by the SQS maximum receive count, and multi-step handlers that restarted from step one on failure and generated a new idempotency key, risking double charges.
Reported outcome
Grab reports an 80 percent reduction in production incidents after the migration, along with distributed renewals replacing the nightly surge and fewer corrupted membership states.
What generalizes
The failure signature transfers even where the tooling does not: when retries restart a multi-step process from the beginning, when locks with a TTL are the only concurrency control, and when a batch job is the scheduler, the work has outgrown a queue.
What does not generalize
This is a report on one migration, and the incident-reduction figure covers a period with other changes. The lesson is when to stop using a queue as a workflow engine, not that Temporal is the required answer; Step Functions covers the same ground natively on AWS.
Delivery semantics and failure handling
Lambda event source mappings process each event at least once, and duplicate processing can occur; AWS recommends making the function idempotent.
The visibility timeout, not the queue, decides when unfinished work reappears; set it above the worst-case processing time including retries.
Return batch item failures so a partial failure retries only the failed messages instead of the whole batch.
Give each queue a dead-letter queue, alarm on it above zero, and redrive only after the cause is understood.
Store the idempotency key with the business record, so a duplicate is detected by the transaction it belongs to rather than by the message identifier.
For FIFO queues, remember that concurrency is bounded by the number of message group IDs, so the grouping key is also a throughput decision.
Operational signals to watch
ApproximateAgeOfOldestMessage, the earliest honest signal that consumers are behind.
ApproximateNumberOfMessagesVisible and NotVisible, read together: high not-visible with low throughput means work is stuck in flight.
Dead-letter queue depth and the ApproximateReceiveCount distribution on redriven messages.
Lambda errors, throttles, duration, and concurrent executions, or ECS task count and CPU for container workers.
Downstream error and throttle rates for the databases and APIs the ceiling is protecting.
End-to-end job latency from enqueue to completion, not consumer duration alone.
Cost drivers and quota pressure
SQS request charges, which long polling reduces and aggressive short polling multiplies; each provisioned-mode event poller can issue up to ten polls per second.
Consumer compute: Lambda invocations and duration, or ECS or EC2 capacity held for workers.
Duplicate and retried work, which is paid compute; a poison message retried to the receive limit costs its full processing time each attempt.
Quota pressure: 1,250 concurrent invocations per SQS event source mapping by default, the account concurrency quota shared across functions, and downstream database connection limits.
AWS Periodic Table recommendations
Put a concurrency ceiling on every queue whose consumer touches a shared database, before the first incident rather than after it.
Keep one queue per workload where the failure domains differ; sharing a queue shares the outage.
Move to Step Functions when the unit of work grows a second step that must not repeat, needs compensation, or waits on a human.
Alarm on queue age and dead-letter depth. Throughput dashboards look healthy right up to the moment a consumer stops deleting messages.