Sync vs Async and Amazon SQS

By the end of this lesson you will be able to

• Tell synchronous from asynchronous communication between applications, and explain why a sudden spike or a failing service hurts a tightly coupled system.
• Name the three decoupling models on AWS (queue with SQS, pub/sub with SNS, streaming with Kinesis) and say in one sentence what each does with a message.
• Describe an SQS standard queue: unlimited throughput and backlog, retention (default 4 days, up to 14), message size (1 MiB today), at-least-once delivery and best-effort ordering.
• Follow a message through SendMessage, ReceiveMessage (up to 10 at a time) and DeleteMessage, and predict what happens when a consumer crashes, a duplicate arrives or a message is too big or too old.
• Use SQS to decouple a web tier from a worker tier and as a buffer in front of a database.
• Scale a worker fleet with an Auto Scaling group on the queue, and calculate backlog per instance and the desired capacity.
• Secure a queue: HTTPS in flight, SSE-SQS or SSE-KMS at rest, client-side encryption, IAM policies, and SQS access policies for other accounts and for S3 or SNS.

#Why applications should not always wait for each other

As soon as you run more than one application, they need to talk to each other. An online shop has a buying service that takes orders and a shipping service that books couriers; a payment app has an API that accepts transfers and a back office that settles them. There are two ways for them to talk.

#Synchronous communication

Synchronous means application to application: the buying service calls the shipping service directly (an HTTP request, an RPC) and waits for the answer before it answers its own user.

It is simple, and the caller knows at once whether the work succeeded. The price is tight coupling:

• Both must be up at the same moment. If shipping is down or being redeployed, buying fails too, although nothing is wrong with buying.
• The caller is as slow as the slowest service behind it. If shipping takes two seconds, every shopper waits two seconds.
• Spikes travel straight through. When orders jump a hundredfold, every one of them hits shipping, and then the database, at once. What breaks first takes the rest down with it.

#Asynchronous communication

Asynchronous (event-based) means application to queue to application: the buying service writes the order into a queue and answers the shopper immediately; the shipping service takes orders from the queue when it is ready.

• The producer and the consumer do not need to be up at the same time. The queue holds the work.
• Each side works at its own speed. A spike becomes a longer queue, not an outage.
• Each side scales independently. You can add shipping workers without touching the buying service, and the buying service does not even know how many there are.

The cost: the shopper is told “order received”, not “order shipped”, so the product must be designed around eventual completion (order status pages, notifications), and the system has one more component to monitor.

The slides ask the classic question: you usually encode 10 videos, but suddenly you must encode 1,000. With synchronous calls, the encoding servers are flooded and requests fail. With a queue in the middle, the 1,000 jobs wait in line, the encoders take them as fast as they can, and you add encoders if the line grows. The notes tell the same story with an e-commerce Black Friday (for us, an Eid sale): without a queue the worker tier crashes under the load and new orders stop; with a queue the web tier keeps saying “success” and the workers catch up.

#Three ways to decouple: queue, pub/sub and stream

AWS offers three managed models for asynchronous communication, and this chapter covers each of them. They scale on their own, independently of your application.

Data Firehose (lesson 13.6) delivers streams into storage, and lesson 13.7 puts all of them side by side, with Amazon MQ for applications that already speak standard messaging protocols.

#What Amazon SQS is

Amazon Simple Queue Service (SQS) is a fully managed, serverless message queue. There are no servers, brokers or disks to manage: you create a queue, send messages to it and receive them from it with API calls, and AWS stores them redundantly across several Availability Zones in the Region. It is the oldest AWS service: it went into public beta in 2004 and has been generally available since 2006 (lesson 1.1), so the slides’ “over 10 years old” is now about 20.

It is used to decouple microservices, distributed systems and serverless applications.

#A queue, producers and consumers

• A producer sends messages to the queue. There can be many producers.
• The queue stores each message until a consumer deletes it.
• A consumer polls the queue, receives messages, processes them and deletes them. There can be many consumers; each message goes to one of them at a time.

Producers and consumers can be EC2 instances, containers, servers in your own data centre, or Lambda functions. They only need network access to the SQS endpoint and permission to call it.

#The standard queue at a glance

SQS has two queue types: standard (this lesson and the next) and FIFO (lesson 13.3). A standard queue has these properties:

#Producing messages

A producer calls SendMessage through an SDK, the CLI or an AWS service integration. To send an order to be processed:

• the body: {"order_id": 1001, "customer_id": "C-552", "amount": 1450},
• optional message attributes (metadata outside the body, such as priority=normal), which consumers can read without parsing the body,
• the queue URL, for example https://sqs.ap-southeast-1.amazonaws.com/111122223333/orders-example.

SQS stores the message and returns a message ID. It stays in the queue until a consumer deletes it, or until the retention period ends. SendMessageBatch sends up to 10 messages in one call (up to 1 MiB in total), which saves requests and money. A message can also be delayed before it becomes visible (delay queues and timers, lesson 13.2).

#Consuming messages

A consumer runs a loop:

1. Poll

   Call ReceiveMessage and ask for up to 10 messages at a time. With long polling (lesson 13.2) the call waits up to 20 seconds for a message instead of returning empty at once.

2. Process

   Do the work: for example, insert the order into an RDS database, send an invoice email, resize an image. While the consumer works, the message is hidden from other consumers for the visibility timeout (default 30 seconds; lesson 13.2).

3. Delete

   Call DeleteMessage with the message’s receipt handle. Only now is the message gone. Receiving is not deleting: if the consumer crashes before this step, the message becomes visible again and another consumer gets it.

That last point is what makes SQS safe: a message is removed only after someone confirms the work is done.

#Many consumers: at least once, best-effort order

Several consumers can poll the same queue. Each receives a different set of messages, they work in parallel, and each deletes what it has processed. To process more messages per second, you scale consumers horizontally: add instances, containers or Lambda concurrency. Nothing changes on the producer side.

Two properties of standard queues follow from how SQS spreads messages over many servers:

• At-least-once delivery. Occasionally a message is delivered twice (for example, a server holding a copy was unavailable when you deleted it). Design consumers to be idempotent: processing the same message twice must have the same effect as once. Store the order ID with a unique constraint, or check “already paid?” before charging.
• Best-effort ordering. SQS tries to keep the order of sending, but messages can arrive out of order. If order matters (bank transactions, stock trades) or duplicates are not acceptable, use a FIFO queue (lesson 13.3).

#Decoupling application tiers

#Web tier and worker tier

The slides draw the classic picture: a front-end web application in an Auto Scaling group receives requests and calls SendMessage; a back-end processing application in its own Auto Scaling group calls ReceiveMessage; between them, an SQS queue that is effectively infinitely scalable.

• The web tier stays fast, because it only does the quick part (validate, enqueue, answer).
• The worker tier does the slow part (video processing, invoices, payments, emails) at its own pace, on instance types chosen for that work (GPU instances for video, for example).
• Each tier scales on its own metric: the web tier on requests, the worker tier on the queue.

This is the notes’ e-commerce example. Without SQS, a Black Friday spike can crash the worker tier, and then new orders cannot be placed. With SQS, the web tier pushes every order into the queue and shows the user “success”; the workers pull orders as their capacity allows; pressure on the workers never reaches the users. Elastic Beanstalk’s worker environment (lesson 9.4) is exactly this pattern, packaged.

#SQS as a buffer in front of a database

The slides show a second version of the problem. An application in an Auto Scaling group writes transactions straight into a database (RDS, Aurora or DynamoDB). When the load is too big, the database runs out of connections or write capacity, inserts fail, and some transactions may be lost.

Put a queue in front: the application enqueues each transaction (SendMessage), and a group of workers dequeues it (ReceiveMessage) and inserts it at a rate the database can take. The queue absorbs the spike; nothing is lost; the database sees a steady load. The trade-off is that the write is no longer immediate, so this suits writes the user does not need to read back the next second (orders, logs, payments to settle), not a profile update the user expects to see at once.

#Scaling consumers with an Auto Scaling group

A queue tells you exactly how much work is waiting, which is the perfect signal for scaling the workers.

#The notes’ version: an alarm on queue length

The notes’ example: 10,000 messages arrive suddenly, but there are only 2 EC2 instances, so processing takes a long time. The flow:

1. SQS publishes the CloudWatch metric ApproximateNumberOfMessagesVisible, the number of messages ready to be processed (the slides call it the queue length, ApproximateNumberOfMessages, which is the same number as a queue attribute).
2. A CloudWatch alarm fires when it goes above a threshold, for example 1,000.
3. The alarm triggers a scaling policy of the Auto Scaling group, which launches instances (scale out). When the queue shrinks, another alarm removes them (scale in).

This works, and it is what the slides draw. Two corrections, though. The notes call the policy “Step Tracking”: there is no such thing. The policy types are step scaling (and simple scaling) and target tracking (lesson 6.5). And a fixed threshold on the raw queue length has a weakness: it does not know how many workers you already have. Adding instances does not make the queue length fall in proportion, so the alarm stays red while the queue drains and keeps adding instances, or flaps up and down around the threshold.

#The better version: backlog per instance

AWS recommends a target tracking policy on backlog per instance:

• Backlog per instance = visible messages ÷ number of running (InService) instances.
• Target (acceptable backlog per instance) = longest acceptable wait for a message ÷ average time one instance takes to process a message.

AWS’s own example: 10 instances, 1,500 visible messages, 0.1 s per message, at most 10 s of waiting. Target = 10 ÷ 0.1 = 100 messages per instance. Backlog per instance is 1,500 ÷ 10 = 150, above the target, so the group scales out to 1,500 ÷ 100 = 15 instances.

You no longer have to publish this as a custom metric from a script: a target tracking policy can compute it with metric math from the SQS metric and the group’s GroupInServiceInstances metric. Lesson 6.5 shows the policy; here is what it does to the fleet.

Try your own numbers, including the notes’ 10,000 messages and 2 instances:

Two details that matter in production:

• Scale-in is safe for the messages. When an instance is terminated while it holds a message, the message was never deleted, so it comes back after the visibility timeout. For long jobs, set instance scale-in protection while a worker is busy, so the group terminates idle workers first.
• Lambda scales itself. If the consumer is a Lambda function with an SQS event source mapping, Lambda polls the queue and adds concurrency on its own; there is no Auto Scaling group (serverless chapter, lesson 15.1).

#SQS security

#Encryption

The notes put SSE-SQS under KMS; strictly, only SSE-KMS uses AWS KMS. SSE-SQS keys are managed inside SQS. Either way, SSE encrypts the message body; queue names, message attributes, message IDs and metrics are not encrypted, so do not put secrets in attributes. Encryption applies to messages sent after you turn it on. KMS itself is covered in lesson 21.2; the same SSE-S3 vs SSE-KMS choice appeared for S3 in lesson 10.8.

#Access control: IAM policies and SQS access policies

Two kinds of policy decide who may call the queue (lesson 2.1 and 2.2):

The notes’ exam trick, which the slides repeat: “allow another AWS account to send messages to your queue” or “allow S3 (or SNS) to send messages to your queue” → SQS access policy. For cross-account access, both sides must agree: the queue policy allows account B, and account B’s own IAM policy allows its user or role to call SQS.

A queue policy that lets an S3 bucket send its event notifications (lesson 10.7) to the queue, and only that bucket:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowS3Events",
    "Effect": "Allow",
    "Principal": { "Service": "s3.amazonaws.com" },
    "Action": "sqs:SendMessage",
    "Resource": "arn:aws:sqs:ap-southeast-1:111122223333:orders-example",
    "Condition": {
      "ArnLike": { "aws:SourceArn": "arn:aws:s3:::uploads-example" },
      "StringEquals": { "aws:SourceAccount": "111122223333" }
    }
  }]
}

The conditions matter: without them any bucket in any account could send to your queue through S3 (the “confused deputy” problem). One more trap: if the queue uses SSE-KMS with the AWS managed key alias/aws/sqs, S3 and SNS cannot send to it, because you cannot change that key’s policy to let them use it. Use a customer managed key whose key policy allows the service kms:GenerateDataKey and kms:Decrypt.

Finally, network access: workers in a private subnet with no NAT gateway can still reach SQS through an interface VPC endpoint (AWS PrivateLink), and a queue policy can require that requests come through it (aws:SourceVpce). Details in lesson 22.2.

#What goes wrong

#SQS in system design

• Web tier / worker tier. The front end only validates and enqueues; slow work (video encoding, invoice files, emails, image resizing) runs on workers that scale on the backlog. Users get fast answers and never see a worker outage.
• Write buffer for a database. Absorb bursts of writes so that RDS, Aurora or DynamoDB is sized for the normal load, not the peak.
• Microservice decoupling. Services publish work to queues instead of calling each other, so a deployment or failure in one service does not cascade.
• Event targets. S3 event notifications (lesson 10.7), SNS topics (lesson 13.4) and EventBridge rules (lesson 19.5) can deliver into a queue, so bursts of events are processed at a controlled pace.
• Cost and Spot. Because a message survives the loss of a worker, queue workers are a good fit for Spot Instances: an interruption only means the message is processed by someone else (lesson 4.4).
• What it is not. SQS is not a database (no queries, retention at most 14 days), not a broadcast system (use SNS), and not a replayable log (use Kinesis).

#Try it yourself

Create a queue, send and receive a message, look at its attributes, and read the queue metric. The output is shortened.

# 1. Create a standard queue: 4-day retention by default; set 14 days (in seconds)
aws sqs create-queue --queue-name orders-example \
  --attributes MessageRetentionPeriod=1209600 --region ap-southeast-1
# { "QueueUrl": "https://sqs.ap-southeast-1.amazonaws.com/111122223333/orders-example" }

Q=https://sqs.ap-southeast-1.amazonaws.com/111122223333/orders-example

# 2. Check the defaults: visibility 30 s, max size 1 MiB, SSE-SQS on
aws sqs get-queue-attributes --queue-url $Q --attribute-names All
# "VisibilityTimeout": "30", "MaximumMessageSize": "1048576",
# "MessageRetentionPeriod": "1209600", "SqsManagedSseEnabled": "true", ...

# 3. Produce: a body plus one message attribute
aws sqs send-message --queue-url $Q \
  --message-body '{"order_id":1001,"customer_id":"C-552","amount":1450}' \
  --message-attributes 'priority={DataType=String,StringValue=normal}'
# { "MD5OfMessageBody": "6b1e1c...", "MessageId": "5fea7756-0ea4-451a-a703-a558b933e274" }

# 4. Consume: up to 10 messages, waiting up to 20 s (long polling)
aws sqs receive-message --queue-url $Q --max-number-of-messages 10 \
  --wait-time-seconds 20 --message-attribute-names All
# { "Messages": [{ "MessageId": "5fea7756-...", "ReceiptHandle": "AQEBzbVv...",
#     "Body": "{\"order_id\":1001,...}" }] }

# 5. Delete it with the receipt handle (not the message ID)
aws sqs delete-message --queue-url $Q --receipt-handle "AQEBzbVv..."

# 6. Look at the backlog: the attribute, and the CloudWatch metric the ASG uses
aws sqs get-queue-attributes --queue-url $Q \
  --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible
aws cloudwatch get-metric-statistics --namespace AWS/SQS \
  --metric-name ApproximateNumberOfMessagesVisible \
  --dimensions Name=QueueName,Value=orders-example \
  --start-time 2026-10-26T00:00:00Z --end-time 2026-10-26T01:00:00Z \
  --period 300 --statistics Average

# 7. Allow account 444455556666 to send (queue policy from a file)
aws sqs set-queue-attributes --queue-url $Q --attributes file://policy.json

# 8. Switch to SSE-KMS with your own key, then clean up
aws sqs set-queue-attributes --queue-url $Q \
  --attributes KmsMasterKeyId=alias/orders-key,KmsDataKeyReusePeriodSeconds=300
aws sqs delete-queue --queue-url $Q

policy.json holds {"Policy": "<the queue policy as an escaped JSON string>"}. Run step 4 twice without step 5 and wait 30 seconds: the same message comes back with a new receipt handle. That is the visibility timeout, the subject of the next lesson.

#Summary

Key takeaways

• Synchronous (application to application) couples services: both must be up, the caller is as slow as the slowest, spikes go straight through. Asynchronous (application to queue to application) lets each side run, fail and scale on its own.
• Three models: queue (SQS), one consumer per message, deleted after processing; pub/sub (SNS), a copy pushed to every subscriber; stream (Kinesis), records kept and replayable for many readers. Lessons 13.4, 13.5, 13.7.
• SQS standard: fully managed, serverless, nearly unlimited throughput, unlimited backlog, retention 4 days by default, 1 minute to 14 days, messages up to 1 MiB (256 KB before August 2025; bigger via S3 pointer), at-least-once delivery, best-effort ordering, low latency.
• SendMessage to produce; ReceiveMessage (up to 10) to consume; DeleteMessage after processing. Receiving hides a message, it does not delete it, so a crash means redelivery. Consumers must be idempotent.
• Decouple web and worker tiers, and buffer database writes, so that spikes become a longer queue instead of lost transactions.
• Scale workers with an Auto Scaling group on the queue metric ApproximateNumberOfMessagesVisible; best as target tracking on backlog per instance (target = acceptable wait ÷ time per message). “Step Tracking” is not a policy type.
• Security: HTTPS in flight; at rest SSE-SQS (default, free) or SSE-KMS (your key policy, CloudTrail, KMS cost); client-side encryption for full control. IAM policies for your own account; SQS access policies for other accounts and for S3 / SNS sending to the queue (with aws:SourceArn, and a customer managed KMS key if the queue uses SSE-KMS).

Self-check: Sync vs Async and Amazon SQS31 questions