SQS FIFO Queues

By the end of this lesson you will be able to

• Explain what a standard queue does not promise (order, no duplicates) and when that breaks a real system, such as a wallet ledger or a stock order book.
• Create a FIFO queue correctly: the .fifo name, chosen at creation, with the message group ID that every message must carry.
• Predict how messages flow through message groups: strict order inside a group, parallel processing across groups, and why a group with a message in flight is held back.
• Use message deduplication IDs and content-based deduplication, and say exactly what the 5-minute deduplication interval catches and what it misses.
• Say what “exactly-once processing” really guarantees, and why consumers must still delete in time and stay idempotent.
• Calculate FIFO throughput: 300 calls per second per API action, 3,000 messages with batching, and the much higher Regional limits of high-throughput mode.
• Combine a FIFO queue with a FIFO dead-letter queue and redrive, and know the price in ordering that this pays.
• Decide when a standard queue is the right answer after all.

#Why order and duplicates matter

#What a standard queue promises

Lesson 13.1 (Sync vs Async and Amazon SQS) introduced the standard queue, SQS’s original queue type. To give nearly unlimited throughput and never lose a message, SQS stores every message redundantly on many servers. That design makes two promises weaker than you might expect:

• Best-effort ordering. Messages are usually delivered in the order they were sent, but sometimes not. Two consumers polling at the same time can pick up message 2 before message 1.
• At-least-once delivery. Every message is delivered at least once, and occasionally more than once, for example when a server holding a copy was unavailable during the delete.

For most work this is fine. Resizing photos, sending a newsletter, indexing a document: the order does not matter and doing a job twice does no harm (the work is idempotent). But some work breaks under both rules.

#When that is not good enough

The answer SQS gives is a second queue type, FIFO (First In, First Out): messages leave in the order they arrived, and duplicates are not introduced. Press play and compare the two:

#What a FIFO queue is

A FIFO queue keeps two promises that a standard queue does not:

1. Ordering. Messages are delivered and processed in the order they were sent, within a message group (next section). The slides draw it as a producer sending 4 3 2 1 and the consumer polling 4 3 2 1, in the same order.
2. Exactly-once processing. SQS does not introduce duplicates: a message sent again with the same deduplication ID within 5 minutes is accepted but not delivered a second time.

The price is limited throughput (300 API calls per second per action, 3,000 messages with batching, more in high-throughput mode) and a slightly higher price per request.

#The .fifo name and other creation rules

• The queue name must end in .fifo, for example orders.fifo. The suffix counts towards the 80-character limit on queue names. You can tell a FIFO queue by its name alone.
• The type is chosen at creation with the attribute FifoQueue=true and can never be changed. You cannot convert a standard queue into a FIFO queue or back: you create a new queue and move producers and consumers to it.
• Everything else you learned still applies: the same visibility timeout, long polling, retention (default 4 days, 1 minute to 14 days), message size (up to 1 MiB) and encryption (lessons 13.1 and 13.2, Visibility Timeout, Long Polling and Dead-Letter Queues).
• Every message sent to a FIFO queue must carry a message group ID, and either a message deduplication ID or the queue must have content-based deduplication turned on. Without them, SendMessage fails.

#Standard vs FIFO at a glance

The notes say a question on this table is almost certain in the exam. Here it is, with the details the notes leave out:

#Message groups

#The message group ID

The ordering of a FIFO queue is not one line for the whole queue. Each message carries a message group ID (MessageGroupId, up to 128 characters), and SQS keeps a separate ordered line per group:

• Inside a group, messages are stored and delivered in the exact order they were sent.
• Between groups there is no order: a message of group B may be processed before an older message of group A.
• If you need the whole queue in one strict order, give every message the same group ID. If several producers send to one group, SQS orders them by arrival; to keep each producer’s own order, give each producer its own group.
• There is no limit on the number of groups in a queue.

The group ID is your answer to the question “what must stay in order?”: one wallet, one order, one bank account, one share symbol, one device. Different wallets have nothing to do with each other, so they can be processed in parallel.

#One message of a group in flight at a time

This is the rule that makes FIFO work, and it surprises people:

• When a consumer receives messages of a group, SQS will not hand out more messages of that group until those messages are deleted or their visibility timeout ends and they become visible again.
• One ReceiveMessage call can return up to 10 messages of the same group at once (still in order), and may fill the rest of the batch with messages of other groups.
• Other consumers polling at that moment get messages of other groups, never the next message of the busy group. You cannot ask for a specific group; SQS picks.
• So the parallelism of a FIFO queue is the number of groups that have messages waiting. Two consumers and one group: one consumer works, the other gets empty replies. A thousand groups: a thousand consumers can work at once.
• If a consumer crashes, the unfinished message becomes visible again after the visibility timeout and is still the head of its group, so it is delivered again before the next one. Order survives the crash; that one group just pauses.

AWS Lambda follows the same rule: with a FIFO queue as event source, the function’s concurrency is capped by the number of active message groups (or the maximum concurrency setting, whichever is lower), and when a batch fails Lambda retries it before taking more messages from the same group. Lambda and event source mappings are covered in depth in chapter 15 (Lambda concurrency).

#Choosing the group key

Good and bad group keys:

• Good: wallet ID, account number, order ID, device ID, share symbol, chat conversation ID. Many distinct values, and order matters only inside each one.
• Too coarse: a constant ("all"), the shop, the restaurant, the Region. A single hot value becomes a bottleneck; one group can only be processed one message at a time.
• Too fine: a random UUID per message. Every message is its own group, so there is no order at all; you have paid for FIFO and got a standard queue with a limit.

#Deduplication

A FIFO queue removes duplicates at the send side. Every message has a message deduplication ID; if a message with the same ID was already accepted in the last 5 minutes (the deduplication interval), the new one is accepted (the producer gets a normal success reply) but not delivered. You give SQS that ID in one of two ways.

#Message deduplication ID

The producer sets MessageDeduplicationId (up to 128 characters) on each message, usually to an ID it already has: the transaction ID, the order ID plus the event type, the hash of a file name and version.

• This is what makes producer retries safe. A phone on a weak connection sends a payment, the reply is lost, the app does not know whether it worked and sends again with the same ID. SQS recognises the ID and does not add a second copy, and the group’s order is not disturbed.
• SQS remembers the ID even after the message has been received and deleted, until the 5 minutes are over.
• An explicit ID always wins: if the queue also has content-based deduplication on, your ID overrides the generated one.
• After a successful send, SQS returns a sequence number: a large, non-consecutive number that keeps increasing within a message group. It shows the order SQS stored.

#Content-based deduplication

If the producer has no good ID, you can turn on ContentBasedDeduplication for the queue. SQS then computes a SHA-256 hash of the message body and uses it as the deduplication ID.

• Only the body is hashed, not the message attributes. Two messages with the same body and different attributes are duplicates.
• It is only safe when equal bodies always mean the same event. If a customer really recharges the same phone number with 100 taka twice within 5 minutes and the body is {"phone":"01711…","amount":100} both times, the second recharge is silently dropped. Put a unique request ID or timestamp in the body, or send an explicit ID.
• If neither an explicit ID nor content-based deduplication is set, the send fails with an error.

#The 5-minute window, and deduplication scope

• The interval is fixed at 5 minutes. A retry that arrives after the window is a new message, and a duplicate gets in. Keep producer retries (with their exponential backoff) inside 5 minutes, and keep an idempotency check in the consumer anyway.
• Deduplication scope can be the whole queue (default) or each message group. With message-group scope, the same deduplication ID in two different groups means two different messages. High-throughput mode requires message-group scope.

#What exactly-once processing really means

The notes’ table says FIFO gives “exactly-once processing (never duplicate)”. That is the exam answer, and it is mostly true, but it is worth knowing exactly where the promise ends:

• What SQS guarantees: it does not introduce duplicates. Producer retries within 5 minutes with the same deduplication ID do not create a second message, and because a group is locked while a message is in flight, two consumers never process the same message at the same time.
• Receive retries: if a ReceiveMessage call fails on the network, the consumer can retry with the same receive request attempt ID (ReceiveRequestAttemptId) and get the same messages back, without disturbing the order.
• What you still owe: if your consumer does the work but does not delete the message before the visibility timeout ends (it was too slow, or it crashed after writing to the database), the message becomes visible again and is processed again. No duplicate was sent, yet the work ran twice. Set the visibility timeout well above the slowest processing time (for Lambda, AWS recommends at least six times the function timeout), extend it with ChangeMessageVisibility for long jobs (lesson 13.2), and make the side effect idempotent (for example, “insert if this transaction ID is not in the processed table”).
• Outside the window: a producer retry after 5 minutes is a new message (previous section).

#Throughput

#300 per second, 3,000 with batching

The limits of a FIFO queue are counted in API calls (transactions) per second, per API action, not in messages:

• 300 TPS for SendMessage, 300 for ReceiveMessage, 300 for DeleteMessage, each counted separately.
• With batching (SendMessageBatch, DeleteMessageBatch, and receives that return up to 10 messages), one call carries up to 10 messages, so 300 × 10 = 3,000 messages per second. This is the slides’ “300 msg/s without batching, 3000 msg/s with”, made precise.
• Each message costs one send, one receive and one delete, so every action has to fit. A load of 1,000 messages per second needs 1,000 calls of each kind without batching (too many), or 100 calls of each kind with batches of 10 (fine).

#High-throughput mode

Since 2021 a FIFO queue can switch to high-throughput mode, and the limits have been raised several times since. You turn it on with one console option, Enable high throughput FIFO, which sets two attributes:

• DeduplicationScope = messageGroup (deduplication per group, see above);
• FifoThroughputLimit = perMessageGroupId (the throughput quota applies per group, not per queue).

If you change either one back, normal throughput applies again. The limits per API action then depend on the Region (checked October 2026):

You can ask for more in the Service Quotas console. The inventory’s “up to 70,000” is correct for the largest Regions; for a Bangladeshi team deploying in Mumbai or Singapore the number to remember is 9,000.

#Partitions and why group IDs must be many

How does a FIFO queue go from 300 to 70,000? It stores its messages in partitions, each replicated across several Availability Zones and managed entirely by SQS. The message group ID is hashed to choose the partition, and each partition supports about 300 calls per second (3,000 messages with batching). SQS adds partitions when traffic grows and removes them when it falls.

The consequence: high-throughput mode only helps if your messages are spread over many distinct group IDs. With one group, everything lands in one partition and you are back to 300 calls per second, whatever the Region allows. AWS recommends group IDs with a large number of distinct values, and batching messages of the same group together when you can.

Try the numbers yourself:

There are two ceilings, and the lower one wins: the queue ceiling (calls per second times batch size), and the ordered processing ceiling, because one group is processed one message at a time: with 100 ms per message, one group does at most 10 messages per second, so 2,500 messages per second need at least 250 active groups.

#FIFO queues and dead-letter queues

Dead-letter queues (DLQs) were introduced in lesson 13.2. With FIFO they work the same way, with a few rules of their own:

• The DLQ of a FIFO queue must also be a FIFO queue (and the DLQ of a standard queue must be standard). It must be in the same account and Region.
• A poison message (one that always fails) is worse in a FIFO queue: it is always the head of its group, so it comes back again and again and blocks every later message of that group. This is head-of-line blocking. Other groups are not affected.
• The redrive policy with maxReceiveCount moves it to the DLQ after that many receives, which unblocks the group. But the next messages of the group are now processed without it, so the order inside that group is broken. AWS’s own advice: don’t use a DLQ with a FIFO queue if breaking the exact order of messages or operations is worse than stopping (their example is the edit list of a video editing suite, where each edit depends on the previous one). In that case alarm on the stuck group and fix the problem instead.
• Retention: in a FIFO DLQ the message’s enqueue timestamp is reset when it moves, so it gets the DLQ’s full retention period. (In a standard DLQ the original timestamp is kept, which is why a standard DLQ needs a longer retention than its source.)
• Redrive: since November 2023, a FIFO DLQ can be redriven back to its FIFO source queue or to another FIFO queue (StartMessageMoveTask, or the console’s “Start DLQ redrive”). Redriven messages get a new message ID and go to the back of their group.

#Other FIFO rules worth knowing

• Delays. A FIFO queue can have a delay for the whole queue (0 to 15 minutes), but no per-message delay: DelaySeconds on an individual message is not allowed, because a delayed message would overtake the order.
• S3 event notifications cannot target a FIFO queue (lesson 10.7, S3 events). Send S3 events to EventBridge and from there to the FIFO queue (EventBridge is covered in chapter 19, EventBridge).
• SNS FIFO topics can fan out ordered, deduplicated messages to several SQS queues; they are covered in lesson 13.4 (SNS, Fan-Out and Message Filtering).
• In-flight limit: up to 120,000 messages received but not yet deleted (AWS raised it from 20,000 in 2024); beyond that SQS returns no error, but processing slows.
• Elastic Beanstalk worker environments can use a FIFO queue, but periodic tasks (cron.yaml) do not work with one (lesson 9.4, Elastic Beanstalk).
• Price: FIFO requests cost a little more (about $0.50 per million in us-east-1, against about $0.40 for standard), and each 64 KB chunk of a payload counts as one request. Only send, receive, delete and change-visibility calls are charged at the FIFO rate.
• Side note, fair queues (July 2025): you may now see MessageGroupId on standard queues too. There it does not mean order: it marks the tenant, and SQS’s fair queues keep one noisy tenant from delaying everybody else, while keeping standard throughput. For the exam, “message group ID” still means a FIFO queue.

#When standard is the right answer

FIFO is not the “better” queue; it is the queue with stronger guarantees and lower limits. Pick standard when:

• Order does not matter or the consumer can restore it (each message carries a timestamp or version, and the consumer ignores older versions).
• The work is idempotent, so an occasional duplicate is harmless (making a thumbnail, sending a cache invalidation, indexing a document).
• Throughput is very high or spiky and you do not want to design message groups.
• The source only supports standard queues, such as S3 event notifications.

Pick FIFO when the order of operations on the same thing is part of correctness (money, stock, state machines, commands to a device) or when duplicates are unacceptable, and the load fits (with batching or high-throughput mode). If you need order per key, very high volume and the ability to replay the data to several consumers, look at Kinesis Data Streams in lesson 13.5 (Kinesis Data Streams); lesson 13.7 compares all of them.

#What goes wrong

#FIFO queues in system design

• Ledgers and payments. Wallet or account ID as group ID, transaction ID as deduplication ID, an idempotent consumer, and a DLQ only if a stuck account is worse than an out-of-order one. Low volume per account, many accounts: perfect FIFO territory.
• Order and workflow state. Order ID as group ID keeps “created → paid → shipped” in sequence while thousands of orders run in parallel. The same pattern fits a ticket booking or a loan application workflow.
• Inventory and catalogue sync. Item ID as group ID, so “set stock to 5, then 3” is applied in that order; last write wins correctly.
• Commands to devices. A tea garden’s irrigation controller must receive “open valve 3” before “close valve 3”: device ID as group ID.
• Fan-out with order. An SNS FIFO topic in front of several FIFO queues delivers the same ordered stream to billing, notifications and analytics (lesson 13.4).
• High-volume ordered streams. Above 3,000 messages per second, high-throughput mode with many groups; for replay or many independent readers of the same data, Kinesis (lesson 13.5).
• Keep FIFO small. Many systems use FIFO only for the part that needs order (the ledger), and a standard queue for everything around it (emails, thumbnails, analytics). FIFO everywhere costs throughput for nothing.

#Try it yourself

The account ID and names are examples. FIFO queues need the .fifo suffix in the name and FifoQueue=true.

# 1. Create a FIFO queue with content-based deduplication off (we will send explicit IDs)
aws sqs create-queue --queue-name payments.fifo \
  --attributes FifoQueue=true,ContentBasedDeduplication=false,VisibilityTimeout=60
# {
#     "QueueUrl": "https://sqs.ap-southeast-1.amazonaws.com/111122223333/payments.fifo"
# }
Q=https://sqs.ap-southeast-1.amazonaws.com/111122223333/payments.fifo

# 2. Send without a group ID: it fails
aws sqs send-message --queue-url $Q --message-body '{"txn":"txn-7781","amount":1200}'
# An error occurred (MissingParameter) when calling the SendMessage operation:
# The request must contain the parameter MessageGroupId.

# 3. Send properly: group = wallet, deduplication ID = transaction
aws sqs send-message --queue-url $Q --message-body '{"txn":"txn-7781","amount":1200}' \
  --message-group-id wallet-0171 --message-deduplication-id txn-7781
# {
#     "MD5OfMessageBody": "3f0e8a8b2c5d4e6f7a8b9c0d1e2f3a4b",
#     "MessageId": "5fea7756-0ea4-451a-a703-a558b933e274",
#     "SequenceNumber": "18849496460467696128"
# }

# 4. "Retry" the same message within 5 minutes: success, same MessageId, no second copy
aws sqs send-message --queue-url $Q --message-body '{"txn":"txn-7781","amount":1200}' \
  --message-group-id wallet-0171 --message-deduplication-id txn-7781
aws sqs get-queue-attributes --queue-url $Q --attribute-names ApproximateNumberOfMessages
# { "Attributes": { "ApproximateNumberOfMessages": "1" } }

# 5. Receive with an attempt ID (safe to retry if the call fails), and see the group and sequence
aws sqs receive-message --queue-url $Q --max-number-of-messages 10 --wait-time-seconds 20 \
  --receive-request-attempt-id try-001 \
  --message-system-attribute-names MessageGroupId SequenceNumber MessageDeduplicationId
#   "Attributes": { "MessageGroupId": "wallet-0171", "SequenceNumber": "18849496460467696128",
#                   "MessageDeduplicationId": "txn-7781" }, "ReceiptHandle": "AQEB…"

# 6. While that message is in flight, a second consumer gets nothing from wallet-0171
aws sqs receive-message --queue-url $Q --wait-time-seconds 5
# (no output: empty reply after 5 s)

aws sqs delete-message --queue-url $Q --receipt-handle "AQEB…"

# 7. A FIFO dead-letter queue, then a redrive policy on the source
aws sqs create-queue --queue-name payments-dlq.fifo --attributes FifoQueue=true
aws sqs set-queue-attributes --queue-url $Q --attributes '{
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:ap-southeast-1:111122223333:payments-dlq.fifo\",\"maxReceiveCount\":\"5\"}"
}'

# 8. Turn on high-throughput mode
aws sqs set-queue-attributes --queue-url $Q \
  --attributes DeduplicationScope=messageGroup,FifoThroughputLimit=perMessageGroupId

# 9. After fixing a bug, move messages from the DLQ back to the source
aws sqs start-message-move-task \
  --source-arn arn:aws:sqs:ap-southeast-1:111122223333:payments-dlq.fifo \
  --destination-arn arn:aws:sqs:ap-southeast-1:111122223333:payments.fifo
# { "TaskHandle": "eyJ0YXNrSWQiOiJk…" }

What to notice: step 2 shows that the group ID is mandatory; step 4 shows deduplication (the retry returns the same MessageId and the queue still holds one message); step 6 shows the group lock. Try step 4 again after 6 minutes: the queue then holds two messages. Delete the queues when you are done.

#Summary

Key takeaways

• A standard queue gives best-effort ordering and at-least-once delivery with nearly unlimited throughput. A FIFO queue gives strict ordering per message group and exactly-once processing, with limited throughput.
• FIFO queue names end in .fifo; the type is fixed at creation (no conversion). Visibility timeout, long polling, retention, message size and encryption work as in a standard queue.
• Every message needs a message group ID. Order is kept inside a group; groups run in parallel; while a group has a message in flight, its next message waits. Parallelism = number of active groups. Pick the smallest key that needs order.
• Deduplication: an explicit message deduplication ID, or content-based deduplication (SHA-256 of the body only). Duplicates within the 5-minute interval are accepted but not delivered. Explicit IDs override content-based ones. Scope: queue or message group.
• Exactly-once means SQS adds no duplicates and never gives one message to two consumers at once. A consumer that misses the visibility timeout still processes twice: stay idempotent.
• Throughput: 300 calls per second per API action, 3,000 messages per second with batches of 10. High-throughput mode (deduplication scope = message group, throughput limit = per message group ID): up to 70,000 calls per second in the largest Regions, 9,000 in Mumbai and Singapore, spread over partitions by group ID hash, so use many groups.
• A FIFO queue’s DLQ must be FIFO. A poison message blocks its group (head-of-line); moving it to the DLQ unblocks the group but breaks that group’s order. FIFO DLQ redrive exists since November 2023.
• No per-message delay on FIFO; S3 events cannot target FIFO (use EventBridge); SNS FIFO topics in lesson 13.4.
• Exam keywords: order critical / no duplicates → FIFO; maximum throughput / order does not matter → standard.

Self-check: SQS FIFO queues30 questions