Visibility Timeout, Long Polling and Dead-Letter Queues

By the end of this lesson you will be able to

• Follow one SQS message through its whole life: sent, delayed, visible, in flight, deleted, expired or moved to a dead-letter queue, and say what moves it from one state to the next.
• Explain what the visibility timeout protects against, pick a value for a given job (default 30 seconds, 0 seconds to 12 hours), and fix duplicate processing with a longer timeout or ChangeMessageVisibility heartbeats.
• Tell short polling from long polling, explain empty and false empty responses, turn on long polling (up to 20 seconds) and calculate what it saves.
• Configure a dead-letter queue with a redrive policy and maxReceiveCount, avoid the retention trap, alarm on it, and redrive the messages back after a fix.
• Use delay queues and message timers (up to 15 minutes), and send payloads larger than the 1 MiB message limit with the Extended Client Library and S3.
• Write consumers that survive at-least-once delivery: idempotent, with the right timeouts, batch sizes and clean-up calls.

#The life of a message

Lesson 13.1 showed the big picture: producers send messages to a queue, consumers pull them at their own pace, and the queue absorbs the spikes. This lesson opens the box and looks at what happens to one message between SendMessage and the moment it is gone. These mechanics are where most SQS exam questions, and most SQS production incidents, come from.

#Send, receive, hide, delete

1. Sent. A producer calls SendMessage. SQS stores the message redundantly on several servers across Availability Zones and returns a message ID.
2. Delayed (optional). If the queue has a delay, or the message has its own timer, the message stays invisible for that time first. More on this later.
3. Visible (stored). The message waits in the queue. Any consumer that calls ReceiveMessage can get it. There is no limit on how many messages can be stored.
4. In flight. A consumer receives the message. SQS does not delete it; it hides it from every consumer for the visibility timeout. The consumer now does the work.
5. Deleted. When the work is done, the consumer calls DeleteMessage. Only this removes the message for good.

From the in-flight state there are two other exits. If no delete arrives before the visibility timeout ends, the message becomes visible again (back to step 3) and someone else can try. And if the message has been received too many times, or has been in the queue longer than the retention period (default 4 days, 1 minute to 14 days, from lesson 13.1), it leaves the queue: to a dead-letter queue in the first case (if one is configured), deleted in the second.

SQS never pushes a message to anyone and never decides on its own that the work is done. It only knows two things: someone received the message, and someone deleted it (or did not).

#Receipt handles and receive counts

Every receive returns two things you need to know about:

• Receipt handle. A ticket for this receive of the message. DeleteMessage and ChangeMessageVisibility need it, not the message ID. If the same message is received again later, the new receive gets a new receipt handle; always delete with the most recent one, because a delete with an old handle can succeed without removing the message.
• ApproximateReceiveCount. A counter on the message: how many times it has been received so far. A message received for the third time has a count of 3. Dead-letter queues use this counter.

A single receive call can return 1 to 10 messages (MaxNumberOfMessages); each has its own receipt handle and its own visibility timer.

#The visibility timeout

#What it is and why it exists

When a consumer such as an EC2 instance or a Lambda function receives a message, SQS hides that message from all consumers for a set time. That time is the visibility timeout. Its purpose is to stop several consumers from processing the same message at the same time.

• If the consumer finishes and calls DeleteMessage within the window, the message is gone for good.
• If the consumer fails or crashes, no delete comes. When the window ends, the message is visible again and another consumer (or the same one) can process it.

The owner’s notes call this mechanism the heartbeat of SQS: it is what guarantees that a message is never lost just because the machine working on it died.

#Default, range and where you set it

• Default 30 seconds, minimum 0, maximum 12 hours (43,200 seconds).
• On the queue (VisibilityTimeout attribute): the value every receive uses unless told otherwise.
• On a receive call (VisibilityTimeout parameter of ReceiveMessage): a different value for the messages this call returns.
• On one received message with ChangeMessageVisibility: extend or shorten the timeout while you work. The new value counts from the moment of the call, not from the original receive.
• The 12-hour limit counts from the first receive; extensions never reset it. A job that needs more than 12 hours does not belong in one SQS message: split it into steps or use AWS Step Functions (chapter 15).

#Too short or too long

The default 30 seconds means a consumer has 30 seconds to process and delete a message. Picking the wrong value fails in two opposite ways:

#Extending the timeout: ChangeMessageVisibility and heartbeats

A worker that knows a job will take long, or does not know how long it will take, can ask for more time while it works:

• One extension. Before starting a long job, call ChangeMessageVisibility with a timeout that covers it.
• Heartbeats. Better for jobs whose length varies: start with a modest timeout, and from a background timer extend it again before it ends (for example every 20 seconds, add 60 seconds). While the worker is alive, the message stays hidden; if the worker dies, the heartbeats stop and the message reappears within a minute, not hours later.
• Release early. ChangeMessageVisibility with 0 makes the message visible immediately. Use it when a worker is shutting down (an Auto Scaling scale-in, a Spot interruption notice) or when it receives a message it knows it cannot handle.

#Choosing a value

A good starting point: the visibility timeout is a little longer than the normal processing time of one message, including retries inside your code (for a receive of 10 messages processed one after another, the time for all 10). Add heartbeats for the rare long job. When SQS feeds a Lambda function, AWS recommends a queue visibility timeout of at least six times the function timeout, because Lambda may retry a batch when it is throttled; the details belong to the serverless chapter (lesson 15.1).

Try it yourself: find a combination that duplicates work, one that is safe, and see what a crash does with a short and a very long timeout.

#At-least-once means idempotent consumers

Even a perfect visibility timeout is not a promise of exactly-once processing. A standard queue delivers at least once: on rare occasions a second copy is delivered, for example when SQS cannot confirm a delete on one of its servers, or when a worker dies after doing the work but before deleting. So the consumer must be idempotent: processing the same message twice must have the same effect as processing it once.

Common ways:

• Use a business ID from the message (order ID, payout ID) as an idempotency key when calling a payment provider or writing to the database.
• Keep a table of processed message IDs (with a conditional write, for example in DynamoDB) and skip a message whose ID is already there.
• Make the operation naturally idempotent: “set the order status to PAID” is safe to repeat; “add 500 taka to the balance” is not.

If duplicates are simply not acceptable and you cannot make the consumer idempotent, a FIFO queue with deduplication is the tool, covered in lesson 13.3.

#Short polling and long polling

#Short polling: answers at once

By default a ReceiveMessage call answers immediately, even when the queue is empty. That is short polling. Two things make it wasteful:

• Empty responses. A worker in a tight loop asks again and again and mostly hears “nothing”. Every call is a billed request, so an idle queue still costs money, and the loop burns CPU.
• False empty responses. A short poll asks only a subset of the SQS servers (picked at random) and answers from those. If your message lives on a server that was not asked, the answer is empty although the queue is not. A later poll finds the message, so nothing is lost, but you pay for extra calls and wait longer.

#Long polling: wait up to 20 seconds

With long polling, SQS does not answer empty straight away. It waits up to the wait time (1 to 20 seconds) for a message to arrive, and it asks all servers. As soon as a message is there, it returns at once; only if nothing arrives during the whole wait does it answer empty.

Long polling therefore:

• cuts the number of API calls and the empty responses, which cuts cost;
• removes false empty responses;
• lowers latency: a message is returned the moment it arrives, instead of at the next poll.

#How to turn it on

• On the queue: set ReceiveMessageWaitTimeSeconds above 0, usually 20 (the maximum). Every receive then long-polls.
• On a call: pass WaitTimeSeconds (0 to 20) to ReceiveMessage. The call’s value wins over the queue’s.
• Set your SDK’s HTTP read timeout longer than the wait time, or the client gives up before SQS answers.

Short polling still has a use: when a single thread must poll several queues and cannot block on one of them, or when an application needs an immediate answer. In almost every other case, long polling is the better default.

#Dead-letter queues

#The poison message problem

Sometimes the message itself is bad: a field has the wrong type, a referenced customer was deleted, or the code has a bug that only this input triggers. A consumer receives it, fails, does not delete it; the visibility timeout ends; the next consumer receives it and fails the same way. This infinite loop (a poison message or poison pill) wastes workers and API calls, floods the logs, and in the end the message just expires after the retention period, so nobody can investigate it.

#The redrive policy and maxReceiveCount

The answer is a dead-letter queue (DLQ): an ordinary SQS queue that you create and then name as the target for messages that keep failing. You connect them with a redrive policy on the source queue:

• deadLetterTargetArn: the ARN of the DLQ.
• maxReceiveCount: how many receives are allowed (1 to 1,000). When a message has been received that many times and comes back again, SQS moves it to the DLQ instead of delivering it once more.

With a maxReceiveCount of 3, the message gets three chances; after the third failure it is set aside, the healthy messages behind it flow normally, and your team can read it in the DLQ later to see what was wrong.

How to choose the number: high enough that a short outage of a dependency (a database failover, a throttled API) does not push healthy messages into the DLQ, low enough that a real poison message does not burn many tries. Values between 3 and 10 are common.

#The rules

• The DLQ must be in the same account and Region as the source queue.
• The DLQ must be the same type: a standard queue uses a standard DLQ, a FIFO queue uses a FIFO DLQ.
• The redrive allow policy on the DLQ says which source queues may use it: all (the default), up to 10 named queues (byQueue), or none.
• Do not use a DLQ on a FIFO queue if breaking the exact order is worse than blocking (for example, a list of edits that only make sense in sequence): moving one message aside changes the order of the rest.
• A recent behaviour worth knowing: on a standard queue whose maxReceiveCount is greater than 3, a message that has been received 3 times without being deleted is moved to the back of the queue, so it no longer holds up the age metric of the queue.

#The retention trap

For a standard queue, moving a message to the DLQ does not reset its enqueue time: its age keeps counting from the original send. If the source queue keeps messages for 4 days, the message waited 3 days there, and the DLQ also keeps messages for 4 days, the message disappears from the DLQ after just 1 more day. For a FIFO queue, the timestamp is reset on the move. Either way, the best practice is the same: give the DLQ a longer retention than the source, usually the maximum 14 days.

#Watch it, then redrive

A DLQ only helps if someone notices it. Put a CloudWatch alarm on the DLQ’s ApproximateNumberOfMessagesVisible (greater than 0) and send it to the on-call team (monitoring is covered in chapter 19).

When the bug is fixed, move the messages back with a DLQ redrive: the console’s “Start DLQ redrive” button, or the StartMessageMoveTask API. It moves messages to their source queue (or to another queue of the same type), oldest first, optionally at a capped rate up to 500 messages per second so the workers are not flooded. Redriven messages arrive as new messages with a new message ID, a fresh receive count and a fresh retention period. A redrive task runs for at most 36 hours, and SQS does not filter or change messages during a redrive.

#Dead-letter queues beyond SQS

The idea is the same wherever retries can run out (owner’s notes, 18.others):

• SNS can attach a DLQ, which is an SQS queue, to each subscription, for messages it could not deliver after its retries (lesson 13.4).
• Lambda invoked asynchronously can send failed events to a DLQ (SQS or SNS), though on-failure destinations are now the preferred option (lesson 15.1). When Lambda reads from SQS, the DLQ is configured on the SQS queue, not on the function.
• Elastic Beanstalk worker environments use exactly this mechanism, with their own max retries (lesson 9.4).

#Delay queues and message timers

Sometimes a message should not be processed straight away. A delay queue hides every new message for a set time after it is sent:

• Queue attribute DelaySeconds, 0 to 900 seconds (15 minutes), default 0.
• During the delay the message cannot be received; after it, the message is visible like any other.
• A delay hides a message before its first receive; a visibility timeout hides it after a receive.
• Changing the delay of a standard queue does not affect messages already waiting; for a FIFO queue the change is retroactive.

A message timer sets the delay on one message (DelaySeconds on SendMessage) and overrides the queue’s delay. Timers work on standard queues only; a FIFO queue supports only the queue-level delay. For anything later than 15 minutes, schedule the send with EventBridge Scheduler (chapter 19) instead.

#Large messages: the Extended Client Library

One SQS message can carry at most 1 MiB (1,048,576 bytes). Until August 2025 the limit was 256 KiB, which is what the owner’s notes, the slides and many practice questions still say. Anything bigger has to travel another way.

The Amazon SQS Extended Client Library (for Java and for Python) does it for you:

1. The producer’s client writes the payload to an S3 bucket as an object.
2. It sends a small SQS message holding only a pointer: the bucket and the object key.
3. The consumer, using the same library, receives the pointer and fetches the payload from S3 automatically.
4. When the message is deleted, the library can delete the S3 object too; otherwise add an S3 lifecycle rule (lesson 10.5) so orphaned payloads do not pile up.

Payloads up to 2 GB work this way. The same pattern without the library, “put the data in S3, send its key through the queue”, is how S3 event notifications already work (lesson 10.7).

#Other useful API calls and limits

#Newer features worth a side note

• 1 MiB messages (August 2025), up from 256 KiB. The exam may still use the old number.
• DLQ redrive to source or to a custom queue through the API (2023), and for FIFO queues too.
• Fair queues (2025): on a standard queue, if producers set a MessageGroupId per tenant (on a standard queue it only identifies the tenant and does not order anything), SQS stops one noisy tenant’s backlog from delaying every other tenant’s messages. No consumer change is needed. It is too new to be an exam answer, but useful for multi-tenant SaaS.

#What goes wrong

#SQS mechanics in system design

• Worker tiers behind a queue. Visibility timeout a little above the job time, long polling at 20 seconds, a DLQ with an alarm, idempotent workers. This is the standard recipe, and it is what Elastic Beanstalk worker environments set up for you (lesson 9.4).
• Scaling on the backlog. Combine these settings with the Auto Scaling “backlog per instance” target from lessons 6.5 and 13.1. A worker that holds messages too long (a timeout far above the job time) makes the backlog look smaller than the real work.
• Lambda as the consumer. The Lambda service long-polls the queue for you (an event source mapping), deletes messages when the function succeeds, and with partial batch responses returns only the failed ones. Set the queue’s visibility timeout to at least six times the function timeout, and put the DLQ on the queue (chapter 15).
• Delayed work without a scheduler. Reminders, retries with back-off, “cancel the order if unpaid after 10 minutes” within 15 minutes → delay queues and message timers; longer → EventBridge Scheduler.
• Big documents and media. Payload in S3, pointer in SQS, either with the Extended Client Library or by sending the S3 key yourself.
• Long workflows. Anything that can exceed 12 hours, or has many steps with waits and branches, belongs in Step Functions, with SQS for the individual tasks.

#Try it yourself

The queue URLs below use the example account 111122223333 in Singapore (ap-southeast-1).

# 1. A DLQ with 14-day retention, then a source queue with long polling, a 3-minute visibility
#    timeout and a redrive policy (maxReceiveCount 3)
aws sqs create-queue --queue-name orders-dlq-example \
  --attributes MessageRetentionPeriod=1209600
aws sqs create-queue --queue-name orders-example --attributes '{
  "VisibilityTimeout": "180",
  "ReceiveMessageWaitTimeSeconds": "20",
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:ap-southeast-1:111122223333:orders-dlq-example\",\"maxReceiveCount\":\"3\"}"
}'
#   "QueueUrl": "https://sqs.ap-southeast-1.amazonaws.com/111122223333/orders-example"

Q=https://sqs.ap-southeast-1.amazonaws.com/111122223333/orders-example

# 2. Send an order, and one with a 60-second message timer
aws sqs send-message --queue-url $Q --message-body '{"order":4521,"amount":500}'
aws sqs send-message --queue-url $Q --message-body '{"order":4522}' --delay-seconds 60

# 3. Receive with long polling and ask for the receive count
aws sqs receive-message --queue-url $Q --wait-time-seconds 20 --max-number-of-messages 10 \
  --message-system-attribute-names ApproximateReceiveCount
#   "Messages": [{ "MessageId": "5fea7756-0ea4-451a-a703-a558b933e274",
#     "ReceiptHandle": "AQEBzbVv...fqNzFw==",
#     "Body": "{\"order\":4521,\"amount\":500}",
#     "Attributes": { "ApproximateReceiveCount": "1" } }]

# 4. Need more time? Extend this receive by 5 minutes (counted from now)
aws sqs change-message-visibility --queue-url $Q \
  --receipt-handle "AQEBzbVv...fqNzFw==" --visibility-timeout 300

# 5. Done: delete it with the latest receipt handle
aws sqs delete-message --queue-url $Q --receipt-handle "AQEBzbVv...fqNzFw=="

# 6. See what is waiting, in flight and delayed
aws sqs get-queue-attributes --queue-url $Q --attribute-names \
  ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible ApproximateNumberOfMessagesDelayed
#   "ApproximateNumberOfMessages": "0", "ApproximateNumberOfMessagesNotVisible": "0",
#   "ApproximateNumberOfMessagesDelayed": "1"

# 7. Make a poison message: receive it 3 times without deleting, and let the 180 s pass each time
#    (or set the visibility timeout to 5 s for the test). On the next receive it is in the DLQ:
aws sqs get-queue-attributes --attribute-names ApproximateNumberOfMessages \
  --queue-url https://sqs.ap-southeast-1.amazonaws.com/111122223333/orders-dlq-example
#   "ApproximateNumberOfMessages": "1"

# 8. After the fix, redrive the DLQ back to the source queue at 10 messages per second
aws sqs start-message-move-task \
  --source-arn arn:aws:sqs:ap-southeast-1:111122223333:orders-dlq-example \
  --max-number-of-messages-per-second 10
#   "TaskHandle": "eyJ0YXNrSWQiOiJkYzE..."
aws sqs list-message-move-tasks \
  --source-arn arn:aws:sqs:ap-southeast-1:111122223333:orders-dlq-example
#   "Status": "COMPLETED", "ApproximateNumberOfMessagesMoved": 1

# 9. Clean up (purge cannot be undone)
aws sqs purge-queue --queue-url $Q
aws sqs delete-queue --queue-url $Q
aws sqs delete-queue --queue-url https://sqs.ap-southeast-1.amazonaws.com/111122223333/orders-dlq-example

Leaving out --destination-arn in step 8 sends the messages back to the queue they came from. In the console, the same steps are under the queue’s Send and receive messages page (poll for messages, see the receive count, delete) and the DLQ’s Start DLQ redrive button.

#Summary

Key takeaways

• A message lives through sent → (delayed) → visible → in flight → deleted. Receiving never deletes it; only DeleteMessage with the latest receipt handle does. Other exits: back to visible when the timeout ends, to the DLQ after too many receives, deleted when retention (default 4 days, max 14) runs out.
• Visibility timeout: hides a received message from all consumers so it is not processed twice at once. Default 30 s, 0 s to 12 h. Set on the queue, per receive, or per message with ChangeMessageVisibility (counted from the call; 12 h cap from the first receive; 0 releases it at once).
• Too short → duplicate processing; too long → slow retries after a crash. Use a value a little above the normal job time and heartbeats for long jobs. Exam: 2-minute job processed twice → increase the visibility timeout / ChangeMessageVisibility.
• Standard queues deliver at least once, so consumers must be idempotent (idempotency keys, processed-ID tables). FIFO for exactly-once processing (lesson 13.3).
• Short polling (wait 0, default) answers at once, asks only some servers: empty and false empty responses, higher cost. Long polling waits 1–20 s (20 preferred), asks all servers, returns the moment a message arrives. Turn on with ReceiveMessageWaitTimeSeconds (queue) or WaitTimeSeconds (call, wins). Exam: reduce cost / empty responses → long polling.
• DLQ: an ordinary queue that a redrive policy on the source names, with maxReceiveCount (1–1,000). Same account, Region and type (FIFO → FIFO DLQ). Give it longer retention (14 days), alarm on it, and redrive back after the fix (new messages). Exam: poison pill, infinite loop, troubleshoot failures → DLQ.
• Delay queue 0–15 min for every message; message timer per message (standard only); later than 15 min → EventBridge Scheduler.
• Max message 1 MiB (256 KiB until 2025). Bigger → Extended Client Library: payload in S3 (up to 2 GB), pointer in SQS.
• Handy calls: batches of 10, PurgeQueue (all messages, up to 60 s, irreversible), about 120,000 in-flight messages, every 64 KB chunk billed as a request.

Self-check: SQS visibility timeout, polling and DLQs30 questions