Event Processing Patterns

By the end of this lesson you will be able to

• Wire SQS, SNS and Lambda together end to end, and pick the right one from a question’s words: buffer and decouple (SQS), push to many (SNS), run code on an event (Lambda).
• Explain the three retry shapes behind a Lambda function: SQS + Lambda (pollers, visibility timeout, maxReceiveCount, the queue’s DLQ, partial batch responses), SQS FIFO + Lambda (order per message group, a failing message blocks its group), and SNS + Lambda (asynchronous invoke, two Lambda retries, on-failure destination, the subscription DLQ).
• Set up a dead-letter queue correctly for each shape, and redrive messages after a fix.
• Choose SNS → SQS fan-out over writing to several queues from code, and make it work (queue policy, filter policies, S3 events through SNS).
• Design S3 event notifications (event types, prefix and suffix, destinations, the overlap rule, loops) and know when to send S3 events to EventBridge instead.
• React to any AWS API call with CloudTrail + EventBridge, and avoid the Region and read-only traps.
• Ingest a stream through API Gateway straight into Kinesis Data Streams with an AWS service integration, and handle throttling.

#Event processing: the architect’s view

Every building block of this lesson already has its own lesson. Here the question is different: how do several of them fit together, what each hop can and cannot do, and where the whole thing breaks. The notes start with the same warning: this area produces many scenario questions, and they are won by recognising the shape.

#The core trio: Lambda, SNS and SQS

The notes call these three the foundation of event-driven architecture on AWS, and lesson 13.7 compares them in depth. The short version, with the one correction the exam likes:

• SQS decouples producer and consumer. It is pull-based: an EC2 worker, a container or Lambda takes messages when it is ready.
• SNS is pub/sub and push-based: one publish reaches every subscriber at once (email, SMS, HTTPS, SQS, Lambda, Firehose).
• Lambda runs code when an event happens. With SNS, SNS invokes the function. With SQS, nobody pushes: the Lambda service runs an event source mapping whose pollers read the queue for you.

Why not just call the next service directly? A direct call ties the caller to the callee: the caller waits, fails when the callee fails, and changes whenever a new consumer appears. Every pattern below removes one of those ties.

#Three retry shapes behind a Lambda function

The first slide of the section draws the same function behind three sources and labels each with how it retries. This is the heart of the lesson, because the source decides who retries, how often, in what order, and where a failed message ends.

#SQS + Lambda: the pollers try and retry

1. The event source mapping long-polls the queue and receives a batch (up to 10 messages by default; more with a batching window).
2. It invokes the function synchronously with the batch. The messages are now in flight, hidden for the queue’s visibility timeout.
3. If the function succeeds, Lambda deletes the messages.
4. If it fails, nothing is deleted. After the visibility timeout the messages are visible again and are received again. Each receive raises the message’s receive count.
5. When the receive count passes the redrive policy’s maxReceiveCount, SQS moves the message to the queue’s DLQ.

Two settings make this shape work well:

• Partial batch responses. By default one exception fails the whole batch, so nine good messages are processed again with the bad one. Turn on ReportBatchItemFailures on the event source mapping and return the IDs of only the failed messages: Lambda deletes the rest.
• Visibility timeout at least 6 times the function timeout. Otherwise a message reappears while the first attempt is still running, and two copies run at once.

A standard queue never blocks behind a bad message: the others keep flowing. And the DLQ is configured on the queue, not on the function, because the function never “owns” the message; the queue does.

#SQS FIFO + Lambda: try, retry, and block

The same polling, with ordering added (13.3):

• Messages are delivered in order per message group ID, and only one batch per group is in flight at a time. Lambda scales with the number of active groups, not with the number of messages.
• When a message fails, it comes back as the head of its group. Every later message of that group waits behind it: head-of-line blocking. That is the slide’s word “blocking”. Other groups are not affected.
• With partial batch responses on a FIFO queue, the function must stop at the first failure and report that message and every later one in the batch as failed, or the order breaks.
• After maxReceiveCount the message goes to a DLQ that must itself be FIFO. The group unblocks, but the later messages now run before the failed one. If order matters more than progress (a ledger where payment 3 depends on payment 2), leave the DLQ off, alarm on the stuck group, and fix the cause.

#SNS + Lambda: asynchronous retries

SNS invokes the function asynchronously (15.1):

1. Lambda puts the event in its internal queue and answers 202 at once. For SNS, the delivery has succeeded.
2. If the function throws, Lambda retries it twice by default (after about 1 minute, then about 2 more), configurable to 0, 1 or 2. Throttled events are retried for up to the maximum event age (6 hours by default).
3. When the retries run out, Lambda sends a record to the on-failure destination (SQS, SNS, EventBridge, another Lambda function, or S3), with the event, the error and the reason. The older alternative is a function DLQ (an SQS queue or SNS topic) that receives only the event. With neither, the event is discarded.

SNS has its own safety net for a different failure: a subscription DLQ (an SQS queue on the subscription) catches messages SNS could not deliver, for example because someone removed the function’s resource policy (a client-side error, not retried). A message that was delivered but failed in the code never reaches the SNS DLQ.

There is no ordering in this shape (SNS FIFO topics deliver only to SQS queues, not to Lambda), and asynchronous delivery is at least once, so the function must be idempotent.

#The three shapes side by side

Try the three shapes yourself: pick the source, make the message a poison pill, switch the safety net on and off, and compare the timelines.

#Dead-letter queues across the shapes

The notes define the idea: when SQS, SNS or Lambda keeps failing on a message and the maximum retries are used up, the message is moved to a separate queue so it is not lost and does not block or loop. Developers read it later to debug. Lesson 13.2 covers the mechanics; here is what an architect must get right.

• On SQS a DLQ is a redrive policy on the source queue: deadLetterTargetArn plus maxReceiveCount (1 to 1,000; 3 to 10 is common). The DLQ is an ordinary queue you create first.
• A poison pill is a message that can never succeed (wrong type, deleted customer, a bug that only this input triggers). Without a DLQ it loops until the retention period ends and then disappears. With a DLQ it leaves after maxReceiveCount tries.
• Same type, same place: a FIFO queue needs a FIFO DLQ, a standard queue a standard DLQ, in the same account and Region.
• Retention longer than the source (usually 14 days): for a standard queue the message keeps its original enqueue time, so a DLQ with the same retention can delete it almost at once.
• Watch it: a CloudWatch alarm on the DLQ’s ApproximateNumberOfMessagesVisible greater than 0. A DLQ nobody looks at is a slower way to lose messages.
• Redrive after the fix: “Start DLQ redrive” in the console or StartMessageMoveTask moves messages back to the source queue (or another queue of the same type), optionally at a capped rate.

#Fan-out: deliver one event to many queues

The second slide shows two ways to get one event into three SQS queues.

Option 1: the producer writes to each queue itself. The code calls SendMessage three times, one queue after another. It works on a good day, but:

• There is no transaction across queues. If the process dies after the second write, two consumers have the event and the third never will. Retrying the whole job sends duplicates to the first two.
• Each write adds latency to the producer, and each new consumer means changing and redeploying the producer.

Option 2: fan-out. The producer publishes once to an SNS topic; the queues subscribe to it; SNS pushes a copy into every queue. The producer makes one call and knows nobody. Each consumer has its own queue, so it keeps its own pace, retries and DLQ, and a consumer that is down simply builds a backlog in its queue: the “without data loss” of the exam phrase. SNS retries delivery to SQS for up to 23 days, so the copy is practically guaranteed.

What makes fan-out work in practice (13.4):

• The queue’s access policy must allow the topic to send: principal sns.amazonaws.com, action sqs:SendMessage, condition aws:SourceArn = the topic. The console adds it when you subscribe; the CLI, CloudFormation and Terraform do not, and deliveries fail with access denied. An SSE-KMS queue also needs a customer managed key that allows SNS.
• Filter policies on a subscription send each queue only the messages it needs (by message attributes or by the body), instead of every worker discarding most of what it gets.
• Raw message delivery gives the queue exactly the published body, without the SNS JSON envelope.
• Cross-Region and cross-account subscriptions work (queue policy plus topic policy).
• FIFO end to end: an SNS FIFO topic with SQS FIFO queues keeps order and deduplication through the fan-out.
• S3 events → SNS → many queues is the classic way around S3’s rule that one event goes to one destination (next section).

#S3 event notifications

A bucket can announce what happens to its objects (10.7 has the full detail).

• Event types: s3:ObjectCreated:* (Put, Post, Copy, CompleteMultipartUpload), s3:ObjectRemoved:*, s3:ObjectRestore:*, s3:Replication:*, plus lifecycle, tagging, ACL and Intelligent-Tiering events. Listen to s3:ObjectCreated:* rather than only :Put, or large multipart uploads are missed.
• Filtering: a key prefix and/or suffix only, such as prefix uploads/ and suffix .jpg. No wildcards (the slides’ *.jpg is written as the suffix .jpg), no filter on size or metadata.
• Destinations: an SNS topic, an SQS queue or a Lambda function, in the same Region as the bucket. Only standard queues and topics: FIFO is not supported as a native destination. The fourth option, EventBridge, is switched on for the whole bucket (next section).
• Permissions: S3 delivers as the service s3.amazonaws.com, so the destination’s resource policy must allow it (sns:Publish, sqs:SendMessage, lambda:InvokeFunction), ideally limited to the bucket with aws:SourceArn and aws:SourceAccount. A missing permission makes the configuration fail to save.
• Timing and delivery: usually within seconds, sometimes a minute or longer; at least once, not ordered (compare the sequencer for one key). The key arrives URL-encoded.
• How many rules: you can create as many rules as you need, as long as no two overlap.

The overlap rule, made precise: two rules cannot coexist if their event types overlap and their prefixes overlap and their suffixes overlap. So the notes’ “no overlapping rules for the same prefix and suffix” is true per event type: ObjectCreated to a function and ObjectRemoved to a queue on the same prefix is fine; ObjectCreated on uploads/ to a function and to a queue is rejected. Native notifications give one event one destination. To get it to several, send it to one SNS topic and subscribe them all, or use EventBridge.

The recursive loop: a function triggered by uploads/ that writes its output back into uploads/ triggers itself again and again. Write to a different prefix (thumbs/) or bucket. Lambda now detects such S3 loops and stops them after about 16 invocations, but that is a safety net, not a design.

#S3 events through Amazon EventBridge

One bucket setting, Send notifications to Amazon EventBridge, makes S3 send all its events to the default event bus of the same Region. You choose nothing on the S3 side and S3 needs no permission. EventBridge rules then do the filtering and routing (19.5).

Two corrections to the sources. The slide says “over 18 AWS services as destinations” and the notes “18+”: today the EventBridge target list has more than 20 kinds of targets, and the number keeps growing, so remember “many, including Step Functions and Kinesis”. And both say EventBridge can filter on metadata: it filters on the fields in the event (key, size, ETag, version, requester, source IP, reason…), not on the user-defined x-amz-meta- metadata of the object, which is not in the event.

#Intercepting API calls with EventBridge and CloudTrail

Many changes do not produce a service event of their own: nobody announces that a DynamoDB table was deleted or an instance was launched without tags. But every such change is an API call, and CloudTrail records every API call (19.6). CloudTrail delivers those records to the default event bus as events with detail-type AWS API Call via CloudTrail, so a rule can react to any call.

The slide’s example: a user calls DeleteTable on DynamoDB → CloudTrail logs it → an EventBridge rule matches → SNS alerts the team. The notes’ example: someone calls RunInstances → the rule invokes a Lambda function that checks the tags and stops the instance (or tags it, or opens a ticket) when a required tag is missing.

{
  "source": ["aws.ec2"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["ec2.amazonaws.com"],
    "eventName": ["RunInstances"]
  }
}

What must be true for it to work:

• A trail with logging on must cover the account (an organization trail counts). Management events are logged by default in a trail; data events (S3 object calls, Lambda invokes) only if you add them.
• Rules are regional. The event arrives on the bus of the Region where the call was recorded. A rule in ap-southeast-1 never sees a call made in eu-west-1.
• Global services such as IAM record their calls in us-east-1, so an IAM rule must live there. Root console sign-ins are recorded in us-east-1, us-east-2 or us-west-2.
• Read-only calls (Get, List, Describe) do not reach a rule in the normal enabled state. Set the rule’s state to ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS (CLI or CloudFormation) and expect much higher volume.
• Near real time: seconds to a few minutes. It detects and reacts; to prevent a call, use IAM, SCPs or the resource’s own protection (DynamoDB deletion protection).

#API Gateway straight into Kinesis Data Streams

Behind API Gateway we usually put a Lambda function. But API Gateway can call most AWS APIs directly, with no function in between: an AWS service integration (15.6). The slide’s example is a streaming pipeline:

client → API Gateway → Kinesis Data Streams → Data Firehose → Amazon S3 (.json files)

How it is built:

• A REST API method with integration type AWS, service Kinesis, action PutRecord (or PutRecords for batches).
• A mapping template (VTL) that turns the request into the action’s input: stream name, partition key (for example the rider ID, so one rider’s points stay in order on one shard), and the body Base64-encoded as Data.
• An IAM role that API Gateway assumes, allowing exactly kinesis:PutRecord on that stream.
• Integration responses that map Kinesis errors to proper HTTP status codes. With a non-proxy integration, an error without a matching integration response can come back through the default response as a 200 with the error in the body.
• Behind the stream: Firehose buffers records into files in S3 (Athena queries them later), and real-time consumers read the stream directly.

Why remove the Lambda function: it only copied the request, yet it added latency (run time and the occasional cold start), cost per request, and concurrency consumption from the account’s shared pool. Without it there is no code to write, patch or scale. Keep a function only when it does real work (validation a model cannot express, enrichment, routing).

The same trick works for many services: SQS SendMessage, SNS Publish, Step Functions StartExecution, DynamoDB PutItem, S3 objects, EventBridge PutEvents. HTTP APIs offer some of these as ready-made first-class integrations (SQS, Kinesis Data Streams, Step Functions, EventBridge and more).

The limits that matter on this path:

When the stream is too small, Kinesis throttles the extra writes. Fix it with more shards or on-demand capacity mode, a partition key that spreads the load (a city as key makes one hot shard), client retries with backoff and batching with PutRecords, and API Gateway throttling so a burst is refused at the door.

#Reading the question: which pattern?

#What goes wrong

• A poison message loops for days: no redrive policy. Add a DLQ with a sensible maxReceiveCount and an alarm on it.
• Good messages are processed twice after one failure in the batch: no partial batch responses. Turn on ReportBatchItemFailures and make the consumer idempotent anyway.
• Messages are processed by two invocations at once: the visibility timeout is shorter than the function timeout. Use at least six times the function timeout.
• A FIFO group is stuck: its head message keeps failing. Fix the cause, or accept the broken order and add a FIFO DLQ.
• The DLQ was configured on the function, and SQS messages still loop: for SQS sources the DLQ belongs on the queue.
• Failed SNS-triggered events vanish and the SNS DLQ is empty: the delivery succeeded; the failure was in the code. Configure an on-failure destination on the function.
• One fan-out queue receives nothing: its access policy does not allow the topic, or its KMS key does not allow SNS.
• The S3 notification configuration cannot be saved: overlapping rules for the same event type, or a destination policy that does not allow S3, or a FIFO queue as destination.
• The bill explodes after a deploy: a function writes back into the prefix that triggers it.
• The API-call rule never fires: no trail, wrong Region, a global service in us-east-1, or a read-only call.
• Pings disappear at peak: Kinesis throttles, and the integration maps the error to a 200. Add integration responses, more shards or on-demand mode, and client retries.

#Event processing in system design

Put the pieces together for one wallet:

• The mobile API writes high-volume telemetry (screen events, location) through API Gateway → Kinesis → Firehose → S3, no code.
• A completed payment is published once to an SNS topic; ledger, SMS and fraud each have an SQS queue with its own DLQ; the ledger queue is FIFO grouped by account because order matters there.
• Customers’ uploaded KYC documents land in S3; EventBridge rules send large files to a Step Functions review workflow and every upload to an audit FIFO queue.
• CloudTrail + EventBridge rules alert security on DeleteTable, security-group changes and root sign-ins, in every Region used and in us-east-1.
• Every asynchronous Lambda function has an on-failure destination; every queue has an alarm on its DLQ and on the age of its oldest message.

Lesson 15.8 builds complete serverless applications from the same parts, and lesson 24.2 adds caching at every layer of such a design.

#Try it yourself

These commands use the AWS CLI v2 in ap-southeast-1 with made-up names; queues, topics and streams cost money while they exist, so delete them at the end.

# 1. A DLQ and a source queue with a redrive policy (maxReceiveCount 3), DLQ kept 14 days
aws sqs create-queue --queue-name payments-dlq --attributes MessageRetentionPeriod=1209600
aws sqs create-queue --queue-name payments --attributes VisibilityTimeout=180
aws sqs set-queue-attributes \
  --queue-url https://sqs.ap-southeast-1.amazonaws.com/111122223333/payments \
  --attributes '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"arn:aws:sqs:ap-southeast-1:111122223333:payments-dlq\",\"maxReceiveCount\":\"3\"}"}'

# 2. Let Lambda poll it, with partial batch responses
aws lambda create-event-source-mapping --function-name ledger-writer \
  --event-source-arn arn:aws:sqs:ap-southeast-1:111122223333:payments \
  --batch-size 10 --function-response-types ReportBatchItemFailures
# { "UUID": "a1b2c3d4-5678-90ab-cdef-11111EXAMPLE", "BatchSize": 10, "State": "Creating",
#   "FunctionResponseTypes": [ "ReportBatchItemFailures" ], ... }

# 3. After the fix: move the DLQ's messages back to the source queue
aws sqs start-message-move-task \
  --source-arn arn:aws:sqs:ap-southeast-1:111122223333:payments-dlq \
  --destination-arn arn:aws:sqs:ap-southeast-1:111122223333:payments \
  --max-number-of-messages-per-second 50
# { "TaskHandle": "eyJ0YXNrSWQiOiJkYzE2...EXAMPLE" }

# 4. Asynchronous Lambda: 2 retries, 1 hour maximum age, failures to a queue
aws lambda put-function-event-invoke-config --function-name ledger-writer \
  --maximum-retry-attempts 2 --maximum-event-age-in-seconds 3600 \
  --destination-config '{"OnFailure":{"Destination":"arn:aws:sqs:ap-southeast-1:111122223333:ledger-failures"}}'

# 5. Fan-out: subscribe a queue to the topic with a filter policy and raw delivery
#    (the queue policy must already allow sns.amazonaws.com with aws:SourceArn = the topic)
aws sns subscribe \
  --topic-arn arn:aws:sns:ap-southeast-1:111122223333:payment-done \
  --protocol sqs --notification-endpoint arn:aws:sqs:ap-southeast-1:111122223333:sms \
  --attributes '{"RawMessageDelivery":"true","FilterPolicy":"{\"amount\":[{\"numeric\":[\">\",5000]}]}"}'
# { "SubscriptionArn": "arn:aws:sns:ap-southeast-1:111122223333:payment-done:6f1c2b3a-EXAMPLE" }

# 6. S3: one native rule to Lambda, and EventBridge switched on for the bucket
#    (this call replaces the whole notification configuration of the bucket)
aws s3api put-bucket-notification-configuration --bucket news-media-example \
  --notification-configuration '{
    "LambdaFunctionConfigurations": [{
      "LambdaFunctionArn": "arn:aws:lambda:ap-southeast-1:111122223333:function:resize",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {"Key": {"FilterRules": [{"Name": "prefix", "Value": "uploads/"},
                                         {"Name": "suffix", "Value": ".jpg"}]}}}],
    "EventBridgeConfiguration": {} }'

# 7. A rule that reacts to DeleteTable calls recorded by CloudTrail
aws events put-rule --name alert-delete-table \
  --event-pattern '{"source":["aws.dynamodb"],"detail-type":["AWS API Call via CloudTrail"],"detail":{"eventSource":["dynamodb.amazonaws.com"],"eventName":["DeleteTable"]}}'
# { "RuleArn": "arn:aws:events:ap-southeast-1:111122223333:rule/alert-delete-table" }
aws events put-targets --rule alert-delete-table \
  --targets Id=sns,Arn=arn:aws:sns:ap-southeast-1:111122223333:security-alerts

# 8. API Gateway → Kinesis: an AWS service integration calling PutRecord
#    kinesis-template.json holds the mapping template:
#    {"application/json": "{\"StreamName\":\"rider-pings\",\"PartitionKey\":\"$input.path('$.riderId')\",\"Data\":\"$util.base64Encode($input.body)\"}"}
aws apigateway put-integration --rest-api-id a1b2c3d4e5 --resource-id r2s3t4 \
  --http-method POST --type AWS --integration-http-method POST \
  --uri arn:aws:apigateway:ap-southeast-1:kinesis:action/PutRecord \
  --credentials arn:aws:iam::111122223333:role/apigw-kinesis-putrecord \
  --request-templates file://kinesis-template.json
# { "type": "AWS", "httpMethod": "POST",
#   "uri": "arn:aws:apigateway:ap-southeast-1:kinesis:action/PutRecord", ... }
# Map Kinesis 4xx errors (such as throttling) to a 500 the client can see
# (a 500 method response must exist on the method first)
aws apigateway put-integration-response --rest-api-id a1b2c3d4e5 --resource-id r2s3t4 \
  --http-method POST --status-code 500 --selection-pattern '4\d{2}'

# 9. Clean up
aws events remove-targets --rule alert-delete-table --ids sns
aws events delete-rule --name alert-delete-table
aws lambda delete-event-source-mapping --uuid a1b2c3d4-5678-90ab-cdef-11111EXAMPLE
aws sqs delete-queue --queue-url https://sqs.ap-southeast-1.amazonaws.com/111122223333/payments
aws sqs delete-queue --queue-url https://sqs.ap-southeast-1.amazonaws.com/111122223333/payments-dlq

Also try in the console: in SQS, send a message that your function rejects and watch Receive count in Poll for messages climb until it appears in the DLQ; then press Start DLQ redrive. In EventBridge → Rules, create a rule from the AWS API Call via CloudTrail template for EC2 and use the sandbox to test the pattern.

#Summary

Key takeaways

• SQS = pull, buffer, decouple; SNS = push to many; Lambda = code on an event. SQS does not push to Lambda: an event source mapping polls.
• SQS + Lambda: retries by visibility timeout until maxReceiveCount, then the queue’s DLQ; use partial batch responses and a visibility timeout of at least 6× the function timeout. Others are not blocked.
• SQS FIFO + Lambda: order per message group; a failing message blocks its group until it succeeds or moves to a FIFO DLQ (which breaks the group’s order).
• SNS + Lambda: asynchronous; Lambda retries twice by default, then the on-failure destination (preferred, with error details) or function DLQ, else discarded. The SNS subscription DLQ catches only delivery failures.
• DLQ = redrive policy (maxReceiveCount) on the source queue; same type (FIFO→FIFO), same account and Region, longer retention, an alarm, redrive after the fix. Keywords: poison pill, infinite loop, troubleshoot failed messages.
• Fan-out: publish once to SNS, many SQS queues subscribe; queue policy must allow the topic; filter policies; beats writing to each queue from code (no partial failures, no producer changes).
• S3 events: ObjectCreated / Removed / Restore / Replication…, prefix and suffix only, destinations SNS, SQS, Lambda (standard only), overlap rule per event type, seconds to a minute, destination policy must allow S3, avoid recursive loops. S3 → EventBridge: all events, rich filters (size, key…), 20+ target types, many rules per event, archive and replay.
• Intercept API calls: CloudTrail → EventBridge rule (AWS API Call via CloudTrail) → SNS or Lambda; needs a trail; regional rules; IAM events in us-east-1; read-only calls need the rule to opt in.
• API Gateway → Kinesis: AWS service integration with an IAM role and a mapping template; no Lambda = less latency, cost and concurrency; watch shard throughput and map errors.

Self-check: Event processing patterns30 questions