Course outline

10.7

Events, Requester Pays, Batch Operations and Storage Lens

Event notifications and EventBridge, who pays for downloads, bulk jobs over millions of objects, and organisation-wide storage insight.

22 min Reading + animations

Tip: tap or hover any dotted-underlined term to see what it means.

By the end of this lesson you will be able to

  • Configure S3 event notifications: choose event types, filter by prefix and suffix, pick SNS, SQS or Lambda as the destination, and read the event message a target receives.
  • Explain why a notification needs a resource-based policy on its destination (and a key policy for an encrypted queue or topic), and fix the “Unable to validate the following destination configurations” error.
  • Apply the overlap rule, avoid recursive loops, and decide when to send S3 events to Amazon EventBridge instead.
  • Explain Requester Pays: who pays for storage, requests and transfer, what the requester must send, and when the owner still pays.
  • Plan an S3 Batch Operations job: build the object list with S3 Inventory (and Athena) or generated filters, choose the operation, give it an IAM role, read the completion report, and predict when a job fails.
  • Choose between S3 Storage Lens free and advanced metrics, and know where Amazon Athena fits next to all of this.

#A bucket that reacts and a bucket you operate

The first six lessons of this chapter treated S3 as a place to put objects: buckets and keys (lesson 10.1), who can read them (10.2), websites (10.3), versions and copies (10.4), storage classes (10.5) and speed (10.6). This lesson is about two things that start once a bucket is busy:

  • Reacting to change. A new photo should get a thumbnail, a new invoice should be booked, a deleted file should raise an alert. Nobody should poll the bucket every minute to find out what changed: S3 can tell you, with event notifications or through Amazon EventBridge.
  • Operating at scale. A bucket with 2 billion objects raises questions a script cannot answer well: who pays when outsiders download a public dataset (Requester Pays), how to change tags or encryption on every object at once (S3 Batch Operations, fed by S3 Inventory), and where the storage and the requests actually go (S3 Storage Lens, with Athena for questions SQL can answer).

#Event notifications

#What S3 can tell you about

An event notification is a message S3 sends when something happens to an object in a bucket. You store the configuration on the bucket (its notification subresource), and S3 publishes a message for every matching event. The event types you will meet most:

Event typeFires when
s3:ObjectCreated:* (or :Put, :Post, :Copy, :CompleteMultipartUpload)An object is created, by any API or by the one you name
s3:ObjectRemoved:* (:Delete, :DeleteMarkerCreated)An object is deleted, or a delete marker is added in a versioned bucket (lesson 10.4)
s3:ObjectRestore:* (:Post, :Completed, :Delete)A restore from Glacier Flexible Retrieval or Deep Archive starts, finishes, or the temporary copy expires (lesson 10.5)
s3:Replication:*Replication with metrics or RTC fails, misses the 15-minute threshold, or catches up late (lesson 10.4)
s3:LifecycleExpiration:*, s3:LifecycleTransitionA lifecycle rule deletes or moves an object
s3:IntelligentTieringIntelligent-Tiering moves an object to an archive tier
s3:ObjectTagging:*, s3:ObjectAcl:Put, s3:ObjectRetention:PutTags are added or removed, an ACL changes, Object Lock retention changes
s3:ReducedRedundancyLostObjectAn object in the old Reduced Redundancy class is lost

Two details catch people out:

  • s3:ObjectRemoved:* does not fire when a lifecycle rule deletes an object. For those, subscribe to s3:LifecycleExpiration:*.
  • A large upload done with multipart (lesson 10.1) is s3:ObjectCreated:CompleteMultipartUpload, not :Put. A rule that listens only for :Put misses it; s3:ObjectCreated:* catches every way an object is created.

#Filtering by prefix and suffix

Each rule can narrow the events by the object key:

  • a prefix, such as uploads/ (the key must start with it), and/or
  • a suffix, such as .jpg (the key must end with it).

When both are set, both must match. That is all the filtering native notifications can do: no wildcards (*.jpg is not valid; write the suffix .jpg), no filter on object size, tags or metadata, and the match is case-sensitive, so Uploads/ is not uploads/. Spaces in a filter are written as +, and other special characters must be URL-encoded.

#Where events can go

A rule sends its events to exactly one destination, and there are three kinds, plus EventBridge for the whole bucket:

  • an Amazon SNS topic (standard only, not FIFO): a push to many subscribers at once, such as email, mobile push, or several SQS queues (fan-out, covered in depth in lesson 13.4);
  • an Amazon SQS queue (standard only, not FIFO): a buffer that a worker drains at its own pace (lesson 13.1);
  • an AWS Lambda function: code that runs once per event (lesson 15.1);
  • Amazon EventBridge: switched on for the whole bucket, with its own section below.

The SNS topic, SQS queue and Lambda function must be in the same Region as the bucket. An SQS queue should allow messages of at least 64 KB, the largest an S3 event message can be.

#The event message

The target receives a JSON document with a Records array. Each record carries the event name, the time, the bucket, and the object’s key, size, ETag, version ID (in a versioned bucket) and a sequencer:

{
  "Records": [
    {
      "eventSource": "aws:s3",
      "awsRegion": "ap-south-1",
      "eventTime": "2026-10-25T09:14:03.512Z",
      "eventName": "ObjectCreated:Put",
      "s3": {
        "bucket": { "name": "shop-media-example", "arn": "arn:aws:s3:::shop-media-example" },
        "object": {
          "key": "uploads/summer+sale+shirt.jpg",
          "size": 3145728,
          "eTag": "d41d8cd98f00b204e9800998ecf8427e",
          "sequencer": "0066A1B2C3D4E5F6A7"
        }
      }
    }
  ]
}

Three properties matter when you write the code that receives it:

  1. Timing. Notifications usually arrive within seconds, but sometimes take a minute or longer. They are not a real-time guarantee.
  2. At least once, not in order. S3 is designed to deliver every notification at least once. On rare occasions a retry delivers the same event twice, and two events for different writes can arrive out of order. Make the handler idempotent (doing the same work twice is harmless), and when order matters for one key, compare the sequencer values: for the same key, a larger sequencer means a later event.
  3. The key is URL-encoded. The object uploads/summer sale shirt.jpg arrives as uploads/summer+sale+shirt.jpg. Code that passes the key straight to GetObject gets “NoSuchKey”; decode it first.

Watch a full flow, including a filter that does not match, a destination that refuses S3, and a loop:

One upload, one event, one thumbnail

A Dhaka clothing shop uploads product photos to uploads/ in its bucket. A notification rule (prefix uploads/, suffix .jpg) calls a Lambda function that writes a small copy to thumbs/.

Scenario

The upload matches the rule, Lambda is invoked within seconds, and its own output does not trigger it again.

Shop admin
Bucketshop-media-example
Lambdamake-thumb
PUT uploads/shirt-42.jpg
Step 1 of 5
1

The admin uploads uploads/shirt-42.jpg with a normal PUT. S3 stores the object first; the event is a side effect of a successful write, never a condition for it.

#The overlap rule

S3 refuses a configuration in which two rules could both claim the same event. Two rules overlap, and the configuration cannot be saved, when all three are true:

  1. their event types overlap (s3:ObjectCreated:* overlaps s3:ObjectCreated:Put, but not s3:ObjectRemoved:*);
  2. their prefixes overlap: one starts with the other. An empty prefix overlaps every prefix;
  3. their suffixes overlap: one ends with the other (jpg and .jpg overlap, pg and jpg overlap, .jpg and .png do not). An empty suffix overlaps every suffix.

So the owner’s notes are right in practice: you cannot have two rules for the same prefix and the same event type unless their suffixes are different. Prefix images/ + suffix .jpg to one function and prefix images/ + suffix .png to another is fine; prefix images/ to a queue and prefix images/ to a function, for the same event, is rejected. To send the same events to several places, either:

  • send them to one SNS topic and subscribe several SQS queues or functions to it (the classic fan-out), or
  • turn on EventBridge and write as many rules as you like.

Try it: switch rules on and off, save, then upload test files.

Event rule lab

Switch rules on or off, save the configuration the way S3 would check it, then upload test files and see which rules fire.

Notification rules on shop-media-example

No saved configuration yet. Press “Save configuration”, then “Upload test files”.

Self-checkQuestion

A bucket already has a rule that sends every object-created event under logs/ to a Lambda function. A second team adds a rule that sends every object-created event under logs/ to their SQS queue. What happens?

#Letting S3 deliver: resource policies

#Why the destination must say yes

S3 delivers a notification by calling the destination as the service principal s3.amazonaws.com. Your IAM user’s permissions do not travel with the event, so S3 can only deliver if the destination itself allows it, through its resource-based policy (the policy attached to the resource, lesson 2.2):

DestinationPolicyAction S3 needs
SNS topicTopic access policysns:Publish
SQS queueQueue access policysqs:SendMessage
Lambda functionFunction resource policy (aws lambda add-permission)lambda:InvokeFunction
EventBridgeNone: S3 needs no extra permission to deliver to EventBridge–

The owner’s notes call this “IAM permission”. It is IAM in the broad sense, but the exam and the console mean a resource (access) policy on the SNS topic, SQS queue or Lambda function, not a policy on a user or role. The Lambda one is easy to forget: the console adds it for you when you pick a function in the S3 console, but a template or the CLI must add it explicitly.

A queue policy that lets only one bucket in one account send to it:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowS3ShopMedia",
      "Effect": "Allow",
      "Principal": { "Service": "s3.amazonaws.com" },
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:ap-south-1:111122223333:audit",
      "Condition": {
        "ArnLike": { "aws:SourceArn": "arn:aws:s3:*:*:shop-media-example" },
        "StringEquals": { "aws:SourceAccount": "111122223333" }
      }
    }
  ]
}

The two conditions stop the confused deputy problem: without them, any bucket in any account could point its notifications at your queue, and S3 would deliver because the policy trusts “the S3 service” in general. aws:SourceArn names the bucket; aws:SourceAccount pins the bucket owner, because a bucket ARN has no account ID in it.

#Encrypted queues and topics

If the SQS queue or SNS topic uses server-side encryption with KMS (lesson 10.8 covers KMS), S3 must also be able to use the key: the key policy must allow s3.amazonaws.com to call kms:GenerateDataKey and kms:Decrypt. You cannot edit the key policy of the AWS managed keys (aws/sqs, aws/sns), so an encrypted destination for S3 needs a customer managed key.

#What a missing permission looks like

When you save a configuration, S3 checks every destination and sends it an s3:TestEvent. If the destination’s policy does not allow S3, the save fails with “Unable to validate the following destination configurations” and the bucket keeps its old configuration. If someone removes the permission later, nothing fails loudly: events simply stop arriving. When notifications go missing, check the destination’s policy first, then the KMS key policy, then the filters.

#Sending S3 events to Amazon EventBridge

#What changes

Amazon EventBridge is a serverless event bus (it has its own lesson, 19.5). One setting on the bucket, “Send notifications to Amazon EventBridge for all events in this bucket”, makes S3 send every supported event to the default event bus in the same Region. You do not pick event types and S3 needs no permission. EventBridge rules then match events and route them to targets.

Native notificationsThrough EventBridge
DestinationsSNS (standard), SQS (standard), Lambda20+ kinds of AWS targets, such as Step Functions, Kinesis Data Streams, Data Firehose, SQS FIFO, ECS tasks, API destinations; up to 5 targets per rule
FilteringKey prefix and suffix onlyJSON patterns on any field: key prefix, suffix or wildcard, object size, bucket, the API that created the object, requester, and more
Same event, many consumersNot on overlapping rules; use SNS fan-outAny number of rules may match the same event
Archive and replayNoYes: archive events, replay a time window later
Event namess3:ObjectCreated:Put style“Object Created”, “Object Deleted”, “Object Restore Completed”, “Object Storage Class Changed”, “Object Tags Added”…
CostNo charge from S3 for notificationsEventBridge pricing for events and replays

A rule that starts a transcoding workflow only for big videos:

{
  "source": ["aws.s3"],
  "detail-type": ["Object Created"],
  "detail": {
    "bucket": { "name": ["videos-example"] },
    "object": {
      "key": [{ "suffix": ".mp4" }],
      "size": [{ "numeric": [">", 104857600] }]
    }
  }
}

S3 → EventBridge → many rules

A video platform turns on EventBridge for its upload bucket. Every event goes to the default event bus, and rules decide who gets what.

Scenario

One 2 GB .mp4 upload matches three rules with three different targets.

Bucketvideos-example
EventBridgedefault bus
Archive
Step Functionstranscode
Lambdacatalog
SQS FIFObilling
Step 1 of 5
1

Turning EventBridge on is a single bucket setting. You do not choose event types, and S3 needs no extra permission: every supported event of the bucket goes to the default bus in the same Region.

#Requester Pays

#Who pays for what

Normally the bucket owner pays for everything: storage, every request, and every gigabyte downloaded. That is fine for your own application, and a problem when you publish a 50 TB dataset that hundreds of other organisations download: their curiosity becomes your bill. A Requester Pays bucket moves part of that bill:

CostNormal bucketRequester Pays bucket
StorageOwnerOwner (always)
Requests (GET, PUT, list…)OwnerRequester
Data transfer out of S3OwnerRequester
Retrieval from an archive class on restoreOwnerOwner (the requester pays only the restore request)

You turn it on per bucket (aws s3api put-bucket-request-payment … Payer=Requester). It is how large public datasets are often shared, for example many datasets in the Registry of Open Data on AWS.

#The rules that make it work

  • Requests must be authenticated. S3 has to know which AWS account to bill, so a Requester Pays bucket refuses anonymous requests. It cannot serve an anonymous public website.
  • The requester must accept the charge. Every request carries the header x-amz-request-payer: requester (in the CLI: --request-payer requester). Without it, S3 answers 403 Access Denied, even if the bucket policy allows the read. The bucket owner’s own requests do not need the header.
  • Requester Pays does not grant access. The requester still needs permission: a bucket policy that allows the other account (lesson 10.2), or a public dataset policy that allows authenticated principals.
  • Roles count for the account that owns them. If the requester assumes a role in the owner’s account, the request comes from the owner’s account and the owner pays. Requester Pays only shifts the bill when requesters use credentials from their own accounts.
  • Some failed requests are still the owner’s. The owner pays for a request that fails with 403 when it comes from inside the owner’s own account or AWS Organization.
  • A Requester Pays bucket cannot be the target bucket of server access logging (lesson 10.9 covers access logs).

Play with the setting, the downloader and the destination:

Requester Pays: who gets the bill?

A research institute in Dhaka shares 10 TB of satellite images. Choose the setting and the downloader, then press Download.

Requester Pays on the bucket

Who downloads

Data goes to

Press Download to see the result and the bill.

Calculating it. 20 TB (20,480 GB) in S3 Standard at $0.023 per GB-month costs the owner $471.04 a month, with or without Requester Pays. A partner downloads 5 TB (5,120 GB) to the internet at $0.09 per GB: $460.80 of transfer, plus about 52,000 GET requests for 100 MB objects at $0.0004 per 1,000 (2 cents). Without Requester Pays the owner pays $931.86; with it, the owner pays $471.04 and the partner $460.82. (Example list prices; real bills use tiers and a monthly free allowance.)

#S3 Batch Operations

#Why not a script

“Add the tag retention=7y to every object”, “re-encrypt every old object with our own KMS key”, “copy 400 million objects to the new account’s bucket”, “restore every scan from 2020 out of Deep Archive”. A script can do that for a thousand objects. For a billion it has to list the bucket for hours, run on a machine for days, handle throttling and retries, survive a reboot, and prove at the end which objects failed.

S3 Batch Operations is S3 doing that work for you: a job runs one operation on every object in a list, with retries, progress tracking, notifications and a completion report, as a fully managed, serverless service. One job can cover up to 20 billion objects (and since December 2025 large jobs run up to 10 times faster).

#What a job is made of

  1. A manifest: the list of objects. Either an S3 Inventory report (its manifest.json), a CSV file of bucket,key (with an optional version ID column), or no file at all: Batch Operations can generate the list for you from filters such as prefix, suffix, substring, creation date range, size, storage class or encryption type, and save it as a manifest. A replication configuration can also generate the list (that is Batch Replication, lesson 10.4).
  2. One operation, with its parameters (see the table below). One job, one operation; for two operations, run two jobs.
  3. An IAM role that Batch Operations assumes. Its trust policy allows the service principal batchoperations.s3.amazonaws.com; its permissions cover reading the manifest, writing the report, and the operation itself (for example s3:GetObject on the source and s3:PutObject on the destination for a copy, plus KMS permissions for encrypted data). The console can create it for you. Roles and trust policies were lesson 2.4.
  4. A completion report (optional but recommended): a CSV in a bucket you choose, listing all tasks or only the failed ones with their error codes.
  5. A priority among your jobs, and optional tags and description.

#What a job can do

OperationTypical use
CopyCopy objects to another bucket or Region, change storage class, metadata or encryption; copy in place to encrypt old unencrypted objects. Objects up to 5 GB; all sources in one bucket, all destinations in one bucket
Invoke LambdaAny custom per-object action: resize, scan, transcode, call an API, or copy objects larger than 5 GB with your own multipart code
Replace all object tags / Delete all object tagsBulk tagging for cost allocation, lifecycle or access control
Replace ACLBulk ACL changes on buckets that still use ACLs
RestoreStart restores from Glacier Flexible Retrieval or Deep Archive for millions of objects
Object Lock retention / legal holdSet retention dates or legal holds in bulk (Object Lock is lesson 10.9)
Update object encryptionChange SSE-S3 objects to SSE-KMS, add S3 Bucket Keys, or move to another customer managed key, without copying (not for unencrypted or SSE-C objects)
Compute checksumsCalculate checksums of existing objects to verify integrity
Replicate (Batch Replication)Replicate existing objects that live replication skipped (lesson 10.4)

There is no “delete objects” operation. To remove millions of objects, use a lifecycle expiration rule (lesson 10.5), or Invoke Lambda if the choice of objects is complex.

Some rules of the job itself: a copy job runs in the destination Region, every other job in the Region of the objects; objects are processed in no particular order; a job created in the console waits for your confirmation (status Suspended) and fails if left there 30 days; and once at least 1,000 tasks have run, a failure rate above 50% makes S3 stop the job as Failed, so a bad list cannot run for hours. The manifest is a list, not a snapshot: if an object is overwritten while the job runs and the manifest has no version IDs, the job acts on the newest version.

What it costs: $0.25 per job plus $1.00 per million objects processed, on top of the normal request charges of the operation (and Lambda charges for Invoke Lambda).

#S3 Inventory: the list for the job

S3 Inventory produces a scheduled list of the objects in a bucket (or a prefix) with the metadata you choose: size, last modified date, storage class, ETag, encryption status (SSE-S3, SSE-KMS, DSSE-KMS, SSE-C, NOT-SSE), replication status, Object Lock settings, Intelligent-Tiering tier, version IDs, and more.

  • Delivered daily or weekly, as CSV, ORC or Parquet, to a destination bucket in the same Region (which can belong to another account and needs a bucket policy that lets S3 write to it). The first report can take up to 48 hours.
  • It does not use List requests and does not touch your bucket’s request rate, which makes it far cheaper than listing a billion keys.
  • Each list is eventually consistent: very recent writes and deletes may be missing.
  • You can query it with Amazon Athena, which is how you turn “every object” into “every object that is still NOT-SSE under scans/2019/”, and save the result as the job’s CSV manifest.

The slides draw this pipeline: S3 Inventory → Athena (filter) → filtered list → S3 Batch Operations (operation + parameters) → processed objects.

A Batch Operations job, start to finish

A hospital must re-encrypt 3 million old scan images with its own KMS key. Follow the list, the job, the role and the report.

Scenario

Inventory → Athena filter → manifest → job → confirm → run → report.

Engineer
Reports bucketinventory + report
Athena
Batch Operations
Job role
Scans bucket
daily inventory (CSV)
Step 1 of 9
1

S3 Inventory writes a list of every object in the scans bucket, with the fields you chose (size, storage class, encryption status), plus a manifest.json that points at the files. It reads the bucket in the background, so it does not use your request rate.

Batch Operations is also one of the thirteen ways data moves into or out of S3; lesson 10.12 puts it next to multipart upload and pre-signed URLs in that chooser.

#S3 Storage Lens

#One view of all your storage

A company with 40 accounts and 3,000 buckets cannot answer simple questions by clicking through buckets: Which buckets grow fastest? Where are terabytes of incomplete multipart uploads hiding? Which buckets have no versioning? S3 Storage Lens collects usage and activity metrics daily and shows them in dashboards, aggregated by organisation, account, Region, storage class, bucket and, with advanced metrics, prefix (and custom Storage Lens groups defined by prefix, suffix, tags, size or age).

  • Every account has a default dashboard, default-account-dashboard, configured by S3. You can upgrade its metrics or disable it, but not delete it.
  • You can create up to 50 more dashboards per home Region, scoped to Regions, buckets or accounts. Organisation-wide dashboards need trusted access in AWS Organizations and are created from the management account or a delegated administrator.
  • A metrics export sends the metrics daily as CSV or Parquet to a bucket (or into an AWS managed S3 table bucket, ready for SQL queries).
  • Recommendations and outlier call-outs compare today with your last 30 days and flag unusual changes.

#Free metrics and advanced metrics

Free metricsAdvanced metrics and recommendations
PriceFree, on by defaultPaid: about $0.20 per million objects monitored per month (cheaper above 25 billion)
Metric categoriesSummary (StorageBytes, ObjectCount), cost optimisation (NonCurrentVersionStorageBytes, IncompleteMultipartUploadStorageBytes), data protection (buckets with versioning, MFA Delete or default encryption), access management (Object Ownership), event (buckets with notifications), performance (buckets with Transfer Acceleration)Everything free, plus activity (AllRequests, GetRequests, PutRequests, BytesDownloaded…), detailed status codes (200OKStatusCount, 403ForbiddenErrorCount, 503s…), advanced cost optimisation (lifecycle rule counts), advanced data protection (replication rule counts, SSE-KMS usage, requests with outdated TLS or signatures), advanced performance metrics
Lowest levelBucketPrefix (prefix aggregation, expanded prefixes report), Storage Lens groups
History you can query14 days15 months
CloudWatchNoPublish to CloudWatch for alarms and dashboards
RecommendationsBasicContextual recommendations next to the metrics

The owner’s notes say free data is kept “14 days (now 28 days in many Regions)”. It is 14 days: the “28” is the number of usage metrics the free tier had when AWS last counted them (2023), and the free categories have grown since. The slides’ “30 days usage and activity metrics” is also misleading: the 30 days are the window for outlier detection; activity metrics are advanced only.

What Storage Lens can see

Storage Lens rolls up metrics from the whole organisation down to one bucket, and with advanced metrics down to one prefix.

Scenario

Organizationevery member account, through trusted access
Account 111122223333the default dashboard covers one account
Region ap-south-1and storage class within it
Bucket shop-media-examplelowest level of the free tier
Step 1 of 3
1

Every account has a default dashboard, default-account-dashboard, already on. You cannot delete it, only disable it.

#What goes wrong

  • Notification configuration will not save (“Unable to validate…”): the SNS topic, SQS queue or Lambda function has no resource policy for s3.amazonaws.com, the destination is in another Region, the queue or topic is FIFO, or its KMS key does not allow S3.
  • Events stopped arriving: someone tightened the queue or topic policy or the key policy after the configuration was saved, or a deployment replaced the Lambda function without its permission.
  • Some files never trigger: the suffix or prefix does not match (case, .jpeg vs .jpg), or the rule listens to :Put and the file arrived by multipart upload or copy.
  • A rule cannot be added: it overlaps an existing rule for the same event type. Use SNS fan-out or EventBridge.
  • Lambda fails with NoSuchKey on some files: the key in the event is URL-encoded; decode it.
  • Duplicate processing: delivery is at least once. Make the handler idempotent.
  • Cost explodes and the function runs constantly: a recursive loop; write to another prefix or bucket.
  • Partners get 403 from a Requester Pays bucket: they did not send x-amz-request-payer, they are anonymous, or the bucket policy does not allow them.
  • The owner still pays for partner downloads: partners assume a role in the owner’s account.
  • Batch job stuck in Suspended: it was created in the console and waits for confirmation.
  • Batch job Failed: the role cannot read the manifest or write the report, or more than half the first 1,000+ tasks failed (deleted objects, missing KMS permission). Read the completion report.
  • Storage Lens shows no prefixes, no request counts: the dashboard uses free metrics.

#Exam patterns at a glance

#Events and operations in system design

  • Use events instead of polling. “Something landed in S3, now do X” is the most common serverless pattern on AWS: an upload bucket, a notification, a function or a queue. It scales from ten files a day to millions without changing the design.
  • Put a queue in front of slow work. Lambda directly from S3 suits quick, independent jobs. For slow or bursty work, or when the consumer may be down, send events to SQS and let workers drain it, with a dead-letter queue for poison files (the pattern of lesson 9.4’s worker tier).
  • Plan for fan-out from day one. If more than one team will care about new objects, start with EventBridge (or one SNS topic) instead of a direct notification to one function, so the second consumer does not need to change the first one’s rule.
  • Design handlers for at-least-once, out-of-order delivery. Idempotent writes, the sequencer for ordering per key, and a periodic reconciliation (an inventory compared with your database) for anything that must never be missed.
  • Keep input and output apart. Separate prefixes or buckets for raw uploads and processed results, so rules never see their own output.
  • Charge the right party. Public or partner datasets go in Requester Pays buckets; internal consumers use roles in their own accounts so cost allocation stays clean.
  • Treat bulk changes as jobs. Inventory + Athena to choose the objects, Batch Operations to change them, the completion report as the audit record, Storage Lens to confirm the result across the organisation.

#Try it yourself

Event notifications to Lambda and SQS:

# 1. Let S3 invoke the function (resource-based policy on the function)
aws lambda add-permission --function-name make-thumb \
  --statement-id s3-shop-media --action lambda:InvokeFunction \
  --principal s3.amazonaws.com \
  --source-arn arn:aws:s3:::shop-media-example --source-account 111122223333

# 2. Let S3 send to the queue (queue policy from the lesson saved in queue-policy.json)
aws sqs set-queue-attributes \
  --queue-url https://sqs.ap-south-1.amazonaws.com/111122223333/audit \
  --attributes file://queue-attributes.json   # {"Policy": "<the policy JSON as a string>"}

# 3. The notification configuration: .jpg and .png to Lambda, everything under invoices/ to SQS
cat > notification.json <<'EOF'
{
  "LambdaFunctionConfigurations": [
    { "Id": "thumbs-jpg",
      "LambdaFunctionArn": "arn:aws:lambda:ap-south-1:111122223333:function:make-thumb",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": { "Key": { "FilterRules": [
        { "Name": "prefix", "Value": "uploads/" }, { "Name": "suffix", "Value": ".jpg" } ] } } },
    { "Id": "thumbs-png",
      "LambdaFunctionArn": "arn:aws:lambda:ap-south-1:111122223333:function:make-thumb",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": { "Key": { "FilterRules": [
        { "Name": "prefix", "Value": "uploads/" }, { "Name": "suffix", "Value": ".png" } ] } } }
  ],
  "QueueConfigurations": [
    { "Id": "invoices",
      "QueueArn": "arn:aws:sqs:ap-south-1:111122223333:audit",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": { "Key": { "FilterRules": [ { "Name": "prefix", "Value": "invoices/" } ] } } }
  ],
  "EventBridgeConfiguration": {}
}
EOF
aws s3api put-bucket-notification-configuration --bucket shop-media-example \
  --notification-configuration file://notification.json
aws s3api get-bucket-notification-configuration --bucket shop-media-example

# 4. Test: upload, then watch the queue and the function's logs
aws s3 cp shirt-42.jpg s3://shop-media-example/uploads/shirt-42.jpg
aws sqs receive-message --queue-url https://sqs.ap-south-1.amazonaws.com/111122223333/audit
aws logs tail /aws/lambda/make-thumb --since 5m

The "EventBridgeConfiguration": {} line is the whole switch for EventBridge. A rule on the default bus:

aws events put-rule --name big-videos \
  --event-pattern file://big-videos-pattern.json      # the JSON pattern from the lesson
aws events put-targets --rule big-videos \
  --targets Id=transcode,Arn=arn:aws:states:ap-south-1:111122223333:stateMachine:transcode,RoleArn=arn:aws:iam::111122223333:role/events-to-sfn

Requester Pays:

# Owner: turn it on and check
aws s3api put-bucket-request-payment --bucket bay-images-example \
  --request-payment-configuration Payer=Requester
aws s3api get-bucket-request-payment --bucket bay-images-example

# Partner (its own account's credentials): accept the charges on every request
aws s3 ls s3://bay-images-example/2026/ --request-payer requester
aws s3 cp s3://bay-images-example/2026/10/scene-0042.tif . --request-payer requester
aws s3 cp s3://bay-images-example/2026/10/scene-0042.tif .      # → 403 Access Denied

S3 Inventory and a Batch Operations job:

# Daily CSV inventory of scans-example, with encryption status, into reports-example
aws s3api put-bucket-inventory-configuration --bucket scans-example --id daily-all \
  --inventory-configuration '{"Id":"daily-all","IsEnabled":true,"IncludedObjectVersions":"Current",
    "Schedule":{"Frequency":"Daily"},
    "OptionalFields":["Size","StorageClass","EncryptionStatus"],
    "Destination":{"S3BucketDestination":{"Bucket":"arn:aws:s3:::reports-example",
      "Format":"CSV","Prefix":"inventory"}}}'

# A tagging job over a CSV manifest, no console confirmation needed
aws s3control create-job --account-id 111122223333 --region ap-south-1 \
  --operation '{"S3PutObjectTagging":{"TagSet":[{"Key":"retention","Value":"7y"}]}}' \
  --manifest '{"Spec":{"Format":"S3BatchOperations_CSV_20180820","Fields":["Bucket","Key"]},
    "Location":{"ObjectArn":"arn:aws:s3:::reports-example/manifests/scans-2019.csv","ETag":"<etag of the csv>"}}' \
  --report '{"Bucket":"arn:aws:s3:::reports-example","Prefix":"batch-reports",
    "Format":"Report_CSV_20180820","Enabled":true,"ReportScope":"FailedTasksOnly"}' \
  --priority 10 --role-arn arn:aws:iam::111122223333:role/s3-batch-tagging \
  --no-confirmation-required

aws s3control list-jobs --account-id 111122223333 --region ap-south-1
aws s3control describe-job --account-id 111122223333 --region ap-south-1 --job-id <job-id>
aws s3control update-job-status --account-id 111122223333 --region ap-south-1 \
  --job-id <job-id> --requested-job-status Ready   # confirm a Suspended job

Storage Lens:

aws s3control list-storage-lens-configurations --account-id 111122223333 --region us-east-1
aws s3control get-storage-lens-configuration --account-id 111122223333 \
  --config-id default-account-dashboard --region us-east-1

#Summary

Key takeaways

  • Event notifications publish object events (created, removed, restore, replication, lifecycle, tagging…) to SNS (standard), SQS (standard) or Lambda in the same Region, one destination per rule, filtered only by key prefix and suffix (no wildcards, case-sensitive).
  • Delivery is usually seconds, sometimes a minute or more, at least once and not ordered; keys arrive URL-encoded. Make handlers idempotent and keep output out of the watched prefix to avoid recursive loops.
  • The destination must allow s3.amazonaws.com in its resource-based policy (sns:Publish, sqs:SendMessage, lambda:InvokeFunction), ideally limited with aws:SourceArn and aws:SourceAccount; encrypted queues and topics need a customer managed KMS key whose policy allows S3. Missing permission → “Unable to validate the following destination configurations”.
  • Rules for the same event type cannot have overlapping prefixes and suffixes. For many consumers use SNS fan-out or EventBridge.
  • EventBridge: one bucket switch, every event, no permission needed; JSON rules on size, metadata and more; 20+ target types including Step Functions and SQS FIFO; archive and replay.
  • Requester Pays: owner pays storage, requester pays requests and transfer; requesters must be authenticated and send x-amz-request-payer: requester; a role in the owner’s account means the owner pays.
  • S3 Batch Operations: one operation on billions of objects from a manifest (S3 Inventory, CSV, or generated by filters) with an IAM role, retries, a completion report, console confirmation, and a 50% failure threshold after 1,000 tasks. Copy (≤ 5 GB, encrypt in place), Lambda, tags, ACL, restore, Object Lock, update encryption, checksums, Batch Replication; no delete.
  • S3 Inventory: daily or weekly CSV/ORC/Parquet lists with encryption and replication status; query with Athena (serverless SQL on S3, ~$5 per TB scanned; Parquet, compression and partitions cut cost).
  • S3 Storage Lens: organisation-wide dashboards. Free: bucket level, 14 days. Advanced: prefix level, activity and status-code metrics, 15 months, CloudWatch, recommendations.