10.7
Events, Requester Pays, Batch Operations and Storage Lens
Event notifications and EventBridge, who pays for downloads, bulk jobs over millions of objects, and organisation-wide storage insight.
22 min Reading + animations
Tip: tap or hover any dotted-underlined term to see what it means.
By the end of this lesson you will be able to
- Configure S3 event notifications: choose event types, filter by prefix and suffix, pick SNS, SQS or Lambda as the destination, and read the event message a target receives.
- Explain why a notification needs a resource-based policy on its destination (and a key policy for an encrypted queue or topic), and fix the “Unable to validate the following destination configurations” error.
- Apply the overlap rule, avoid recursive loops, and decide when to send S3 events to Amazon EventBridge instead.
- Explain Requester Pays: who pays for storage, requests and transfer, what the requester must send, and when the owner still pays.
- Plan an S3 Batch Operations job: build the object list with S3 Inventory (and Athena) or generated filters, choose the operation, give it an IAM role, read the completion report, and predict when a job fails.
- Choose between S3 Storage Lens free and advanced metrics, and know where Amazon Athena fits next to all of this.
#A bucket that reacts and a bucket you operate
The first six lessons of this chapter treated S3 as a place to put objects: buckets and keys (lesson 10.1), who can read them (10.2), websites (10.3), versions and copies (10.4), storage classes (10.5) and speed (10.6). This lesson is about two things that start once a bucket is busy:
- Reacting to change. A new photo should get a thumbnail, a new invoice should be booked, a deleted file should raise an alert. Nobody should poll the bucket every minute to find out what changed: S3 can tell you, with event notifications or through Amazon EventBridge.
- Operating at scale. A bucket with 2 billion objects raises questions a script cannot answer well: who pays when outsiders download a public dataset (Requester Pays), how to change tags or encryption on every object at once (S3 Batch Operations, fed by S3 Inventory), and where the storage and the requests actually go (S3 Storage Lens, with Athena for questions SQL can answer).
#Event notifications
#What S3 can tell you about
An event notification is a message S3 sends when something happens to an object in a bucket. You store the configuration on the bucket (its notification subresource), and S3 publishes a message for every matching event. The event types you will meet most:
| Event type | Fires when |
|---|---|
s3:ObjectCreated:* (or :Put, :Post, :Copy, :CompleteMultipartUpload) | An object is created, by any API or by the one you name |
s3:ObjectRemoved:* (:Delete, :DeleteMarkerCreated) | An object is deleted, or a delete marker is added in a versioned bucket (lesson 10.4) |
s3:ObjectRestore:* (:Post, :Completed, :Delete) | A restore from Glacier Flexible Retrieval or Deep Archive starts, finishes, or the temporary copy expires (lesson 10.5) |
s3:Replication:* | Replication with metrics or RTC fails, misses the 15-minute threshold, or catches up late (lesson 10.4) |
s3:LifecycleExpiration:*, s3:LifecycleTransition | A lifecycle rule deletes or moves an object |
s3:IntelligentTiering | Intelligent-Tiering moves an object to an archive tier |
s3:ObjectTagging:*, s3:ObjectAcl:Put, s3:ObjectRetention:Put | Tags are added or removed, an ACL changes, Object Lock retention changes |
s3:ReducedRedundancyLostObject | An object in the old Reduced Redundancy class is lost |
Two details catch people out:
s3:ObjectRemoved:*does not fire when a lifecycle rule deletes an object. For those, subscribe tos3:LifecycleExpiration:*.- A large upload done with multipart (lesson 10.1) is
s3:ObjectCreated:CompleteMultipartUpload, not:Put. A rule that listens only for:Putmisses it;s3:ObjectCreated:*catches every way an object is created.
#Filtering by prefix and suffix
Each rule can narrow the events by the object key:
- a prefix, such as
uploads/(the key must start with it), and/or - a suffix, such as
.jpg(the key must end with it).
When both are set, both must match. That is all the filtering native notifications can do: no wildcards (*.jpg is not valid; write the suffix .jpg), no filter on object size, tags or metadata, and the match is case-sensitive, so Uploads/ is not uploads/. Spaces in a filter are written as +, and other special characters must be URL-encoded.
#Where events can go
A rule sends its events to exactly one destination, and there are three kinds, plus EventBridge for the whole bucket:
- an Amazon SNS topic (standard only, not FIFO): a push to many subscribers at once, such as email, mobile push, or several SQS queues (fan-out, covered in depth in lesson 13.4);
- an Amazon SQS queue (standard only, not FIFO): a buffer that a worker drains at its own pace (lesson 13.1);
- an AWS Lambda function: code that runs once per event (lesson 15.1);
- Amazon EventBridge: switched on for the whole bucket, with its own section below.
The SNS topic, SQS queue and Lambda function must be in the same Region as the bucket. An SQS queue should allow messages of at least 64 KB, the largest an S3 event message can be.
#The event message
The target receives a JSON document with a Records array. Each record carries the event name, the time, the bucket, and the object’s key, size, ETag, version ID (in a versioned bucket) and a sequencer:
{
"Records": [
{
"eventSource": "aws:s3",
"awsRegion": "ap-south-1",
"eventTime": "2026-10-25T09:14:03.512Z",
"eventName": "ObjectCreated:Put",
"s3": {
"bucket": { "name": "shop-media-example", "arn": "arn:aws:s3:::shop-media-example" },
"object": {
"key": "uploads/summer+sale+shirt.jpg",
"size": 3145728,
"eTag": "d41d8cd98f00b204e9800998ecf8427e",
"sequencer": "0066A1B2C3D4E5F6A7"
}
}
}
]
}
Three properties matter when you write the code that receives it:
- Timing. Notifications usually arrive within seconds, but sometimes take a minute or longer. They are not a real-time guarantee.
- At least once, not in order. S3 is designed to deliver every notification at least once. On rare occasions a retry delivers the same event twice, and two events for different writes can arrive out of order. Make the handler idempotent (doing the same work twice is harmless), and when order matters for one key, compare the
sequencervalues: for the same key, a larger sequencer means a later event. - The key is URL-encoded. The object
uploads/summer sale shirt.jpgarrives asuploads/summer+sale+shirt.jpg. Code that passes the key straight toGetObjectgets “NoSuchKey”; decode it first.
Watch a full flow, including a filter that does not match, a destination that refuses S3, and a loop:
One upload, one event, one thumbnail
A Dhaka clothing shop uploads product photos to uploads/ in its bucket. A notification rule (prefix uploads/, suffix .jpg) calls a Lambda function that writes a small copy to thumbs/.
Scenario
The upload matches the rule, Lambda is invoked within seconds, and its own output does not trigger it again.
The admin uploads uploads/shirt-42.jpg with a normal PUT. S3 stores the object first; the event is a side effect of a successful write, never a condition for it.
#The overlap rule
S3 refuses a configuration in which two rules could both claim the same event. Two rules overlap, and the configuration cannot be saved, when all three are true:
- their event types overlap (
s3:ObjectCreated:*overlapss3:ObjectCreated:Put, but nots3:ObjectRemoved:*); - their prefixes overlap: one starts with the other. An empty prefix overlaps every prefix;
- their suffixes overlap: one ends with the other (
jpgand.jpgoverlap,pgandjpgoverlap,.jpgand.pngdo not). An empty suffix overlaps every suffix.
So the owner’s notes are right in practice: you cannot have two rules for the same prefix and the same event type unless their suffixes are different. Prefix images/ + suffix .jpg to one function and prefix images/ + suffix .png to another is fine; prefix images/ to a queue and prefix images/ to a function, for the same event, is rejected. To send the same events to several places, either:
- send them to one SNS topic and subscribe several SQS queues or functions to it (the classic fan-out), or
- turn on EventBridge and write as many rules as you like.
Try it: switch rules on and off, save, then upload test files.
Event rule lab
Switch rules on or off, save the configuration the way S3 would check it, then upload test files and see which rules fire.
Notification rules on shop-media-example
No saved configuration yet. Press “Save configuration”, then “Upload test files”.
Self-checkQuestion
A bucket already has a rule that sends every object-created event under logs/ to a Lambda function. A second team adds a rule that sends every object-created event under logs/ to their SQS queue. What happens?
#Letting S3 deliver: resource policies
#Why the destination must say yes
S3 delivers a notification by calling the destination as the service principal s3.amazonaws.com. Your IAM user’s permissions do not travel with the event, so S3 can only deliver if the destination itself allows it, through its resource-based policy (the policy attached to the resource, lesson 2.2):
| Destination | Policy | Action S3 needs |
|---|---|---|
| SNS topic | Topic access policy | sns:Publish |
| SQS queue | Queue access policy | sqs:SendMessage |
| Lambda function | Function resource policy (aws lambda add-permission) | lambda:InvokeFunction |
| EventBridge | None: S3 needs no extra permission to deliver to EventBridge | – |
The owner’s notes call this “IAM permission”. It is IAM in the broad sense, but the exam and the console mean a resource (access) policy on the SNS topic, SQS queue or Lambda function, not a policy on a user or role. The Lambda one is easy to forget: the console adds it for you when you pick a function in the S3 console, but a template or the CLI must add it explicitly.
A queue policy that lets only one bucket in one account send to it:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowS3ShopMedia",
"Effect": "Allow",
"Principal": { "Service": "s3.amazonaws.com" },
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:ap-south-1:111122223333:audit",
"Condition": {
"ArnLike": { "aws:SourceArn": "arn:aws:s3:*:*:shop-media-example" },
"StringEquals": { "aws:SourceAccount": "111122223333" }
}
}
]
}
The two conditions stop the confused deputy problem: without them, any bucket in any account could point its notifications at your queue, and S3 would deliver because the policy trusts “the S3 service” in general. aws:SourceArn names the bucket; aws:SourceAccount pins the bucket owner, because a bucket ARN has no account ID in it.
#Encrypted queues and topics
If the SQS queue or SNS topic uses server-side encryption with KMS (lesson 10.8 covers KMS), S3 must also be able to use the key: the key policy must allow s3.amazonaws.com to call kms:GenerateDataKey and kms:Decrypt. You cannot edit the key policy of the AWS managed keys (aws/sqs, aws/sns), so an encrypted destination for S3 needs a customer managed key.
#What a missing permission looks like
When you save a configuration, S3 checks every destination and sends it an s3:TestEvent. If the destination’s policy does not allow S3, the save fails with “Unable to validate the following destination configurations” and the bucket keeps its old configuration. If someone removes the permission later, nothing fails loudly: events simply stop arriving. When notifications go missing, check the destination’s policy first, then the KMS key policy, then the filters.
#Sending S3 events to Amazon EventBridge
#What changes
Amazon EventBridge is a serverless event bus (it has its own lesson, 19.5). One setting on the bucket, “Send notifications to Amazon EventBridge for all events in this bucket”, makes S3 send every supported event to the default event bus in the same Region. You do not pick event types and S3 needs no permission. EventBridge rules then match events and route them to targets.
| Native notifications | Through EventBridge | |
|---|---|---|
| Destinations | SNS (standard), SQS (standard), Lambda | 20+ kinds of AWS targets, such as Step Functions, Kinesis Data Streams, Data Firehose, SQS FIFO, ECS tasks, API destinations; up to 5 targets per rule |
| Filtering | Key prefix and suffix only | JSON patterns on any field: key prefix, suffix or wildcard, object size, bucket, the API that created the object, requester, and more |
| Same event, many consumers | Not on overlapping rules; use SNS fan-out | Any number of rules may match the same event |
| Archive and replay | No | Yes: archive events, replay a time window later |
| Event names | s3:ObjectCreated:Put style | “Object Created”, “Object Deleted”, “Object Restore Completed”, “Object Storage Class Changed”, “Object Tags Added”… |
| Cost | No charge from S3 for notifications | EventBridge pricing for events and replays |
A rule that starts a transcoding workflow only for big videos:
{
"source": ["aws.s3"],
"detail-type": ["Object Created"],
"detail": {
"bucket": { "name": ["videos-example"] },
"object": {
"key": [{ "suffix": ".mp4" }],
"size": [{ "numeric": [">", 104857600] }]
}
}
}
S3 → EventBridge → many rules
A video platform turns on EventBridge for its upload bucket. Every event goes to the default event bus, and rules decide who gets what.
Scenario
One 2 GB .mp4 upload matches three rules with three different targets.
Turning EventBridge on is a single bucket setting. You do not choose event types, and S3 needs no extra permission: every supported event of the bucket goes to the default bus in the same Region.
#Requester Pays
#Who pays for what
Normally the bucket owner pays for everything: storage, every request, and every gigabyte downloaded. That is fine for your own application, and a problem when you publish a 50 TB dataset that hundreds of other organisations download: their curiosity becomes your bill. A Requester Pays bucket moves part of that bill:
| Cost | Normal bucket | Requester Pays bucket |
|---|---|---|
| Storage | Owner | Owner (always) |
| Requests (GET, PUT, list…) | Owner | Requester |
| Data transfer out of S3 | Owner | Requester |
| Retrieval from an archive class on restore | Owner | Owner (the requester pays only the restore request) |
You turn it on per bucket (aws s3api put-bucket-request-payment … Payer=Requester). It is how large public datasets are often shared, for example many datasets in the Registry of Open Data on AWS.
#The rules that make it work
- Requests must be authenticated. S3 has to know which AWS account to bill, so a Requester Pays bucket refuses anonymous requests. It cannot serve an anonymous public website.
- The requester must accept the charge. Every request carries the header
x-amz-request-payer: requester(in the CLI:--request-payer requester). Without it, S3 answers 403 Access Denied, even if the bucket policy allows the read. The bucket owner’s own requests do not need the header. - Requester Pays does not grant access. The requester still needs permission: a bucket policy that allows the other account (lesson 10.2), or a public dataset policy that allows authenticated principals.
- Roles count for the account that owns them. If the requester assumes a role in the owner’s account, the request comes from the owner’s account and the owner pays. Requester Pays only shifts the bill when requesters use credentials from their own accounts.
- Some failed requests are still the owner’s. The owner pays for a request that fails with 403 when it comes from inside the owner’s own account or AWS Organization.
- A Requester Pays bucket cannot be the target bucket of server access logging (lesson 10.9 covers access logs).
Play with the setting, the downloader and the destination:
Requester Pays: who gets the bill?
A research institute in Dhaka shares 10 TB of satellite images. Choose the setting and the downloader, then press Download.
Who downloads
Data goes to
Press Download to see the result and the bill.
Calculating it. 20 TB (20,480 GB) in S3 Standard at $0.023 per GB-month costs the owner $471.04 a month, with or without Requester Pays. A partner downloads 5 TB (5,120 GB) to the internet at $0.09 per GB: $460.80 of transfer, plus about 52,000 GET requests for 100 MB objects at $0.0004 per 1,000 (2 cents). Without Requester Pays the owner pays $931.86; with it, the owner pays $471.04 and the partner $460.82. (Example list prices; real bills use tiers and a monthly free allowance.)
#S3 Batch Operations
#Why not a script
“Add the tag retention=7y to every object”, “re-encrypt every old object with our own KMS key”, “copy 400 million objects to the new account’s bucket”, “restore every scan from 2020 out of Deep Archive”. A script can do that for a thousand objects. For a billion it has to list the bucket for hours, run on a machine for days, handle throttling and retries, survive a reboot, and prove at the end which objects failed.
S3 Batch Operations is S3 doing that work for you: a job runs one operation on every object in a list, with retries, progress tracking, notifications and a completion report, as a fully managed, serverless service. One job can cover up to 20 billion objects (and since December 2025 large jobs run up to 10 times faster).
#What a job is made of
- A manifest: the list of objects. Either an S3 Inventory report (its
manifest.json), a CSV file ofbucket,key(with an optional version ID column), or no file at all: Batch Operations can generate the list for you from filters such as prefix, suffix, substring, creation date range, size, storage class or encryption type, and save it as a manifest. A replication configuration can also generate the list (that is Batch Replication, lesson 10.4). - One operation, with its parameters (see the table below). One job, one operation; for two operations, run two jobs.
- An IAM role that Batch Operations assumes. Its trust policy allows the service principal
batchoperations.s3.amazonaws.com; its permissions cover reading the manifest, writing the report, and the operation itself (for examples3:GetObjecton the source ands3:PutObjecton the destination for a copy, plus KMS permissions for encrypted data). The console can create it for you. Roles and trust policies were lesson 2.4. - A completion report (optional but recommended): a CSV in a bucket you choose, listing all tasks or only the failed ones with their error codes.
- A priority among your jobs, and optional tags and description.
#What a job can do
| Operation | Typical use |
|---|---|
| Copy | Copy objects to another bucket or Region, change storage class, metadata or encryption; copy in place to encrypt old unencrypted objects. Objects up to 5 GB; all sources in one bucket, all destinations in one bucket |
| Invoke Lambda | Any custom per-object action: resize, scan, transcode, call an API, or copy objects larger than 5 GB with your own multipart code |
| Replace all object tags / Delete all object tags | Bulk tagging for cost allocation, lifecycle or access control |
| Replace ACL | Bulk ACL changes on buckets that still use ACLs |
| Restore | Start restores from Glacier Flexible Retrieval or Deep Archive for millions of objects |
| Object Lock retention / legal hold | Set retention dates or legal holds in bulk (Object Lock is lesson 10.9) |
| Update object encryption | Change SSE-S3 objects to SSE-KMS, add S3 Bucket Keys, or move to another customer managed key, without copying (not for unencrypted or SSE-C objects) |
| Compute checksums | Calculate checksums of existing objects to verify integrity |
| Replicate (Batch Replication) | Replicate existing objects that live replication skipped (lesson 10.4) |
There is no “delete objects” operation. To remove millions of objects, use a lifecycle expiration rule (lesson 10.5), or Invoke Lambda if the choice of objects is complex.
Some rules of the job itself: a copy job runs in the destination Region, every other job in the Region of the objects; objects are processed in no particular order; a job created in the console waits for your confirmation (status Suspended) and fails if left there 30 days; and once at least 1,000 tasks have run, a failure rate above 50% makes S3 stop the job as Failed, so a bad list cannot run for hours. The manifest is a list, not a snapshot: if an object is overwritten while the job runs and the manifest has no version IDs, the job acts on the newest version.
What it costs: $0.25 per job plus $1.00 per million objects processed, on top of the normal request charges of the operation (and Lambda charges for Invoke Lambda).
#S3 Inventory: the list for the job
S3 Inventory produces a scheduled list of the objects in a bucket (or a prefix) with the metadata you choose: size, last modified date, storage class, ETag, encryption status (SSE-S3, SSE-KMS, DSSE-KMS, SSE-C, NOT-SSE), replication status, Object Lock settings, Intelligent-Tiering tier, version IDs, and more.
- Delivered daily or weekly, as CSV, ORC or Parquet, to a destination bucket in the same Region (which can belong to another account and needs a bucket policy that lets S3 write to it). The first report can take up to 48 hours.
- It does not use
Listrequests and does not touch your bucket’s request rate, which makes it far cheaper than listing a billion keys. - Each list is eventually consistent: very recent writes and deletes may be missing.
- You can query it with Amazon Athena, which is how you turn “every object” into “every object that is still
NOT-SSEunderscans/2019/”, and save the result as the job’s CSV manifest.
The slides draw this pipeline: S3 Inventory → Athena (filter) → filtered list → S3 Batch Operations (operation + parameters) → processed objects.
A Batch Operations job, start to finish
A hospital must re-encrypt 3 million old scan images with its own KMS key. Follow the list, the job, the role and the report.
Scenario
Inventory → Athena filter → manifest → job → confirm → run → report.
S3 Inventory writes a list of every object in the scans bucket, with the fields you chose (size, storage class, encryption status), plus a manifest.json that points at the files. It reads the bucket in the background, so it does not use your request rate.
Batch Operations is also one of the thirteen ways data moves into or out of S3; lesson 10.12 puts it next to multipart upload and pre-signed URLs in that chooser.
#S3 Storage Lens
#One view of all your storage
A company with 40 accounts and 3,000 buckets cannot answer simple questions by clicking through buckets: Which buckets grow fastest? Where are terabytes of incomplete multipart uploads hiding? Which buckets have no versioning? S3 Storage Lens collects usage and activity metrics daily and shows them in dashboards, aggregated by organisation, account, Region, storage class, bucket and, with advanced metrics, prefix (and custom Storage Lens groups defined by prefix, suffix, tags, size or age).
- Every account has a default dashboard,
default-account-dashboard, configured by S3. You can upgrade its metrics or disable it, but not delete it. - You can create up to 50 more dashboards per home Region, scoped to Regions, buckets or accounts. Organisation-wide dashboards need trusted access in AWS Organizations and are created from the management account or a delegated administrator.
- A metrics export sends the metrics daily as CSV or Parquet to a bucket (or into an AWS managed S3 table bucket, ready for SQL queries).
- Recommendations and outlier call-outs compare today with your last 30 days and flag unusual changes.
#Free metrics and advanced metrics
| Free metrics | Advanced metrics and recommendations | |
|---|---|---|
| Price | Free, on by default | Paid: about $0.20 per million objects monitored per month (cheaper above 25 billion) |
| Metric categories | Summary (StorageBytes, ObjectCount), cost optimisation (NonCurrentVersionStorageBytes, IncompleteMultipartUploadStorageBytes), data protection (buckets with versioning, MFA Delete or default encryption), access management (Object Ownership), event (buckets with notifications), performance (buckets with Transfer Acceleration) | Everything free, plus activity (AllRequests, GetRequests, PutRequests, BytesDownloaded…), detailed status codes (200OKStatusCount, 403ForbiddenErrorCount, 503s…), advanced cost optimisation (lifecycle rule counts), advanced data protection (replication rule counts, SSE-KMS usage, requests with outdated TLS or signatures), advanced performance metrics |
| Lowest level | Bucket | Prefix (prefix aggregation, expanded prefixes report), Storage Lens groups |
| History you can query | 14 days | 15 months |
| CloudWatch | No | Publish to CloudWatch for alarms and dashboards |
| Recommendations | Basic | Contextual recommendations next to the metrics |
The owner’s notes say free data is kept “14 days (now 28 days in many Regions)”. It is 14 days: the “28” is the number of usage metrics the free tier had when AWS last counted them (2023), and the free categories have grown since. The slides’ “30 days usage and activity metrics” is also misleading: the 30 days are the window for outlier detection; activity metrics are advanced only.
What Storage Lens can see
Storage Lens rolls up metrics from the whole organisation down to one bucket, and with advanced metrics down to one prefix.
Scenario
Every account has a default dashboard, default-account-dashboard, already on. You cannot delete it, only disable it.
#What goes wrong
- Notification configuration will not save (“Unable to validate…”): the SNS topic, SQS queue or Lambda function has no resource policy for
s3.amazonaws.com, the destination is in another Region, the queue or topic is FIFO, or its KMS key does not allow S3. - Events stopped arriving: someone tightened the queue or topic policy or the key policy after the configuration was saved, or a deployment replaced the Lambda function without its permission.
- Some files never trigger: the suffix or prefix does not match (case,
.jpegvs.jpg), or the rule listens to:Putand the file arrived by multipart upload or copy. - A rule cannot be added: it overlaps an existing rule for the same event type. Use SNS fan-out or EventBridge.
- Lambda fails with NoSuchKey on some files: the key in the event is URL-encoded; decode it.
- Duplicate processing: delivery is at least once. Make the handler idempotent.
- Cost explodes and the function runs constantly: a recursive loop; write to another prefix or bucket.
- Partners get 403 from a Requester Pays bucket: they did not send
x-amz-request-payer, they are anonymous, or the bucket policy does not allow them. - The owner still pays for partner downloads: partners assume a role in the owner’s account.
- Batch job stuck in Suspended: it was created in the console and waits for confirmation.
- Batch job Failed: the role cannot read the manifest or write the report, or more than half the first 1,000+ tasks failed (deleted objects, missing KMS permission). Read the completion report.
- Storage Lens shows no prefixes, no request counts: the dashboard uses free metrics.
#Exam patterns at a glance
#Events and operations in system design
- Use events instead of polling. “Something landed in S3, now do X” is the most common serverless pattern on AWS: an upload bucket, a notification, a function or a queue. It scales from ten files a day to millions without changing the design.
- Put a queue in front of slow work. Lambda directly from S3 suits quick, independent jobs. For slow or bursty work, or when the consumer may be down, send events to SQS and let workers drain it, with a dead-letter queue for poison files (the pattern of lesson 9.4’s worker tier).
- Plan for fan-out from day one. If more than one team will care about new objects, start with EventBridge (or one SNS topic) instead of a direct notification to one function, so the second consumer does not need to change the first one’s rule.
- Design handlers for at-least-once, out-of-order delivery. Idempotent writes, the
sequencerfor ordering per key, and a periodic reconciliation (an inventory compared with your database) for anything that must never be missed. - Keep input and output apart. Separate prefixes or buckets for raw uploads and processed results, so rules never see their own output.
- Charge the right party. Public or partner datasets go in Requester Pays buckets; internal consumers use roles in their own accounts so cost allocation stays clean.
- Treat bulk changes as jobs. Inventory + Athena to choose the objects, Batch Operations to change them, the completion report as the audit record, Storage Lens to confirm the result across the organisation.
#Try it yourself
Event notifications to Lambda and SQS:
# 1. Let S3 invoke the function (resource-based policy on the function)
aws lambda add-permission --function-name make-thumb \
--statement-id s3-shop-media --action lambda:InvokeFunction \
--principal s3.amazonaws.com \
--source-arn arn:aws:s3:::shop-media-example --source-account 111122223333
# 2. Let S3 send to the queue (queue policy from the lesson saved in queue-policy.json)
aws sqs set-queue-attributes \
--queue-url https://sqs.ap-south-1.amazonaws.com/111122223333/audit \
--attributes file://queue-attributes.json # {"Policy": "<the policy JSON as a string>"}
# 3. The notification configuration: .jpg and .png to Lambda, everything under invoices/ to SQS
cat > notification.json <<'EOF'
{
"LambdaFunctionConfigurations": [
{ "Id": "thumbs-jpg",
"LambdaFunctionArn": "arn:aws:lambda:ap-south-1:111122223333:function:make-thumb",
"Events": ["s3:ObjectCreated:*"],
"Filter": { "Key": { "FilterRules": [
{ "Name": "prefix", "Value": "uploads/" }, { "Name": "suffix", "Value": ".jpg" } ] } } },
{ "Id": "thumbs-png",
"LambdaFunctionArn": "arn:aws:lambda:ap-south-1:111122223333:function:make-thumb",
"Events": ["s3:ObjectCreated:*"],
"Filter": { "Key": { "FilterRules": [
{ "Name": "prefix", "Value": "uploads/" }, { "Name": "suffix", "Value": ".png" } ] } } }
],
"QueueConfigurations": [
{ "Id": "invoices",
"QueueArn": "arn:aws:sqs:ap-south-1:111122223333:audit",
"Events": ["s3:ObjectCreated:*"],
"Filter": { "Key": { "FilterRules": [ { "Name": "prefix", "Value": "invoices/" } ] } } }
],
"EventBridgeConfiguration": {}
}
EOF
aws s3api put-bucket-notification-configuration --bucket shop-media-example \
--notification-configuration file://notification.json
aws s3api get-bucket-notification-configuration --bucket shop-media-example
# 4. Test: upload, then watch the queue and the function's logs
aws s3 cp shirt-42.jpg s3://shop-media-example/uploads/shirt-42.jpg
aws sqs receive-message --queue-url https://sqs.ap-south-1.amazonaws.com/111122223333/audit
aws logs tail /aws/lambda/make-thumb --since 5m
The "EventBridgeConfiguration": {} line is the whole switch for EventBridge. A rule on the default bus:
aws events put-rule --name big-videos \
--event-pattern file://big-videos-pattern.json # the JSON pattern from the lesson
aws events put-targets --rule big-videos \
--targets Id=transcode,Arn=arn:aws:states:ap-south-1:111122223333:stateMachine:transcode,RoleArn=arn:aws:iam::111122223333:role/events-to-sfn
Requester Pays:
# Owner: turn it on and check
aws s3api put-bucket-request-payment --bucket bay-images-example \
--request-payment-configuration Payer=Requester
aws s3api get-bucket-request-payment --bucket bay-images-example
# Partner (its own account's credentials): accept the charges on every request
aws s3 ls s3://bay-images-example/2026/ --request-payer requester
aws s3 cp s3://bay-images-example/2026/10/scene-0042.tif . --request-payer requester
aws s3 cp s3://bay-images-example/2026/10/scene-0042.tif . # → 403 Access Denied
S3 Inventory and a Batch Operations job:
# Daily CSV inventory of scans-example, with encryption status, into reports-example
aws s3api put-bucket-inventory-configuration --bucket scans-example --id daily-all \
--inventory-configuration '{"Id":"daily-all","IsEnabled":true,"IncludedObjectVersions":"Current",
"Schedule":{"Frequency":"Daily"},
"OptionalFields":["Size","StorageClass","EncryptionStatus"],
"Destination":{"S3BucketDestination":{"Bucket":"arn:aws:s3:::reports-example",
"Format":"CSV","Prefix":"inventory"}}}'
# A tagging job over a CSV manifest, no console confirmation needed
aws s3control create-job --account-id 111122223333 --region ap-south-1 \
--operation '{"S3PutObjectTagging":{"TagSet":[{"Key":"retention","Value":"7y"}]}}' \
--manifest '{"Spec":{"Format":"S3BatchOperations_CSV_20180820","Fields":["Bucket","Key"]},
"Location":{"ObjectArn":"arn:aws:s3:::reports-example/manifests/scans-2019.csv","ETag":"<etag of the csv>"}}' \
--report '{"Bucket":"arn:aws:s3:::reports-example","Prefix":"batch-reports",
"Format":"Report_CSV_20180820","Enabled":true,"ReportScope":"FailedTasksOnly"}' \
--priority 10 --role-arn arn:aws:iam::111122223333:role/s3-batch-tagging \
--no-confirmation-required
aws s3control list-jobs --account-id 111122223333 --region ap-south-1
aws s3control describe-job --account-id 111122223333 --region ap-south-1 --job-id <job-id>
aws s3control update-job-status --account-id 111122223333 --region ap-south-1 \
--job-id <job-id> --requested-job-status Ready # confirm a Suspended job
Storage Lens:
aws s3control list-storage-lens-configurations --account-id 111122223333 --region us-east-1
aws s3control get-storage-lens-configuration --account-id 111122223333 \
--config-id default-account-dashboard --region us-east-1
#Summary
Key takeaways
- Event notifications publish object events (created, removed, restore, replication, lifecycle, tagging…) to SNS (standard), SQS (standard) or Lambda in the same Region, one destination per rule, filtered only by key prefix and suffix (no wildcards, case-sensitive).
- Delivery is usually seconds, sometimes a minute or more, at least once and not ordered; keys arrive URL-encoded. Make handlers idempotent and keep output out of the watched prefix to avoid recursive loops.
- The destination must allow
s3.amazonaws.comin its resource-based policy (sns:Publish,sqs:SendMessage,lambda:InvokeFunction), ideally limited withaws:SourceArnandaws:SourceAccount; encrypted queues and topics need a customer managed KMS key whose policy allows S3. Missing permission → “Unable to validate the following destination configurations”. - Rules for the same event type cannot have overlapping prefixes and suffixes. For many consumers use SNS fan-out or EventBridge.
- EventBridge: one bucket switch, every event, no permission needed; JSON rules on size, metadata and more; 20+ target types including Step Functions and SQS FIFO; archive and replay.
- Requester Pays: owner pays storage, requester pays requests and transfer; requesters must be authenticated and send
x-amz-request-payer: requester; a role in the owner’s account means the owner pays. - S3 Batch Operations: one operation on billions of objects from a manifest (S3 Inventory, CSV, or generated by filters) with an IAM role, retries, a completion report, console confirmation, and a 50% failure threshold after 1,000 tasks. Copy (≤ 5 GB, encrypt in place), Lambda, tags, ACL, restore, Object Lock, update encryption, checksums, Batch Replication; no delete.
- S3 Inventory: daily or weekly CSV/ORC/Parquet lists with encryption and replication status; query with Athena (serverless SQL on S3, ~$5 per TB scanned; Parquet, compression and partitions cut cost).
- S3 Storage Lens: organisation-wide dashboards. Free: bucket level, 14 days. Advanced: prefix level, activity and status-code metrics, 15 months, CloudWatch, recommendations.
Answer on your own first, then read the explanation, even when you got it right.
Self-checkQuestion
Which list contains exactly the destinations to which S3 can send event notifications?
Self-checkQuestion
Which two statements about filtering native S3 event notifications are true?
Choose 2
Self-checkQuestion
An online shop wants a thumbnail to be created as soon as a product image is uploaded to S3, with no servers to manage. What is the simplest design?
Self-checkQuestion
An engineer adds an event notification that sends object-created events to an SQS queue. Saving fails with Unable to validate the following destination configurations. The engineer has full S3 and SQS permissions. What is the most likely cause?
Self-checkQuestion
An S3 bucket must invoke a Lambda function on every upload. The notification is created with the CLI. What else is required?
Self-checkQuestion
Orders are uploaded as files to S3 and must be processed strictly in order per customer by a consumer reading an SQS FIFO queue. How should events reach the queue?
Self-checkQuestion
Which two notification configurations will S3 reject? Assume every rule uses the object-created event type.
Choose 2
Self-checkQuestion
A media company must start three independent workflows (a Step Functions transcoder, a cataloguing function and an analytics stream) when a video is uploaded, and the transcoder only for files over 1 GB. What should the architect use?
Self-checkQuestion
Without EventBridge, two teams each need their own SQS queue to receive every new-object event under uploads/. What is the standard solution?
Self-checkQuestion
A Lambda function triggered by uploads to images/ in a bucket saves a resized copy into images/resized/. Within an hour the Lambda bill and the S3 request count jump enormously. What is wrong, and how is it fixed?
Self-checkQuestion
An order-processing function occasionally receives the same S3 event twice and books an order twice. What is the right fix?
Self-checkQuestion
A thumbnail function works for most files but fails with NoSuchKey for files whose names contain spaces, such as summer sale.jpg. What is the cause?
Self-checkQuestion
Which two statements about sending S3 events to Amazon EventBridge are true?
Choose 2
Self-checkQuestion
A bucket has Requester Pays enabled. Which split of costs is correct for a successful download by another AWS account?
Self-checkQuestion
A genomics institute wants to share a 50 TB dataset with researchers in other AWS accounts. Its budget covers storage only, not the cost of other people’s downloads. What should it do?
Self-checkQuestion
A partner account has read access to a Requester Pays bucket through the bucket policy, yet every download returns 403 Access Denied. What is missing?
Self-checkQuestion
A team turns on Requester Pays for the bucket behind its public static website to save on transfer costs. What happens?
Self-checkQuestion
To avoid managing bucket policies, partners access a Requester Pays bucket by assuming an IAM role that lives in the bucket owner’s account. Who pays for their downloads?
Self-checkQuestion
A Requester Pays bucket stores 20 TB (20,480 GB) in S3 Standard at $0.023 per GB-month. A partner downloads 5 TB (5,120 GB) to the internet at $0.09 per GB. Ignoring request charges, what does each side pay for the month?
Self-checkQuestion
A compliance team must add a retention tag to 2 billion existing objects across one bucket, with a record of any object that failed. Which approach fits best?
Self-checkQuestion
Which two items must every S3 Batch Operations job have?
Choose 2
Self-checkQuestion
An auditor finds that 3 million objects uploaded before 2023 are still unencrypted. The team must encrypt them in place with minimal effort. What should they do?
Self-checkQuestion
A Batch Operations job runs for twenty minutes and ends in the Failed state. Its report shows that most tasks failed because the objects no longer exist. What happened?
Self-checkQuestion
An engineer created a Batch Operations job in the S3 console two days ago. Its status is still Suspended. Why?
Self-checkQuestion
One Batch Operations job replaces the tags on 40 million objects. At $0.25 per job and $1.00 per million objects processed, what is the Batch Operations fee, before the request charges of the tagging calls?
Self-checkQuestion
Which of the following is NOT an operation that S3 Batch Operations offers?
Self-checkQuestion
A company with 60 AWS accounts wants to find which S3 prefix receives the most requests across the whole organisation, to move cold data and save cost. What does it need?
Self-checkQuestion
Which two statements about S3 Storage Lens free metrics are true?
Choose 2
Self-checkQuestion
A team stores ALB access logs in S3 and wants to run occasional SQL queries on them without running or loading a database, at the lowest cost. What should it use?
Self-checkQuestion
Operations wants an alert whenever an S3 lifecycle rule permanently deletes objects in a bucket. They added a notification rule for object-removed events, but it never fires for these deletions. What should they do?