Buckets, Objects and Keys

By the end of this lesson you will be able to

• Explain what Amazon S3 is, why it is object storage rather than a disk or a file system, and name the everyday jobs it does: photo and media storage, backups, log archives, data lakes, software downloads and static websites.
• Explain why a bucket has a globally unique name but lives in one Region, apply every bucket naming rule, and choose between the shared global namespace and your account regional namespace.
• Describe objects, keys and prefixes, and explain why S3 has no real folders even though the console shows them.
• State the size limits (50 TB per object, 5 GB per PUT, 10,000 parts of 5 MiB to 5 GiB), decide when to use multipart upload, and calculate part counts and part sizes.
• Tell system metadata, user metadata, tags and the version ID apart, and say which of them can change after upload.
• Explain S3’s strong read-after-write consistency, what is still eventually consistent, and how conditional writes stop two writers from overwriting each other.
• List what you pay for in S3 and estimate a simple monthly bill.

#What Amazon S3 is

Amazon S3 (Simple Storage Service) is AWS’s object storage service, and one of the oldest and most used building blocks of AWS. It is sold as storage that scales without limit: you never create a volume, choose a size or add capacity. You put data in, you pay for what is stored, and S3 grows with you. Many websites use S3 as their backbone, and many AWS services use it underneath: EBS snapshots and AMIs are kept in S3, CloudTrail and load balancers write their logs to S3, and Athena, Glue and Redshift read data straight from it.

#Object storage, not a disk

Lesson 5.4 compared the three kinds of storage. S3 is the third kind:

Because S3 talks HTTP, every object has an address, and a browser, a phone app, a Lambda function or another AWS service can reach it without mounting anything. That is what makes it the default place to put files that many systems share.

#What people use S3 for

#Buckets

A bucket is the container that holds objects. The slides call buckets “directories”, which is a useful first picture but not quite right: a bucket is a top-level container with a name, a Region, an owner and its own settings (versioning, encryption, policies, lifecycle rules), and it cannot be nested inside another bucket.

#A global name, a regional home

Two facts about buckets confuse almost everyone at first, and the exam tests both:

1. The name is global. A general purpose bucket name must be unique across every AWS account in every Region of the partition. (A partition is a group of Regions: the standard aws partition, China, GovCloud (US) and the European Sovereign Cloud each have their own namespace.) Once you own dhaka-shop-photos-example, nobody else can create it, anywhere, until you delete it.
2. The bucket lives in one Region. You pick the Region when you create it, and its objects are stored only there unless you copy or replicate them yourself. You cannot move a bucket to another Region or rename it later.

The S3 console adds to the confusion: it shows the buckets of all Regions in one list, so S3 looks like a global service. It is not. Every bucket has a Region, and requests go to that Region’s endpoint.

Why choose the Region carefully?

• Latency: a Dhaka shop serving customers in Bangladesh puts its bucket in ap-south-1 (Mumbai) or ap-southeast-1 (Singapore), not in us-east-1.
• Cost: prices differ slightly by Region, and moving data between Regions is charged.
• Law and compliance: a bank or hospital may be required to keep data in a certain country; S3 guarantees objects do not leave the Region on their own.

A few more bucket facts worth knowing:

• Private by default. A new bucket and its objects can be read only by the owner’s account. Block Public Access is on by default and ACLs are disabled by default (lesson 10.2 covers access).
• Encrypted by default. Since January 2023 every new object is encrypted at rest with SSE-S3 unless you choose otherwise (lesson 10.8).
• Quota. An account can create 10,000 general purpose buckets by default (raised from 100 in November 2024) and can request more. There is no limit on objects or bytes per bucket.
• The bucket pays. The owner’s account is billed for storage and requests, and the bucket is the unit that usage reports add up.

#Naming rules

A general purpose bucket name must follow all of these rules:

• 3 to 63 characters long.
• Only lowercase letters, digits, dots (.) and hyphens (-). No uppercase, no underscores, no spaces.
• Begin and end with a letter or a digit, and never two dots side by side.
• Not formatted like an IP address, such as 192.0.2.10.
• Not start with the reserved prefixes xn--, sthree- or amzn-s3-demo-.
• Not end with the reserved suffixes -s3alias (access point aliases), --ol-s3 (Object Lambda access point aliases), .mrap (Multi-Region Access Points), --x-s3 (directory buckets) or --table-s3 (table buckets). The suffix -an is allowed only for names in your account regional namespace.

Why so strict? Because the bucket name becomes part of a DNS host name: dhaka-shop-photos-example.s3.ap-south-1.amazonaws.com. DNS names are case-insensitive and do not allow underscores, so neither do bucket names. Before March 2018, buckets in us-east-1 could have uppercase letters and underscores; a few of those old buckets still exist, but no new one can be created that way.

Try the rules yourself. The checker tests a name against every rule, warns about dots and easy-to-guess names, and shows what the account regional suffix looks like.

#How a bucket is addressed

Every object has a URL. AWS uses three forms:

A website endpoint (…s3-website.ap-south-1.amazonaws.com) is a fourth, separate address that serves pages over plain HTTP; it belongs to lesson 10.3. Having a URL does not make an object public: a GET without permission gets 403 Access Denied.

#Objects and keys

An object is one item stored in a bucket: the data (the value, the bytes of the file) plus information about it. The slides call objects “files”, which is a good everyday word, as long as you remember that S3 never opens or understands them.

#The key is the full name

Each object has exactly one key, its name inside the bucket. The key is the full path, not just the file name:

• s3://dhaka-shop-photos-example/index.html → key index.html
• s3://dhaka-shop-photos-example/photos/2026/eid/cake.jpg → key photos/2026/eid/cake.jpg

A key is any UTF-8 string up to 1,024 bytes (about 1,024 Latin characters; Bengali characters take 3 bytes each). It is case sensitive: Photos/cake.jpg and photos/cake.jpg are two different objects. The bucket + key (and, with versioning, the version ID) identify an object uniquely. You can think of S3 as one huge map from “bucket + key + version” to the object.

Safe key characters are letters, digits and ! - _ . * ' ( ). Characters such as & $ @ = ; : + , ? and spaces work but must be URL-encoded. Avoid the backslash, braces, square and angle brackets, the caret, percent sign, backtick, quotation mark, tilde, hash and pipe, and non-printable characters. Also avoid path segments that are just . or .., which tools may “normalise” away.

#No folders: just prefixes

S3 has no directories. A general purpose bucket is a flat namespace: a long sorted list of keys. The slash in photos/2026/eid/cake.jpg is an ordinary character. The part up to the last slash, photos/2026/eid/, is called the prefix, and the rest, cake.jpg, is the object name. Key = prefix + object name.

The console “tricks you”, as the slides put it, into seeing folders. When you open a bucket, it asks S3 to list keys with the delimiter /. S3 then groups every key by the text up to its first slash and returns those groups as common prefixes, which the console draws as folders. Clicking a folder is just another list call with a longer prefix.

This has practical consequences:

• Create folder in the console uploads a zero-byte object whose key ends in /, such as photos/2027/. The console hides it; the CLI and SDKs see it as a normal object.
• A “folder” that was never created that way exists only while at least one key starts with its prefix. Delete the last such object and the folder disappears.
• There is no rename, for a folder or for an object. Renaming photos/ to images/ means copying every object to a new key and deleting the old one: two requests per object. Plan key names before you load millions of objects.
• Permissions and lifecycle rules can still target “folders”, because they match on prefix: a policy resource arn:aws:s3:::dhaka-shop-photos-example/photos/* covers every key that starts with photos/.
• Prefixes also matter for performance: S3 scales request rates per prefix (3,500 writes and 5,500 reads per second per prefix), so spreading busy keys across prefixes raises the ceiling. Lesson 10.6 covers it.

#How big can an object be?

• An object can be anything from 0 bytes up to 50 TB.
• One PUT request can upload at most 5 GB. Anything bigger must use multipart upload.

#What an object carries

Besides its key and its data, every object carries descriptive information. Knowing which piece is which, and which can change, answers a surprising number of exam questions.

#Metadata: system and user

Metadata is a set of name–value pairs that describe the object. The notes divide it into two kinds:

• System metadata is defined by S3. Some of it only S3 can set, such as Date, Content-Length (the size), Last-Modified and the ETag. Some of it you choose at upload, such as Content-Type (image/jpeg, so browsers display the photo instead of downloading it), Cache-Control, the storage class and the encryption settings.
• User metadata is your own pairs, sent as HTTP headers that start with x-amz-meta-, for example x-amz-meta-photographer: rahim. S3 stores the names in lowercase and returns them on every GET and HEAD. User metadata is limited to 2 KB per object, and the whole PUT request header to 8 KB.

The important rule: metadata cannot be edited after upload. To change it, you copy the object onto itself (or to a new key) with the new metadata, which creates a new object, or a new version if versioning is on. The console’s “Edit metadata” does exactly that behind the scenes.

#Tags

Object tags are up to 10 key–value pairs per object, such as team=catalog and status=live. A tag key can be up to 128 characters and a value up to 256, and both are case sensitive. Unlike metadata, tags are stored beside the object and can be added, replaced or removed at any time without touching the data.

Tags are useful because other parts of AWS can act on them:

• Security: an IAM or bucket policy can allow access only to objects tagged classification=public (lesson 10.2).
• Lifecycle: a rule can move only objects tagged phototype=raw to Glacier after 90 days (lesson 10.5).
• Analysis and cost: Storage Class Analysis, CloudWatch metrics and S3 Batch Operations can filter or act on tags.

#Version ID

When versioning is enabled on a bucket (lesson 10.4), every write to a key gets a new, unique version ID, such as 3HL4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY, and older versions are kept. You can then read or restore any earlier version. Without versioning, objects have the version ID null, and a new PUT simply replaces the old object. Objects that existed before you enabled versioning also keep the version ID null.

#ETag and checksums

The ETag is a fingerprint of the object’s content that S3 returns on every PUT, GET and HEAD. For an object uploaded with one PUT and not encrypted with SSE-KMS, the ETag is the MD5 of the data, so a client can compare it with its own hash. For a multipart upload the ETag is computed from the parts and ends with -N, the number of parts, so it is not the MD5 of the whole file. Do not use it as a file hash in that case.

For real integrity checks, S3 supports additional checksums (CRC-64/NVME, CRC-32, CRC-32C, SHA-1, SHA-256). Since late 2024 the AWS SDKs and the CLI calculate a CRC checksum by default on upload, and S3 verifies it before it stores the object, so corruption on the way is caught.

#Uploading: one PUT or many parts

#One PUT, up to 5 GB

The simplest upload is a single PUT request with the whole file in its body. It is perfect for small objects: a product photo, a JSON document, a log file. It has two weaknesses for big files: one request is one stream, which rarely uses a fast link fully, and if the connection drops at 90%, S3 keeps nothing and the whole file must be sent again. A single PUT is also capped at 5 GB.

#Multipart upload

Multipart upload splits one object into parts, uploads them independently (usually in parallel), and joins them into one object at the end. The notes give the rule of thumb: recommended above 100 MB, mandatory above 5 GB.

1. Start

   The client sends a create-multipart-upload request with the key and any metadata, tags and storage class. S3 returns an upload ID that ties the parts together.

2. Upload the parts

   The client uploads parts, each with a part number from 1 to 10,000, in any order and in parallel. S3 answers each part with an ETag. A part that fails is simply uploaded again with the same number.

3. Complete

   The client sends the list of part numbers and their ETags. S3 joins the parts in part-number order into one object, which appears in the bucket only now. If the client instead aborts the upload, S3 deletes the parts.

The limits:

Why multipart helps:

• Speed: several parts in flight at once use more of your bandwidth (lesson 10.6 goes further, with Transfer Acceleration and byte-range downloads).
• Resilience: a network error costs one part, not the whole file. This is the notes’ exam trick: “uploads keep failing on an unreliable network” → multipart upload, because only the failed part is retried.
• Pause and resume: parts already uploaded stay stored until you complete or abort, so a laptop can continue tomorrow.
• Unknown size: you can start uploading a stream, such as a live video or a database dump being generated, before you know its final size.

You rarely call these steps by hand. aws s3 cp and the SDKs’ transfer managers switch to multipart automatically above a threshold (8 MB by default in the CLI) and choose part sizes for you. Multipart upload is also the first of the three application-level paths in lesson 10.12, which puts it next to pre-signed URLs and Batch Operations.

#The part maths

Two calculations come up again and again:

• How many parts? Divide the file size by the part size and round up. A 6 GB backup with 100 MiB parts: 6,000,000,000 ÷ 104,857,600 = 57.2, so 58 parts.
• What is the smallest part size I can use? Divide the file size by 10,000. A 1.2 TB scan archive needs parts of at least 1,200,000,000,000 ÷ 10,000 = 120,000,000 bytes, about 115 MiB. With 100 MiB parts it would need 11,445 parts, over the limit, so the upload would fail.

Try both in the planner, then make part 3 fail, and compare what a single PUT and a multipart upload have to send again.

#When an upload is never completed

If a client starts a multipart upload and then crashes or simply never sends “complete”, the parts stay in the bucket. They are invisible in normal listings and the console’s object list, but you pay for their storage until the upload is aborted. In a busy bucket with flaky clients this can quietly grow to terabytes. The fix is a lifecycle rule that aborts incomplete multipart uploads after a few days (lesson 10.5). You can see them with aws s3api list-multipart-uploads, and S3 Storage Lens reports them (lesson 10.7).

#Consistency: what a reader sees after a write

#Strong read-after-write

Since December 2020, S3 gives strong read-after-write consistency for every object operation, in every Region, automatically and at no extra cost:

• After a successful PUT of a new object, any GET returns it and any list includes it.
• After a successful overwrite, any GET returns the new data, never the old.
• After a successful DELETE, any GET returns 404 and any list leaves it out.
• Reads of metadata (HEAD), tags and ACLs are strongly consistent too.

“After” means: the read started after the write returned success. A read that overlaps a write may get the old or the new data, but never a mix: each update of a key is atomic.

#Two writers, one key

Strong consistency does not mean locking. If two clients PUT the same key at about the same time, S3 keeps the request with the latest timestamp (last writer wins), and the other update is lost without any error. Updates across several keys are not atomic either: S3 has no transactions.

Since 2024 S3 offers conditional writes on general purpose buckets:

• If-None-Match: * → write only if the key does not exist yet. Two clients creating orders/1001.json at once: one wins, the other gets 412 Precondition Failed instead of overwriting.
• If-Match: <etag> → write only if the object is still the version I read. This is optimistic locking: read, change, write back with the ETag, and retry if you get 412.

A bucket policy can even require these headers on every write. For anything that needs real transactions, such as stock counts or seat bookings, keep the state in a database (DynamoDB, RDS) and keep only the files in S3.

#What is still eventually consistent

• Bucket configuration. Turning on versioning, changing a policy, or deleting a bucket takes a short time to spread through S3. AWS recommends waiting about 15 minutes after enabling versioning before writing objects that must be versioned, and a deleted bucket may still appear in a bucket list for a while.
• Replication to another bucket is asynchronous (lesson 10.4): the copy in the other Region appears seconds to minutes later.

#What S3 costs

S3 has no upfront fee and no minimum. The bill has a few components, and the exam expects you to know which are free.

Prices change and differ by Region, so use the AWS Pricing Calculator for real numbers. What matters for design is the shape of the bill.

#What goes wrong

#S3 in system design

The pattern that appears in almost every architecture: store the bytes in S3, store the facts about them in a database. S3 is excellent at holding very large numbers of objects and serving them to many readers. It is not a database: it has no queries over object contents, no transactions and no partial updates.

• Photo or document app. The app saves products/shirt-42.jpg in S3 and a row in DynamoDB or RDS with the key, the owner, the price and the upload time. Pages read the database, then load images from S3 through CloudFront. Users can upload directly to S3 with a pre-signed URL so files never pass through the web servers (lessons 10.9 and 10.12).
• Log archive. Load balancers, CloudTrail and applications write to keys such as logs/app/2026/10/25/web-1.gz. Date-based prefixes make it cheap to process one day, and lifecycle rules move old logs to Glacier. A bank’s auditors can add Object Lock so logs cannot be altered (lesson 10.9).
• Data lake. Raw data lands in raw/, cleaned data in clean/, query-ready files in curated/, often partitioned in the key itself (trips/year=2026/month=10/day=25/). Athena, Spark and Redshift Spectrum read straight from S3, so storage and compute scale separately.
• Key design. Decide the key layout before loading data: there is no rename, prefixes drive lifecycle rules and permissions, and request rates scale per prefix (lesson 10.6). Avoid putting secrets or personal data in keys; they show up in logs and URLs.
• Large uploads. Clients upload big files with multipart upload, ideally directly to S3, and a lifecycle rule cleans up abandoned uploads.

#Exam patterns

#Try it yourself

These commands use the AWS CLI v2 with a profile allowed to use S3. Replace the bucket names with your own; the -example names here are placeholders.

Create a bucket in Mumbai, and check where it lives:

# Outside us-east-1 you must repeat the Region as a LocationConstraint
aws s3api create-bucket --bucket dhaka-shop-photos-example \
  --region ap-south-1 \
  --create-bucket-configuration LocationConstraint=ap-south-1

aws s3api get-bucket-location --bucket dhaka-shop-photos-example

# The same, in your account regional namespace (only your account can ever own this name)
aws s3api create-bucket --bucket shop-photos-111122223333-ap-south-1-an \
  --bucket-namespace account-regional \
  --region ap-south-1 \
  --create-bucket-configuration LocationConstraint=ap-south-1

Upload an object with a content type, user metadata and tags, then read its metadata without downloading it:

aws s3api put-object --bucket dhaka-shop-photos-example \
  --key photos/2026/eid/cake.jpg --body cake.jpg \
  --content-type image/jpeg \
  --metadata photographer=rahim,shoot=eid-2026 \
  --tagging 'team=catalog&status=live'

aws s3api head-object --bucket dhaka-shop-photos-example --key photos/2026/eid/cake.jpg
aws s3api get-object-tagging --bucket dhaka-shop-photos-example --key photos/2026/eid/cake.jpg

See the flat key list, and the folders the console would draw:

aws s3 ls s3://dhaka-shop-photos-example/ --recursive        # every key, no folders
aws s3 ls s3://dhaka-shop-photos-example/photos/             # PRE 2026/ = a common prefix
aws s3api list-objects-v2 --bucket dhaka-shop-photos-example \
  --prefix photos/ --delimiter /                             # CommonPrefixes in the JSON

Change metadata (there is no edit: copy the object onto itself) and change tags (in place):

aws s3api copy-object --bucket dhaka-shop-photos-example \
  --key photos/2026/eid/cake.jpg \
  --copy-source dhaka-shop-photos-example/photos/2026/eid/cake.jpg \
  --metadata-directive REPLACE \
  --content-type image/jpeg --metadata photographer=karim

aws s3api put-object-tagging --bucket dhaka-shop-photos-example \
  --key photos/2026/eid/cake.jpg \
  --tagging 'TagSet=[{Key=team,Value=catalog},{Key=status,Value=archived}]'

A multipart upload by hand, and the clean-up commands:

aws s3api create-multipart-upload --bucket dhaka-shop-photos-example \
  --key backups/db-2026-10-25.sql.gz                    # returns an UploadId

aws s3api upload-part --bucket dhaka-shop-photos-example \
  --key backups/db-2026-10-25.sql.gz --part-number 1 \
  --body part-001 --upload-id "$UPLOAD_ID"               # returns the part's ETag

aws s3api complete-multipart-upload --bucket dhaka-shop-photos-example \
  --key backups/db-2026-10-25.sql.gz --upload-id "$UPLOAD_ID" \
  --multipart-upload file://parts.json                   # {"Parts":[{"PartNumber":1,"ETag":"..."}]}

aws s3api list-multipart-uploads --bucket dhaka-shop-photos-example   # find abandoned ones
aws s3api abort-multipart-upload --bucket dhaka-shop-photos-example \
  --key backups/db-2026-10-25.sql.gz --upload-id "$UPLOAD_ID"

The high-level command does all of that for you, in parallel, and you can tune when it switches to multipart:

aws configure set default.s3.multipart_threshold 64MB
aws configure set default.s3.multipart_chunksize 16MB
aws s3 cp ./db-2026-10-25.sql.gz s3://dhaka-shop-photos-example/backups/

Create an object only if the key does not exist yet (a conditional write):

aws s3api put-object --bucket dhaka-shop-photos-example \
  --key orders/1001.json --body 1001.json --if-none-match '*'
# Run it twice: the second call fails with 412 Precondition Failed

Clean up when you are done (empty, then delete; remember that a deleted name can be taken by someone else):

aws s3 rm s3://dhaka-shop-photos-example --recursive
aws s3api delete-bucket --bucket dhaka-shop-photos-example --region ap-south-1

#Summary

Key takeaways

• S3 is object storage: whole objects over HTTP, no capacity to provision, eleven nines of durability. Use it for media, backups, archives, logs, data lakes, downloads and static sites.
• A bucket lives in one Region, but its name is unique across all accounts and Regions of the partition. The console looks global; buckets are regional. Names and Regions cannot change.
• Naming: 3–63 characters, lowercase letters, digits, dots and hyphens, start and end with a letter or digit, no .., not an IP address, no xn-- / sthree- / amzn-s3-demo- prefixes, no -s3alias / --ol-s3 / .mrap / --x-s3 / --table-s3 suffixes. Uppercase or underscore = wrong option. Avoid dots.
• Account regional namespace (2026): names ending in -<account>-<region>-an that only your account can ever create. Do not delete buckets that code still references.
• An object = key + data + metadata + tags (+ version ID). The key is the full name, up to 1,024 bytes, case sensitive. No folders: prefixes plus the / delimiter; Create folder makes a zero-byte object; there is no rename.
• Sizes: object 0 B to 50 TB (older figure 5 TB); single PUT ≤ 5 GB; multipart recommended > 100 MB, required > 5 GB; 10,000 parts of 5 MiB–5 GiB. Part size ≥ file ÷ 10,000.
• Multipart = parallel speed, retry only the failed part, resume later. Abort or lifecycle-clean incomplete uploads, because their parts are billed.
• System metadata (S3’s), user metadata (x-amz-meta-*, 2 KB, fixed at upload; change = copy), tags (10 pairs, changeable, used by policies and lifecycle), version ID (null without versioning). Single-part ETag = MD5; multipart ETag ends in -N.
• Strong read-after-write consistency for PUT, overwrite, DELETE and list since Dec 2020; last writer wins with no locking; conditional writes (If-None-Match, If-Match) prevent lost updates; bucket configuration is eventually consistent.
• Costs: storage GB-month, requests, data transfer out, retrieval and management features. Transfer in is free, and so is transfer to CloudFront or to services in the same Region.

Self-check: Buckets, Objects and Keys31 questions