Paths to S3 (3/3): Multipart, Pre-signed URLs and Batch Operations

By the end of this lesson you will be able to

• Tell apart the three paths that start inside your own application: multipart upload, pre-signed URLs and S3 Batch Operations, and say which problem each one solves.
• Walk through a multipart upload call by call, calculate part sizes and part counts within the limits, and stop abandoned uploads from costing money.
• Explain whose permissions a pre-signed URL carries, how long it can live with each kind of credential, and why it fails before its expiry; design a browser upload that never passes through your servers.
• Build a Batch Operations job from a manifest, an operation, an IAM role and a report, and handle a job that finishes with some failures.
• Choose among all 13 paths to S3 from where the data starts, how much there is, how often it moves and what matters most, and recognise each one from its exam keywords.

#Three paths that start in your application

#Where 11–13 fit among the 13 paths

This is the last of three lessons that collect every way data travels into, out of or inside S3, so you can stop mixing them up:

• Paths 1–5 (lesson 10.10) are network routes: how packets from a VPC or a data centre reach S3 privately. Gateway endpoint, interface endpoint, VPC Peering, Transit Gateway, and Site-to-Site VPN or Direct Connect.
• Paths 6–10 (lesson 10.11) are transfer services: tools that move large amounts of data for you. S3 File Gateway, DataSync, Snowball, Transfer Acceleration and Transfer Family.
• Paths 11–13 (this lesson) are things your own code does with the S3 API. Nothing to deploy, no network change:
  • 11. Multipart upload: how one big object gets in, in pieces.
  • 12. Pre-signed URL: how someone without AWS credentials gets one object in or out.
  • 13. S3 Batch Operations: how millions of objects already in S3 get changed in one go.

The three answer three different questions, and the exam keeps them apart by keyword: "large file, unreliable network" is 11, "users upload directly from the browser" is 12, "millions of existing objects" is 13. They also combine with the other ten: a multipart upload can travel through a gateway endpoint, each of its parts can be pre-signed, and its traffic can be accelerated.

Some of this is taught from another angle elsewhere in the chapter: multipart for speed in 10.6, Batch Operations among the bucket tools in 10.7, pre-signed URLs as a security control in 10.9. This lesson is complete on its own, from the "path" point of view: where the bytes start, who sends them, and why you would pick this path over the others.

#Path 11: multipart upload

#Multipart: what it is and why it exists

A normal upload is one PutObject request: one HTTP request carrying the whole object. That has two hard problems for big files:

1. Size limit. A single PUT can carry at most 5 GB.
2. All or nothing. If the connection drops at 95% of a 4 GB upload, the request fails and you start again from zero. On a phone or a long intercontinental link, a big single request may never finish.

Multipart upload splits one object into parts that are uploaded separately, in any order and in parallel, and then joined by S3 into a single object. A failed part is sent again on its own. The owner’s notes put the rule simply, and AWS agrees: consider multipart from about 100 MB, and you must use it above 5 GB. The largest object S3 stores is 5 TB in the exam (see the side note below for the 2025 change).

#How it works, step by step

1. 1. CreateMultipartUpload

   You tell S3 the bucket, the key and the object’s settings (storage class, encryption, metadata, tags). S3 returns an UploadId, which every later call must quote.

2. 2. UploadPart, many times, in parallel

   Each request carries one part with a part number from 1 to 10,000. S3 stores the part and returns its ETag; your code keeps a list of part numbers and ETags. Parts can arrive in any order, from several threads or even several machines. Uploading the same part number again simply replaces that part, which is how a retry works.

3. 3. CompleteMultipartUpload

   You send the list of part numbers with their ETags. S3 assembles the parts in part-number order and the object appears atomically: before this call nobody can read it, after it everybody can. The object’s ETag ends in -N, the number of parts, which is why the ETag of a multipart object is not the MD5 of the file.

4. Or: AbortMultipartUpload

   If you give up, abort with the UploadId. S3 deletes the stored parts and you stop paying for them. ListParts shows the parts of one upload, and ListMultipartUploads shows every upload in progress in a bucket.

You rarely write these calls by hand. The AWS CLI (aws s3 cp) and the SDKs’ transfer managers switch to multipart automatically above a threshold (8 MB by default in the CLI), choose a part size, run several parts at once (10 by default in the CLI) and retry failed parts.

#Part limits and the arithmetic

The 10,000-part limit decides the smallest part size you can use for a given object:

• An 8 GB video with 8 MiB parts: 8,192 MiB ÷ 8 MiB = 1,024 parts. Fine.
• A 100 GiB database dump with the 5 MiB minimum would need 20,480 parts, too many. The part size must be at least 100 GiB ÷ 10,000 ≈ 10.24 MiB, so you would choose 16 MiB (6,400 parts).
• A 1 TiB file needs parts of at least 1,048,576 MiB ÷ 10,000 ≈ 105 MiB.

The CLI and SDKs raise the part size automatically when a file would exceed 10,000 parts; your own code must do the same.

#Retrying one part, and the parallel speed-up

Two different benefits come from the same design:

• Resilience. A broken connection costs one part, not the whole object. With 16 MiB parts on a flaky mobile link, a drop wastes at most 16 MiB. This is the owner’s exam trick: "upload fails because of network issues" → multipart, so only the failed part is uploaded again. An upload can even be paused for days and resumed, because the stored parts wait in S3 until you complete or abort.
• Throughput. One TCP stream rarely fills a fast link, especially over long distances. Ten parts in flight at once fill it far better. Lesson 10.6 covers this angle, along with byte-range fetches, which do the same trick for downloads.

Smaller parts mean cheaper retries but more requests (each part is one PUT request on the bill); larger parts mean fewer requests but more to resend. Between 8 and 100 MiB is a common choice.

#Abandoned uploads still cost money

Parts of an upload that was never completed or aborted are stored and billed at the storage class you chose, but they are not objects: a bucket listing does not show them, and the console’s object list looks empty. A mobile app that loses a thousand uploads a day quietly builds up terabytes of invisible, billed parts.

The fix is a lifecycle rule on every bucket that receives multipart uploads: AbortIncompleteMultipartUpload after a number of days since the upload started (7 is common). S3 then aborts the stale uploads and deletes their parts. Lesson 10.5 covers lifecycle rules; S3 Storage Lens (10.7) shows how many bytes of incomplete uploads each bucket holds.

#Why multipart and not the alternatives

#Path 12: pre-signed URLs

#Pre-signed URL: what it is and why it exists

Your bucket is private, as it should be. Now a customer must download one invoice, or a seller must upload one product photo. Neither has AWS credentials, and you do not want to make the bucket public or create an IAM user for every customer.

A pre-signed URL is a normal S3 URL with a signature in its query string. Someone who has permission (your backend) signs a request for one object, one method, until one time; anyone who holds the URL can then perform exactly that request, without AWS credentials, until it expires. It is a permission slip: "the bearer may PUT sellers/42/photo.jpg until 10:05, signed by the shop’s backend."

#Pre-signed GET and PUT

• GET (download): "let this logged-in student watch this paid video for an hour", "let this customer download this invoice". The URL can be pasted into a browser, an <img> tag or a video player.
• PUT (upload): "let this seller upload one photo to exactly this key". The browser sends the file in the body of an HTTP PUT. If an object with the same key exists, it is replaced, so let the backend choose unique keys (a UUID per upload).
• Other methods can be signed too (HEAD to read metadata, UploadPart for each part of a multipart upload, DELETE).

You can create pre-signed URLs from the S3 console (share an object), the AWS CLI (aws s3 presign, which signs GET only) and the SDKs (any method, including PUT). Creating one is local maths: the SDK computes an HMAC with the signer’s secret key and makes no call to AWS. That is why you can "successfully" create a URL for an object that does not exist, or with credentials that have no permission. The mistake shows up only when the URL is used.

#Whose permissions the URL carries

When S3 receives a pre-signed request, it treats it as a request made by the signer, with the signer’s permissions at that moment. The owner’s notes say the same: the user inherits the permissions of the URL’s creator. Three consequences:

1. The signer must be allowed to do the action. A backend role without s3:PutObject on that key signs URLs that always fail with 403.
2. Revoking the signer revokes the URLs. Remove the permission, delete or deactivate the IAM user’s access key, or let a role session end, and every URL it signed stops working, even before its own expiry.
3. It is a bearer token. Anyone who gets the URL can use it, as many times as they like, until it expires. Treat it like a password: short expiries, HTTPS only, never logged.

Block Public Access does not affect pre-signed URLs, because the request is signed: S3 sees an authenticated request from the signer, not an anonymous one. Bucket policies can still restrict them, for example deny when s3:signatureAge is too old, or when the request does not come from your network (aws:SourceIp, or aws:SourceVpce through an endpoint).

#How long a URL can live

S3 checks the expiry when a request starts. A 2 GB download that began one second before the expiry finishes; if the connection drops and the player reconnects after the expiry, it fails.

#Uploading from the browser straight to S3

The pattern every photo, document and video site uses:

1. The browser asks your backend for an upload URL, sending only the file name, type and size.
2. The backend checks the user, decides the key (sellers/42/2f1c….jpg), signs a PUT URL valid for a few minutes, and returns it.
3. The browser sends the file straight to S3 with that URL. The bucket needs a CORS rule allowing PUT from your site’s origin (lesson 10.3).
4. S3 stores the object; an S3 event (lesson 10.7) starts thumbnails or virus scanning.

Your servers carry a few hundred bytes of JSON instead of the file. They no longer need bandwidth, memory or time for uploads, and S3 absorbs any spike. For big files, combine it with path 11: the backend creates the multipart upload, pre-signs each UploadPart, and completes it when the browser reports the ETags.

#Pre-signed POST: an HTML form with rules

A pre-signed PUT URL fixes the key and the method, but it cannot say "at most 10 MB" or "only images". For that, S3 offers pre-signed POST: the backend signs a policy document with conditions, and the browser uploads with an ordinary HTML form (multipart/form-data). Typical conditions:

• content-length-range from 1 byte to 10 MB;
• the key must start with sellers/42/;
• Content-Type must start with image/;
• an expiry for the policy.

S3 rejects any upload that breaks a condition, before storing it. Use POST when uploaders are untrusted and you must cap size and type; PUT is simpler when your own app code controls the file.

#Why a pre-signed URL and not the alternatives

#Path 13: S3 Batch Operations

#Batch Operations: what it is and why it exists

Paths 1–12 bring data into S3 (or out of it). Path 13 works on data that is already there. Sooner or later every large bucket needs a change applied to every object: new tags for cost allocation, encryption for objects written years ago, a copy to a new account, a restore from Glacier, a custom process on each file.

You could write a script: list the bucket, loop over the keys, call the API, handle throttling, retry failures, remember where you stopped when the laptop sleeps, and produce a list of what failed. For 300 million objects that script runs for days and fails in creative ways. S3 Batch Operations does the same work as a managed service: one request starts a job that runs one operation on every object in a list, with retries, progress tracking, notifications and a completion report. The owner’s exam trick: "perform an action on millions of existing objects", "change tags for billions of files" → Batch Operations, not a custom script or Lambda loop.

#The job: manifest, operation, role, report

A job has four main parts:

1. Manifest: the list of objects. Three ways to give it:
   • an S3 Inventory report (CSV, via its manifest.json); Inventory lists every object daily or weekly, and you can query it with Athena (chapter 17) to filter, for example "all objects over 1 MB without encryption";
   • a CSV file you write: bucket,key or bucket,key,versionId per line, keys URL-encoded;
   • a generated list: the job builds it from filters (prefix or suffix, creation date, size, storage class, encryption type) and can save it as a manifest.
2. Operation: exactly one per job, with its parameters (the tag set, the destination bucket, the restore tier…).
3. IAM role: the job runs as this role. It trusts batchoperations.s3.amazonaws.com and needs the operation’s permissions (for a copy: read on the source, write on the destination), plus read on the manifest and write on the report bucket.
4. Completion report (optional, recommended): written to a bucket you choose, for all tasks or failed tasks only, with each key’s status and error code.

Add a priority (higher numbers run first among your jobs in that Region) and, for jobs created in the console, a confirmation step: the job waits in Suspended until you confirm it.

#The operations it can run

#Job lifecycle, retries and partial failures

A job moves through New → Preparing → (Suspended →) Ready → Active → Complete, or ends in Cancelled or Failed. A higher-priority job can pause a lower one.

• Tasks fail individually. An object deleted since the inventory was taken, an object over 5 GB in a copy, a KMS key the role cannot use: each is one failed task, recorded in the report, and the job carries on. Temporary errors such as throttling are retried automatically.
• The job fails as a whole only if, after at least 1,000 tasks, more than 50% have failed. That usually means a wrong role, a wrong bucket or a wrong manifest, and stopping early saves you money.
• Not a snapshot. The job acts on the latest version of each key when it gets to it. If objects can change during a long job, put version IDs in the manifest.
• Fixing failures. The report lists every failed key with its error. Feed that list back as the manifest of a small second job.
• Cost. About $0.25 per job plus $1.00 per million objects processed, on top of the normal request charges of the operation (and Lambda charges if you invoke a function). 300 million objects ≈ $300 plus requests.

#Why Batch Operations and not the alternatives

#The three side by side

#Choosing among all 13 paths

#The comparison table

Read the table from left to right as questions: where is the data, does it have to stay private, how much and how often, and what speaks the protocol at the other end. Lessons 10.10 and 10.11 teach paths 1–10 in detail; chapters 12 (hybrid storage), 22 (advanced networking) and 23 (migration) go deeper still.

#The 13-path chooser

#Keywords that give the answer away

• "from a private subnet without NAT", "without traversing the internet", same Region, cheapest → gateway endpoint (1).
• "from on-premises over Direct Connect, privately", "private IP for S3" → interface endpoint (2) behind VPN / Direct Connect (5).
• "many VPCs share one endpoint" → interface endpoint in a hub VPC + Transit Gateway (4), or peering (3) for two VPCs.
• "applications must keep using NFS or SMB shares" → S3 File Gateway (6).
• "migrate file servers online", "scheduled copy", "keep permissions and timestamps" → DataSync (7).
• "petabytes", "limited bandwidth", "would take weeks", "no connectivity" → Snowball (8).
• "users around the world upload to one bucket slowly" → Transfer Acceleration (9).
• "partners use SFTP", "AS2", "cannot change their clients" → Transfer Family (10).
• "large file", "network interruptions", "over 5 GB" → multipart upload (11).
• "users upload directly from the browser", "temporary access to a private object" → pre-signed URL (12).
• "millions of existing objects", "retag / re-encrypt / restore in bulk" → Batch Operations (13).

#Paths 11–13 in system design

• Never let your app servers carry user files. Browser → pre-signed URL → S3 → event → processing is the standard upload pipeline. It removes upload bandwidth from your scaling problem and keeps the bucket private.
• Combine paths, do not choose just one. A global video app uses pre-signed multipart parts (12 + 11) with Transfer Acceleration (9); a backend in a private subnet sends its own multipart uploads through a gateway endpoint (11 + 1).
• Put an abort rule on every upload bucket. It costs nothing and prevents a silent, growing bill.
• Keep pre-signed URLs short and signed on demand, by a role with the narrowest permission (only s3:PutObject on uploads/*), and add s3:signatureAge or network conditions in the bucket policy for sensitive data.
• Plan bulk changes as jobs, not scripts. Turn on S3 Inventory before you need it; filter it with Athena; run Batch Operations with a report; re-run the failed list. Test on a small manifest first; in the console the confirmation step is your last chance.
• Use version IDs in manifests when objects can change while a long job runs.

#Try it yourself

Multipart by hand (the CLI does this for you with aws s3 cp):

# 0. Make a 40 MB test file and split it into 8 MiB parts (the last part is smaller)
dd if=/dev/urandom of=big.bin bs=1M count=40
split -b 8M big.bin part-

# 1. Start the upload and keep the UploadId
UPLOAD_ID=$(aws s3api create-multipart-upload --bucket uploads-example \
  --key videos/big.bin --query UploadId --output text)

# 2. Upload the parts (each returns an ETag)
aws s3api upload-part --bucket uploads-example --key videos/big.bin \
  --part-number 1 --body part-aa --upload-id "$UPLOAD_ID"
# ...parts 2 to 5 the same way, then list what S3 has:
aws s3api list-parts --bucket uploads-example --key videos/big.bin --upload-id "$UPLOAD_ID"

# 3. Complete with the part numbers and ETags (parts.json: {"Parts":[{"PartNumber":1,"ETag":"..."},...]})
aws s3api complete-multipart-upload --bucket uploads-example --key videos/big.bin \
  --upload-id "$UPLOAD_ID" --multipart-upload file://parts.json

# Find and abort forgotten uploads
aws s3api list-multipart-uploads --bucket uploads-example
aws s3api abort-multipart-upload --bucket uploads-example --key videos/old.bin --upload-id EXAMPLEID

# The safety net: abort incomplete uploads after 7 days
aws s3api put-bucket-lifecycle-configuration --bucket uploads-example --lifecycle-configuration \
  '{"Rules":[{"ID":"abort-mpu","Status":"Enabled","Filter":{},"AbortIncompleteMultipartUpload":{"DaysAfterInitiation":7}}]}'

# Let the high-level CLI choose parts and run them in parallel
aws configure set default.s3.multipart_chunksize 16MB
aws configure set default.s3.max_concurrent_requests 10
aws s3 cp ./trip-video.mp4 s3://uploads-example/videos/

Pre-signed URLs:

# A GET URL valid for 10 minutes (the CLI signs GET only)
aws s3 presign s3://invoices-example/2026/10/inv-1042.pdf --expires-in 600
curl -o inv.pdf "<the URL>"          # works without AWS credentials
sleep 601; curl -I "<the URL>"       # 403 after expiry

# PUT and POST need an SDK (boto3 shown)
import boto3, uuid
s3 = boto3.client("s3")
key = f"sellers/42/{uuid.uuid4()}.jpg"

put_url = s3.generate_presigned_url(
    "put_object",
    Params={"Bucket": "photos-example", "Key": key, "ContentType": "image/jpeg"},
    ExpiresIn=300,
)
# Browser: fetch(put_url, {method: "PUT", headers: {"Content-Type": "image/jpeg"}, body: file})

post = s3.generate_presigned_post(
    "photos-example", key,
    Fields={"Content-Type": "image/jpeg"},
    Conditions=[["content-length-range", 1, 10 * 1024 * 1024],
                ["starts-with", "$Content-Type", "image/"]],
    ExpiresIn=600,
)
# post["url"] and post["fields"] go into an HTML form with the file input last

Batch Operations (S3 Control API):

# A role Batch Operations can assume
aws iam create-role --role-name s3-batch-tagging --assume-role-policy-document \
  '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"batchoperations.s3.amazonaws.com"},"Action":"sts:AssumeRole"}]}'

# A job that tags every object listed in a CSV manifest, with a report of failures
aws s3control create-job --account-id 111122223333 --region us-east-1 \
  --operation '{"S3PutObjectTagging":{"TagSet":[{"Key":"desk","Value":"sports"}]}}' \
  --manifest '{"Spec":{"Format":"S3BatchOperations_CSV_20180820","Fields":["Bucket","Key"]},"Location":{"ObjectArn":"arn:aws:s3:::manifests-example/sports.csv","ETag":"EXAMPLEETAG"}}' \
  --report '{"Bucket":"arn:aws:s3:::reports-example","Prefix":"batch","Format":"Report_CSV_20180820","Enabled":true,"ReportScope":"FailedTasksOnly"}' \
  --priority 10 --role-arn arn:aws:iam::111122223333:role/s3-batch-tagging \
  --no-confirmation-required

aws s3control describe-job --account-id 111122223333 --job-id <job-id>
aws s3control list-jobs --account-id 111122223333 --job-statuses Active Complete

#Summary

Key takeaways

• Paths 11–13 are things your code does with the S3 API: multipart for one big object, pre-signed URLs for people without credentials, Batch Operations for millions of objects already in S3. They combine with paths 1–10.
• Multipart: CreateMultipartUpload → UploadPart (in parallel, any order, ETag each) → CompleteMultipartUpload (atomic) or Abort. Parts 5 MiB–5 GiB (last smaller), 10,000 parts, 5 TB object (50 TB since Dec 2025). Recommended over 100 MB, required over 5 GB. A failed part is retried alone.
• Incomplete uploads are invisible but billed: add a lifecycle rule AbortIncompleteMultipartUpload. SSE-KMS multipart needs kms:Decrypt too.
• Pre-signed URL: one object, one method, one expiry; acts as the signer, with the signer’s permissions at use time. Console 1 min–12 h; CLI default 1 h, max 7 days with IAM user keys; with role credentials it dies with the session. The CLI presigns GET only; PUT and POST need an SDK.
• Browser → backend signs → browser PUTs straight to S3 (CORS on the bucket) → event. Pre-signed POST adds size, type and key-prefix conditions.
• Batch Operations job = manifest (Inventory, CSV or generated) + one operation + IAM role + report + priority. Operations: copy, Lambda, tags, ACL, restore, Object Lock, replicate, checksums, update encryption. Retries, partial failures in the report, job fails if >50% fail after 1,000 tasks. ~$0.25/job + $1 per million objects.
• Choosing among 13: start from where the data is, then private or not, how much, how often, and who speaks which protocol.

Self-check: multipart, pre-signed URLs, Batch Operations and the 13 paths30 questions