Versioning and Replication

By the end of this lesson you will be able to

• Explain what S3 Versioning protects against, the three versioning states of a bucket, and why “disable” is not one of them.
• Predict what PUT, GET and DELETE do in an enabled and in a suspended bucket: version IDs, the null version, delete markers, 404 versus 405, and permanent deletes.
• Restore a deleted or overwritten object, and estimate what keeping versions costs.
• Say what MFA Delete adds on top of versioning, who can turn it on, and when Object Lock is the better answer.
• Set up Cross-Region and Same-Region Replication: the requirements, the IAM role, cross-account copies and what a rule can filter and change.
• State exactly what is and is not replicated (existing objects, delete markers, version deletes, encrypted and archived objects, replicas), and fix each gap with the right feature: Batch Replication, delete marker replication, KMS settings, a second rule or RTC.

#Why keep old copies of objects?

#Overwrites and deletes are instant and final

In lesson 10.1 you saw that an S3 object is identified by its key. Upload a new file with the same key and the old one is replaced. Delete the key and it is gone. In a bucket without versioning both happen in milliseconds, and there is no recycle bin.

That is a problem in real life, because most data loss is not a disk failure (S3 is built for eleven nines of durability) but a person or a program doing the wrong thing:

• a deployment script uploads an empty catalog.json over the real one;
• an intern runs aws s3 rm --recursive on the wrong prefix;
• ransomware with leaked access keys overwrites every file with encrypted junk;
• a bug in an image resizer writes a corrupt thumbnail over the original photo.

S3 Versioning fixes this by keeping every version of every object. An overwrite adds a new version on top; a delete adds a marker on top. The old data stays underneath until you remove it on purpose.

#Versioning is a bucket setting

Versioning is switched on for a whole bucket, not for one object or one prefix. Once it is on, it applies to every key. The slides call it best practice, and for any bucket that holds data you cannot easily recreate (customer uploads, documents, configuration, backups) it is.

#How versions work

#The three states of a bucket

A bucket is always in one of three states:

The notes stress the key rule: once versioning has been enabled, it can never be fully disabled. A bucket can only move between enabled and suspended. There is no way back to unversioned; if you truly need that, you create a new bucket and copy the current objects into it.

#Every overwrite creates a new version

With versioning enabled:

1. Each PUT, POST or copy to a key creates a new version with its own version ID, returned in the x-amz-version-id response header.
2. The newest version is the current version. All older ones are noncurrent versions.
3. A plain GET (no version ID) returns the current version. GET … ?versionId=… returns any stored version.
4. Versions are never changed in place. Even “rolling back” works by copying an old version on top, which creates yet another version.

The slides draw versions as 1, 2, 3. That is only a picture. Real version IDs are opaque strings of up to 1,024 bytes, such as 3HL4kqtJlcpXroDTDmJ.rmSpXd3dIbrHY, generated by S3. You cannot choose them, and you cannot sort them by reading them; S3 lists versions newest first.

#Objects from before versioning have the version ID null

If a bucket already held objects when you enabled versioning, those objects are not rewritten. They keep the version ID null. When you next overwrite one of them, the old data stays as the noncurrent null version and the new upload gets a real ID. The slides make the same point: any file that was not versioned before enabling versioning has version null.

#What versioning costs

S3 bills every version as a full object. Versions are not stored as differences. A 5 MB report overwritten every hour for 30 days leaves 720 versions, about 3.5 GB, all billed at the storage class price. Add the PUT requests, and versioning on a busy key can cost far more than the current data.

The answer is not to avoid versioning but to manage it with lifecycle rules for noncurrent versions (lesson 10.5): for example, move noncurrent versions to a cheaper class after 30 days and delete them after 90, or keep only the last five. Lifecycle can also remove expired object delete markers (a marker with no versions left under it).

Try every action yourself. Click a row in the stack to pick a version for the version-specific buttons.

#Deleting in a versioned bucket

#A simple delete adds a delete marker

A DELETE without a version ID (what the console’s Delete button, aws s3 rm and most SDK calls send) does not delete anything in a versioned bucket. S3 creates a delete marker: a placeholder with the object’s key and its own version ID, and no data. The marker becomes the current version. From then on:

• a plain GET or HEAD returns 404 Not Found with the header x-amz-delete-marker: true;
• a normal listing (ListObjectsV2, aws s3 ls) does not show the key;
• the versions underneath are untouched and still readable by version ID, and still billed.

A delete marker costs almost nothing: its size is the length of its key name, stored in S3 Standard. Deleting a key whose current version is already a marker simply adds another marker.

#Restore by deleting the marker

To undelete, remove the marker itself: list the versions (aws s3api list-object-versions, or Show versions in the console), then delete the marker by its version ID. The version below becomes current again, with no copy and no new version. The notes put it simply: to restore, delete the delete marker.

To restore an overwritten object, there is no marker to remove. Copy the good version on top (copy-object with ?versionId= in the source), or delete the bad newer versions by ID.

#A delete with a version ID is permanent

A DELETE with a version ID removes exactly that version, permanently. No marker is added, and nothing can bring it back. This is the only way to really free storage in a versioned bucket (lifecycle expiration of noncurrent versions does the same thing automatically).

Two status codes are easy to mix up:

Because permanent deletes are destructive, they use a different permission, s3:DeleteObjectVersion, separate from s3:DeleteObject. Giving applications only s3:DeleteObject means they can hide objects but never destroy history.

#Suspending versioning

Suspending stops new versions from piling up, but it keeps every version that already exists (the slides stress this too). In a suspended bucket:

• PUT stores the object with version ID null. If a null version already exists for that key, it is replaced and its data is lost. Versions with real IDs are untouched.
• DELETE without an ID removes the null version (if any) and adds a delete marker whose version ID is null.
• GET still returns the current version, whatever its ID, and any old version is still readable by ID.

So in a suspended bucket each key behaves like an unversioned object on top of a frozen history. Suspending does not save money on existing versions; only deleting them (by hand or with lifecycle) does.

You cannot suspend versioning on a bucket that is the source of a replication rule: S3 returns an error until you remove the replication configuration. Suspending it on a destination makes replication fail.

#Protecting versions: MFA Delete, and beyond

Versioning protects against mistakes made through simple deletes and overwrites. It does not stop an identity that is allowed to delete versions: a person with s3:DeleteObjectVersion, or an attacker with their keys, can delete every version one by one. The notes’ exam trick pairs versioning with MFA Delete for exactly this reason.

A preview (lesson 2.3 introduced it; lesson 10.9 covers it in full):

• With MFA Delete enabled, two actions require an MFA code from the root user’s MFA device in the request: permanently deleting an object version and changing the versioning state (suspending).
• Enabling versioning, listing versions and simple deletes (markers) do not need MFA.
• Only the root user of the bucket owner’s account can enable or disable it, and only through the CLI, SDK or API, never in the console. Versioning must be enabled.

#Replication: automatic copies in another bucket

#CRR and SRR

S3 Replication copies objects from a source bucket to one or more destination buckets, automatically and asynchronously, after they are written. There are two kinds, the same feature with a different destination Region:

• Cross-Region Replication (CRR): the destination is in another Region.
• Same-Region Replication (SRR): the destination is in the same Region.

The buckets can be in the same or different AWS accounts.

#What you need

The slides and notes list the essentials; here is the full set:

1. Versioning enabled on the source and every destination. Replication works on versions. (And while replication is configured you cannot suspend versioning on the source.)
2. An IAM role that S3 assumes (its trust policy names the service s3.amazonaws.com). It needs permission to read versions from the source (s3:GetReplicationConfiguration, s3:ListBucket, s3:GetObjectVersionForReplication, s3:GetObjectVersionAcl, s3:GetObjectVersionTagging) and to write into the destinations (s3:ReplicateObject, s3:ReplicateDelete, s3:ReplicateTags). Whoever creates the rule needs iam:PassRole for it (roles were lesson 2.4).
3. A replication configuration on the source bucket: the role plus one or more rules.
4. Cross-account only: the destination account adds a bucket policy that allows the source account’s role to replicate into it. Optionally the rule changes the replica owner to the destination account (with Object Ownership set to bucket owner enforced, lesson 10.2, the destination owns replicas anyway). The destination cannot be a Requester Pays bucket.
5. If the source has Object Lock, the destinations must have it too.

Each rule chooses:

• a scope: the whole bucket, a prefix, tags, or a prefix with tags;
• one destination bucket (several rules can point to several destinations, so one source can fan out);
• optionally a different storage class for the replicas (for example Standard-IA or Glacier Deep Archive for a cheap backup copy);
• a priority, used when rules overlap;
• options covered below: delete marker replication, KMS-encrypted objects, replica modification sync, Replication Time Control.

#What a replica looks like

A replica has the same key, version ID, metadata, tags, ACL and Object Lock retention as the source version. Each object reports a replication status in head-object: PENDING or COMPLETED (or FAILED) on the source, and REPLICA on the copy. Most objects replicate within seconds to minutes; large objects can take longer. A lifecycle rule in the destination counts age from the original creation time, not from when the replica arrived.

#Only new objects: Batch Replication for the rest

The notes and slides agree: after you enable replication, only new objects are replicated. Objects that were already in the bucket stay where they are.

To copy them, use S3 Batch Replication: an S3 Batch Operations job (lesson 10.7) that replicates objects using your existing rules. The console offers to create it as soon as you save a new rule. Batch Replication copies:

• existing objects written before the rule;
• objects whose replication failed (fix the cause first, for example a missing KMS permission);
• objects that were already replicated to another destination, when you add or change a destination;
• objects that are replicas created by another rule.

The notes’ exam trick: to sync old data, always look for S3 Batch Replication. Batch Operations and the other ways of moving data in bulk are also compared in lesson 10.12.

#Deletes: what reaches the destination

The notes say delete markers are not replicated. That is the default for current rules, but it is an option, and the slides list both cases:

• Delete markers (simple deletes): not replicated unless you enable delete marker replication on the rule. It cannot be used with tag-based rules. Old “V1” configurations without a Filter element do replicate user delete markers.
• Deletes with a version ID (permanent deletes): never replicated, whatever the settings. A malicious or mistaken permanent delete in the source cannot reach the copy.
• Lifecycle actions: never replicated. Configure lifecycle on each bucket separately.

Choose delete marker replication by purpose. For an active-active pair where users read from both Regions, turn it on so a deleted object disappears everywhere. For a backup copy, leave it off, so an accidental delete in the source is not mirrored.

#No chaining

Replication is not transitive. If bucket A replicates to B and bucket B replicates to C, objects created in A reach B but never C, because by default S3 does not replicate objects that are themselves replicas.

The notes’ exam trick gives the fix: if A’s data must reach both B and C, create two rules on A, A → B and A → C. Two details complete the picture:

• The B → C rule is not broken: objects uploaded directly to B do replicate to C.
• To copy replicas that already sit in B, Batch Replication can replicate replicas.

#Encrypted objects

KMS keys are regional, so a source-Region key cannot encrypt replicas in another Region. Cross-account SSE-KMS replication needs a customer managed key whose key policy lets the source account use it; AWS managed keys cannot be shared. Every replicated SSE-KMS object also uses KMS requests on both sides, so watch the KMS request quota: replicating 1,000 objects per second uses about 2,000 KMS requests per second.

If the destination bucket has default encryption and the source object is unencrypted, the replica is encrypted with the destination’s default, and its ETag differs from the source’s.

#What else is not replicated

• Objects in Glacier Flexible Retrieval, Glacier Deep Archive and the Intelligent-Tiering archive tiers. Replicate first, archive later (lesson 10.5).
• Bucket-level settings: bucket policies, lifecycle, notifications, CORS. Each bucket keeps its own.
• Objects the bucket owner cannot read (objects owned by another account without ACL permission; rare now that ACLs are disabled by default).
• With a tag-based rule, objects tagged after the upload (Batch Replication copies them).

Use the checker to test any combination:

#Replication Time Control

Plain replication has no time guarantee: most objects arrive in seconds, but a backlog or a large object can take hours. S3 Replication Time Control (RTC) is an option on a rule that:

• is designed to replicate 99.99% of new objects within 15 minutes, most within seconds, backed by an SLA (service credits if fewer than 99.9% of objects replicate within 15 minutes in a billing month);
• turns on replication metrics in CloudWatch: bytes and operations pending, maximum replication latency, failed operations (you can also enable metrics without RTC);
• sends S3 event notifications when an object misses the 15-minute threshold and when it replicates late, to SNS, SQS or Lambda.

RTC adds a per-GB fee. The SLA does not apply while you exceed the S3 request-rate guidelines or the default 1 Gbps replication transfer quota. The exam signal is any time limit on the copy: “replicated within 15 minutes”, “predictable replication time”, “compliance requires a guaranteed RPO for the copy”.

#Two-way replication and replica modification sync

You can set rules in both directions (A → B and B → A) for an active-active pair. New objects written in either bucket appear in the other, and S3 does not bounce replicas back. By default, changes made to a replica’s metadata (tags, ACLs, Object Lock settings) stay on the replica. Replica modification sync replicates those metadata changes back to the source, so both sides stay identical. Two-way replication is also what S3 Multi-Region Access Points use for failover (lesson 10.9).

#What goes wrong

• “Replication is on but old files never arrived.” Live replication only copies new writes. Run Batch Replication.
• Some new objects are missing in the destination. Check the object’s replication status: FAILED usually means the role lacks a permission, the destination bucket policy (cross-account) does not allow the role, or KMS permissions are missing. SSE-KMS objects without the opt-in are silently skipped. Objects stored straight into Glacier Flexible or Deep Archive are skipped.
• The destination keeps files that were deleted in the source. Either delete marker replication is off (the default), or they were permanent version deletes, which are never replicated.
• A file deleted by mistake is gone in both Regions. Delete marker replication was on. Delete the marker in each bucket to restore; the data is still there.
• A corrupted upload replaced the good file in both buckets. Replication copies bad writes as faithfully as good ones. Versioning in the destination keeps the old version; replication is not a backup by itself.
• “I cannot suspend versioning.” The bucket is a replication source (remove the rule first), or MFA Delete is on (needs the root MFA code).
• The bill doubled after enabling versioning. An application overwrites the same keys constantly. Add noncurrent version expiration.
• Thousands of delete markers slow down listings. A tool deleted and re-created keys in a loop. Clean expired delete markers with lifecycle.

#Versioning and replication in system design

• Versioning first, everywhere it matters. It is the cheapest insurance against human error, and replication, Object Lock and MFA Delete all build on it. Pair it with noncurrent-version lifecycle rules from day one.
• Replication is for availability and placement, not for history. It copies mistakes too. Versioning (and Object Lock) give history; replication gives a second place. A good disaster-recovery copy combines both: versioned destination, delete marker replication off, ideally in another account with a different set of administrators.
• Choose CRR or SRR by the failure you fear. A Region outage or a regulator’s distance rule → CRR. Account compromise, log centralisation, data-residency or test copies → SRR (cheaper: no inter-Region transfer).
• Recovery point objective. Plain CRR gives a best-effort recovery point; RTC gives a 15-minute one you can write in a contract. Disaster recovery strategies get their own chapter (chapter 23).
• Latency. CRR puts data close to users in another Region, but S3 Multi-Region Access Points or CloudFront (chapter 11) are usually the better way to serve global users.
• Cost model. CRR costs inter-Region transfer per GB, a PUT per replicated object, storage in the destination, and RTC if enabled. Replicating straight into a cheaper storage class for the backup copy saves a lot.

#Try it yourself

With the AWS CLI, two buckets in different Regions (names are examples; bucket names are global, so pick your own):

# 1. Versioning on both buckets
aws s3api create-bucket --bucket scans-example --region ap-south-1 \
  --create-bucket-configuration LocationConstraint=ap-south-1
aws s3api create-bucket --bucket scans-dr-example --region ap-southeast-1 \
  --create-bucket-configuration LocationConstraint=ap-southeast-1
aws s3api put-bucket-versioning --bucket scans-example --versioning-configuration Status=Enabled
aws s3api put-bucket-versioning --bucket scans-dr-example --versioning-configuration Status=Enabled
aws s3api get-bucket-versioning --bucket scans-example

# 2. Play with versions
echo "v1" > menu.txt && aws s3 cp menu.txt s3://scans-example/menu.txt
echo "v2" > menu.txt && aws s3 cp menu.txt s3://scans-example/menu.txt
aws s3api list-object-versions --bucket scans-example --prefix menu.txt
aws s3 rm s3://scans-example/menu.txt                    # adds a delete marker
aws s3api head-object --bucket scans-example --key menu.txt   # 404
aws s3api list-object-versions --bucket scans-example --prefix menu.txt \
  --query 'DeleteMarkers[].{Id:VersionId,Latest:IsLatest}'
aws s3api delete-object --bucket scans-example --key menu.txt --version-id <marker-id>   # undelete
aws s3api get-object --bucket scans-example --key menu.txt --version-id <old-id> old.txt  # read an old version

The replication role (trust policy for s3.amazonaws.com) and a rule:

aws iam create-role --role-name s3-replication-role \
  --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"s3.amazonaws.com"},"Action":"sts:AssumeRole"}]}'
aws iam put-role-policy --role-name s3-replication-role --policy-name replicate \
  --policy-document file://replication-permissions.json

replication.json, one rule for the whole bucket with delete marker replication off and RTC on:

{
  "Role": "arn:aws:iam::111122223333:role/s3-replication-role",
  "Rules": [
    {
      "ID": "scans-to-singapore",
      "Status": "Enabled",
      "Priority": 1,
      "Filter": {},
      "DeleteMarkerReplication": { "Status": "Disabled" },
      "Destination": {
        "Bucket": "arn:aws:s3:::scans-dr-example",
        "StorageClass": "STANDARD_IA",
        "ReplicationTime": { "Status": "Enabled", "Time": { "Minutes": 15 } },
        "Metrics": { "Status": "Enabled", "EventThreshold": { "Minutes": 15 } }
      }
    }
  ]
}

aws s3api put-bucket-replication --bucket scans-example \
  --replication-configuration file://replication.json
aws s3 cp scan-0420.dcm s3://scans-example/
aws s3api head-object --bucket scans-example --key scan-0420.dcm --query ReplicationStatus     # PENDING, then COMPLETED
aws s3api head-object --bucket scans-dr-example --key scan-0420.dcm --query ReplicationStatus  # REPLICA

# Suspending versioning on the source now fails while the rule exists
aws s3api put-bucket-versioning --bucket scans-example --versioning-configuration Status=Suspended

MFA Delete (root user credentials only, CLI only; detail in lesson 10.9):

aws s3api put-bucket-versioning --bucket scans-example \
  --versioning-configuration Status=Enabled,MFADelete=Enabled \
  --mfa "arn:aws:iam::111122223333:mfa/root-account-mfa-device 123456"

#Summary

Key takeaways

• Versioning is a bucket setting that keeps every version of every object. States: unversioned (default), enabled, suspended; once enabled it can never be disabled, only suspended.
• Each write gets a new version ID; the newest is current, older ones noncurrent. Objects from before versioning have the ID null. Every version is billed in full: use noncurrent-version lifecycle rules.
• Simple DELETE → a delete marker becomes current (GET → 404, x-amz-delete-marker: true). Restore = delete the marker. DELETE with a version ID = permanent, no marker. GET of a marker by ID → 405.
• Suspended: new writes get ID null and replace the previous null version; old versions stay.
• MFA Delete: root MFA code needed to permanently delete versions or change versioning state; root only, CLI/API only (lesson 10.9). For ransomware and WORM, Object Lock.
• CRR (other Region) and SRR (same Region), asynchronous, same or different accounts. Requires versioning on both buckets and an IAM role; cross-account needs a destination bucket policy.
• Only new objects replicate; Batch Replication handles existing, failed and already-replicated objects and replicas.
• Delete markers: optional (delete marker replication, not with tag rules). Version deletes: never replicated. Lifecycle actions: never.
• No chaining: A → B → C does not move A’s objects to C; use A → B and A → C.
• SSE-KMS objects need opt-in, a destination-Region key and KMS permissions; SSE-S3 and SSE-C replicate by default; Glacier Flexible / Deep Archive objects do not replicate.
• RTC: 15 minutes (99.99% design, SLA at 99.9%), metrics and events, extra cost. Replica modification sync for two-way metadata.

Self-check: Versioning and Replication30 questions