Moving Large Amounts of Data

By the end of this lesson you will be able to

• Calculate how long a transfer takes over a given link, in decimal units and bits, and explain why the real number is always worse than the textbook one (protocol overhead, a shared line, restarts, the bytes-for-bits mistake).
• Work out the slide’s classic example (200 TB over 100 Mbps, over a 1 Gbps Direct Connect, and by Snowball) and three Bangladesh-sized cases: a television archive, a bank’s petabytes and a large daily change rate.
• Compare the online methods (internet with DataSync, Site-to-Site VPN, Direct Connect, S3 Transfer Acceleration) and the offline one (Snowball), with their setup time, speed, encryption and cost.
• Plan a real migration in two halves: a one-time bulk copy, then an ongoing sync with DataSync or DMS over a VPN or Direct Connect until cutover.
• Read the exam’s keywords (“weeks over the internet”, “petabytes”, “can wait a month for setup”, “users worldwide upload to one bucket”) and know which parts of the Snow Family story have changed since the questions were written.

#Where this lesson fits

The previous four lessons of this chapter moved servers and databases. Every one of those moves sooner or later hits the same wall: the data itself has to cross from your building to an AWS Region, and there may be terabytes or petabytes of it. This lesson is about that wall, and it is a lesson of numbers: the exam gives you a data size and a link speed and expects you to do the arithmetic before you pick a service.

The tools themselves are taught elsewhere, so here we compare them and refer back:

• Snowball Edge, its job life cycle and its status today: lesson 12.1. The bulk paths into S3 (DataSync, Snowball, Transfer Acceleration): lesson 10.11. DataSync in depth: lesson 12.4.
• Site-to-Site VPN and Direct Connect as networks: lesson 22.4 and lesson 22.5. Here they are only pipes with a speed and a setup time.
• DMS and change data capture for databases: lesson 23.3. MGN for whole servers: lesson 23.4.

The notes split the methods the same way the exam does: over the network (online) and offline (physical). We start with the arithmetic, because it decides between them.

#Always do the maths first

#Bits, bytes and decimal units

One formula does all the work:

transfer time = data in bits ÷ usable link speed in bits per second

Three facts make it come out right:

• Links are sold in bits per second (Mbps, Gbps). Data is measured in bytes (GB, TB). One byte is 8 bits, so 100 Mbps carries only 12.5 MB per second, and 1 Gbps carries 125 MB/s.
• Use decimal units, as networks and the exam do: 1 TB = 1,000 GB = 1,000,000 MB, so 1 TB = 8,000 gigabits = 8,000,000 megabits. 1 PB = 1,000 TB.
• A day has 86,400 seconds. Dividing seconds by 86,400 gives days.

A handy rule to keep in your head: a 100 Mbps line moves at most about 1 TB a day (100 Mbps × 86,400 s = 8,640,000 megabits = 1.08 TB). A 1 Gbps line moves about 10.8 TB a day, and 10 Gbps about 108 TB a day. Everything below is this rule, scaled.

#The slide’s example, worked out

The classic case: 200 TB must move to AWS, and the company has a 100 Mbps internet connection.

Over the internet or a Site-to-Site VPN.

1. Data in megabits: 200 TB × 1,000 GB × 1,000 MB × 8 = 1,600,000,000 megabits.
2. Divide by the link: 1,600,000,000 ÷ 100 Mbps = 16,000,000 seconds.
3. Into days: 16,000,000 ÷ 86,400 = about 185 days, six months.

Setup time is zero: the line exists, and a VPN can be up the same day. But a VPN does not make the line wider: it encrypts the same internet traffic.

Over a 1 Gbps Direct Connect.

1. Data in gigabits: 200 × 1,000 × 8 = 1,600,000 gigabits.
2. Divide by the link: 1,600,000 ÷ 1 Gbps = 1,600,000 seconds.
3. Into days: about 18.5 days.

But the line must first exist. The slide says the one-time setup takes over a month; AWS’s own hybrid connectivity guidance says a brand-new dedicated connection typically takes several weeks to months. So from the day you order it, roughly 50 days or more.

By Snowball. About one week end to end for the transfer itself: the device arrives, is filled over the local network in a couple of days (200 TB at about 1 GB/s is about 2.3 days), is shipped back and imported. The slide adds that it can be combined with DMS for databases, which we will see below.

That table is the whole lesson in miniature: the internet starts today but finishes late, Direct Connect is fast but starts late, Snowball skips the wire.

#Why the real number is worse

The slide’s numbers assume the line carries your data at 100% of its speed for six months. It never does:

• Protocol overhead. Every packet carries TCP/IP headers, the copy tool adds TLS, a VPN adds IPsec headers. Lost packets are sent again.
• Real utilisation. On a dedicated line, plan with about 80%. 185 days ÷ 0.8 = about 231 days.
• A shared line. If the office needs the line by day, the copy may get half of it from 9 to 5. A mix of 50 Mbps by day and 80 Mbps by night averages about 60 Mbps: about 300 days.
• Restarts and outages. Over months, a line will drop; good tools (DataSync) resume and verify, a home-made script may start a file again.
• Binary units on the source. Storage tools often show TiB (2^40 bytes, about 1.1 TB). “200 TiB” on the NAS screen is about 220 TB on the wire: 10% more time.
• The bytes-for-bits mistake. 200 TB ÷ “100 MB/s” = 2,000,000 s = 23 days. Wrong by a factor of 8, and a common wrong option in exam questions.

#The online methods

The notes list four ways to move data over the network. Each answers a different exam keyword.

#AWS DataSync: the copy engine

DataSync is not a pipe but the tool that uses one. An agent (a virtual machine next to your storage) reads NFS, SMB, HDFS or S3-compatible object storage, and other clouds’ storage can be read too; the service writes into S3, EFS or any FSx file system. It checks what changed, copies only that, verifies the result, keeps file metadata, encrypts with TLS in transit, and runs on a schedule. A single task can move up to 10 Gbps with an agent. It runs over whatever link you have: internet, VPN or Direct Connect.

Exam keyword (from the notes): “automated, scheduled online data transfer from on-premises storage to AWS storage” → DataSync. Details and pricing: lesson 12.4.

#Site-to-Site VPN

An IPsec tunnel over the internet between your router and AWS. The notes give its profile, and it is still right:

• Quick to set up: minutes to hours, no carrier, no contract.
• Encrypted by design.
• Speed is the internet’s: no faster than your line, and it varies with internet traffic.
• One more ceiling the notes do not mention: a standard tunnel carries up to about 1.25 Gbps. Since November 2025, large bandwidth tunnels of up to 5 Gbps exist on a transit gateway or Cloud WAN, and several tunnels can be used together (lesson 22.4).

Use it when you need encryption and a start today, and the data volume fits the line.

#AWS Direct Connect

A private, dedicated line from your premises to AWS that never crosses the public internet. The notes list its strengths and its catch:

• Fast and consistent: 1, 10, 100 or 400 Gbps dedicated ports, or smaller hosted connections from a partner; latency and throughput do not swing with internet traffic.
• Slow to set up: a brand-new dedicated connection typically takes weeks to months; a hosted connection on a partner’s existing port can be ready in days.
• Not encrypted by default on your link. A correction the notes leave out: to encrypt it, use MACsec (on 10, 100 and 400 Gbps dedicated connections) or run an IPsec VPN over Direct Connect. Lesson 22.5 builds both.
• Cheaper data transfer out than the internet, and the line keeps paying off after the migration.

Exam keyword: “needs a dedicated private connection but can wait a month for setup” → Direct Connect.

#S3 Transfer Acceleration

For a different problem: many people far from the bucket. Uploads enter the AWS network at the nearest edge location (the CloudFront network) and cross the AWS backbone to the bucket’s Region, instead of the long public internet path. It speeds up distance, not your own line: a 100 Mbps office still sends 100 Mbps. It adds a per-GB fee, charged only when the transfer was actually faster. Full details: lesson 10.11.

Exam keyword: “users all over the world upload large files to one S3 bucket” → S3 Transfer Acceleration.

#The online methods side by side

#The offline method: Snowball

When the line is too slow, too shared or too expensive for the volume, put the data in a box and ship it. The notes describe three devices, and that is still how many questions are written:

For new customers AWS points to DataSync over a big link (for example a temporary hosted Direct Connect) and to AWS Data Transfer Terminal: physical sites, 11 of them after February 2026 in the United States, Europe, Sydney and Tokyo, where you bring your own drives and upload over very fast fibre. The device details, prices and job steps are in lesson 12.1; there is no need to repeat them here.

#The “more than a week” rule is a heuristic

The notes call it an AWS rule: if the network would take more than a week, use the Snow Family. It is a study heuristic, not an official AWS rule. It comes from AWS’s old guidance that devices make sense when a network transfer would take more than about a week, and from the fact that a Snowball round trip took about a week. It still works for exam questions. In practice, weigh three things the rule hides: device preparation can take up to 4 weeks; a fat dedicated link beats any courier (60 TB over 10 Gbps at 80% is under 17 hours); and a steady daily stream can never be fixed by a device.

#One-time bulk, then ongoing sync

A real migration almost never stops the business while the data moves. The slide’s last point is the pattern every large migration follows:

1. Bulk copy, once

   Move the big, mostly static part: the archive, the file shares, the database’s tables. By Snowball if the wire is too slow, by DataSync over Direct Connect if it is not.

2. Keep capturing changes

   From the moment the bulk copy starts, the source keeps changing. For files, DataSync runs on a schedule and copies only what changed. For databases, DMS with change data capture records and replays every change (lesson 23.3).

3. Ongoing replication over a VPN or Direct Connect

   The changes are small enough for the wire: Site-to-Site VPN or Direct Connect, with DMS or DataSync, exactly as the slide says.

4. Cutover

   Freeze writes for a few minutes, run the last sync, check counts and checksums, switch users or applications to AWS. Keep the old copy until you are sure.

For a database the bulk can also travel by device: SCT data extraction agents write the tables onto a Snowball Edge, AWS loads them into S3 and the target (typically Redshift), and DMS replays the changes captured meanwhile. That is what “Snowball can be combined with DMS” means.

#Choosing with numbers

Put it all together. For any question, answer four things in this order: how much data, how fast the line, how long you can wait, and does it keep changing?

What the calculator teaches, in four sentences:

• Small data on a decent line: just send it (DataSync or a VPN); nothing else is worth the setup.
• Big data on a slow line, once: a device beats the wire; the exam answer is Snowball.
• Big data and a line you will keep: Direct Connect, even though it takes weeks to arrive.
• A steady daily stream: only the line counts; size it so it carries a day’s changes in well under a day.

#What goes wrong

#In system design

• Every migration plan starts with a transfer budget. Before choosing MGN, DMS or anything else, compute how long the data takes on each path, including setup and the daily change rate.
• Two lines, two jobs. A common hybrid design keeps Direct Connect for the steady traffic and a Site-to-Site VPN as an encrypted backup path (lesson 22.5).
• Size the line for the stream, not the bulk. The bulk happens once; the daily changes happen forever. If the line cannot carry a day in a day, the design is wrong whatever the bulk method.
• Verification is part of the transfer. DataSync verifies; for devices, compare object counts and checksums after the import; keep the source until the checks pass.
• Plan for what exists after 2026. For long projects design with DataSync, Direct Connect and Data Transfer Terminal, and treat Snowball as exam knowledge.

#Try it yourself

First, the arithmetic. Any shell with bc (or a calculator) will do:

# Days to move 200 TB over 100 Mbps at 100% and 80%
echo "200*8000*1000 / 100 / 86400" | bc -l          # megabits ÷ Mbps ÷ s/day
#   185.18518518518518518518
echo "200*8000*1000 / (100*0.8) / 86400" | bc -l
#   231.48148148148148148148

# Over a 1 Gbps Direct Connect (gigabits ÷ Gbps)
echo "200*8000 / 1 / 86400" | bc -l
#   18.51851851851851851851

# How much a line carries per day, in TB (200 Mbps at 80%)
echo "0.2*0.8*86400 / 8000" | bc -l
#   1.72800000000000000000

A DataSync task over whatever link you have, with a bandwidth limit for office hours (IDs are examples):

# Locations were created before: the NFS share (through the agent) and the bucket
aws datasync create-task \
  --source-location-arn arn:aws:datasync:ap-southeast-1:111122223333:location/loc-0abc1234example1 \
  --destination-location-arn arn:aws:datasync:ap-southeast-1:111122223333:location/loc-0def5678example2 \
  --name archive-to-s3 \
  --options VerifyMode=ONLY_FILES_TRANSFERRED,TransferMode=CHANGED,BytesPerSecond=6250000 \
  --schedule ScheduleExpression="cron(0 20 * * ? *)"
#   { "TaskArn": "arn:aws:datasync:ap-southeast-1:111122223333:task/task-0123456789abcdef0" }
# BytesPerSecond=6250000 is 50 Mbps: the copy leaves the rest of a 100 Mbps line to the office

aws datasync start-task-execution \
  --task-arn arn:aws:datasync:ap-southeast-1:111122223333:task/task-0123456789abcdef0
#   { "TaskExecutionArn": ".../task-0123456789abcdef0/execution/exec-0a1b2c3d4e5f60718" }

Transfer Acceleration for far-away uploaders:

aws s3api put-bucket-accelerate-configuration \
  --bucket media-uploads-example --accelerate-configuration Status=Enabled
aws s3 cp ./clip.mp4 s3://media-uploads-example/raw/ --endpoint-url https://s3-accelerate.amazonaws.com

And a Snowball import job, only for accounts that still have access (full walk-through in lesson 12.1):

aws snowball create-job --job-type IMPORT --snowball-type V3_5S --snowball-capacity-preference T240 \
  --resources '{"S3Resources":[{"BucketArn":"arn:aws:s3:::archive-import-example"}]}' \
  --address-id ADID1a2b3c4d-1111-2222-3333-444455556666 \
  --role-arn arn:aws:iam::111122223333:role/SnowballImportRole \
  --shipping-option SECOND_DAY
#   { "JobId": "JID123e4567-e89b-12d3-a456-426655440000" }

#Summary

Key takeaways

• Time = bits ÷ usable bits per second. 1 TB = 8,000 gigabits; a day = 86,400 s; 100 Mbps ≈ 1 TB a day, 1 Gbps ≈ 10.8 TB a day. Multiply bytes by 8.
• The slide’s 200 TB: 185 days over 100 Mbps internet or VPN (no setup), 18.5 days over 1 Gbps Direct Connect (plus over a month of setup), about a week by Snowball. Real numbers are worse: plan with 80%, less on a shared line.
• VPN: quick, encrypted, internet speed, about 1.25 Gbps per standard tunnel (5 Gbps large tunnels since 2025). Direct Connect: private, consistent, weeks to months to set up, not encrypted by default (MACsec or VPN over DX). DataSync: the scheduled, verified copy over any link. Transfer Acceleration: distant users, one bucket.
• Offline: Snowball Edge for weeks-of-network volumes. Snowmobile retired (2024), Snowcone discontinued (2024), Snowball closed to new customers (November 2025), support ends 31 December 2026; the exam still expects Snowball.
• “More than a week → Snow” is a study heuristic, not an AWS rule.
• Real migrations are bulk + ongoing sync: Snowball (or DataSync over DX) for the bulk, then DMS or DataSync over VPN or Direct Connect until cutover. A device fixes a bulk, never a daily stream.

Self-check: moving large amounts of data30 questions