Course outline

10.11

Paths to S3 (2/3): Moving Data in Bulk

S3 File Gateway, DataSync, Snowball, Transfer Acceleration and Transfer Family: which one fits which migration or daily feed.

30 min Reading + animations

Tip: tap or hover any dotted-underlined term to see what it means.

By the end of this lesson you will be able to

  • Place the five bulk paths of this lesson on the 13-path map of how data reaches S3, and say in one sentence what each one is for.
  • Explain how S3 File Gateway turns an NFS or SMB share into objects in a bucket, with a local cache, Active Directory and lifecycle rules, and recognise the other Storage Gateway types (Volume, Tape, FSx File).
  • Describe how DataSync migrates and synchronises data with an agent, schedules, incremental copies and verification, including AWS-to-AWS and other-cloud transfers.
  • Calculate how long a transfer takes over a network link, decide when shipping Snowball devices is faster, and use a lifecycle rule to land Snowball data in Glacier.
  • Explain how S3 Transfer Acceleration uses edge locations, its requirements and price, and when it does not help.
  • Set up a managed SFTP, FTPS, FTP or AS2 endpoint with AWS Transfer Family, map each partner to its own prefix, and choose the right identity provider.
  • Choose between the five for a scenario, using the questions that separate them: one-off or ongoing, online or offline, file protocol or not, who sends the data.

#Where this lesson sits on the map

The owner asked for one place where every way data moves into or out of S3 is laid out side by side, because these services are easy to confuse. There are 13 paths, split over three lessons. Lesson 10.10 covered the private network routes (1 Gateway endpoint, 2 Interface endpoint, 3 VPC Peering, 4 Transit Gateway, 5 Site-to-Site VPN or Direct Connect): they decide which road the packets take. This lesson covers the bulk paths (6 S3 File Gateway, 7 AWS DataSync, 8 AWS Snowball, 9 S3 Transfer Acceleration, 10 AWS Transfer Family): services that move a lot of data for you, each in its own way. Lesson 10.12 covers the application paths (11 multipart upload, 12 pre-signed URLs, 13 Batch Operations) and ends with a chooser across all 13.

The five in this lesson answer different questions, and that is the key to never mixing them up:

#PathThe question it answersOnline or offlineOne-off or ongoing
6S3 File Gateway“My servers on site must keep using a file share, but I want the data in S3.”OnlineOngoing (it stays in the data path)
7DataSync“Copy these terabytes from here to there, quickly, safely, maybe every night.”OnlineOne-off migration or scheduled
8Snowball“The network would take weeks or months. Can I post the data?”OfflineOne-off (per job)
9Transfer Acceleration“Users far from the bucket upload slowly over the internet.”OnlineOngoing
10Transfer Family“Partners send files with SFTP, FTPS, FTP or AS2, and I don’t want to run that server.”OnlineOngoing

Every one of these services is taught in more depth later in the course: Snowball, Storage Gateway, Transfer Family and DataSync in chapter 12 (Snowball Edge, Storage Gateway, Transfer Family, DataSync and choosing), and large migrations in chapter 23 (moving large amounts of data). Transfer Acceleration also appears as a performance tool in lesson 10.6. This lesson is complete on its own: what each one is, how data flows, its limits and price, a real example, and why you would pick it over the other four.

#Path 6: Amazon S3 File Gateway

#Why hybrid storage needs a translator

Many companies run hybrid cloud: part of their systems in AWS, part on premises. The reasons are ordinary ones: a migration that takes years, security or compliance rules that keep some systems in the building, or simply an IT strategy. The servers that stay on premises still need storage, and S3 is attractive: unlimited, eleven nines of durability, cheap archive classes.

But S3 is object storage with its own HTTPS API (PUT, GET, list requests), not a file system. A Linux server mounts file shares with NFS, a Windows server with SMB, and old applications have no idea how to call an S3 API. Unlike EFS, which speaks NFS natively (lesson 5.4), S3 needs a translator. That translator is AWS Storage Gateway: a bridge between on-premises data and cloud storage, used for backup and restore, disaster recovery, tiered storage and a low-latency local cache. It comes in several types; the one that puts files into S3 is Amazon S3 File Gateway.

#How a file becomes an object

S3 File Gateway is a virtual appliance you run on premises: a VM on VMware, Microsoft Hyper-V, Linux KVM or Nutanix, or an EC2 instance from an AMI. You give it a cache disk and connect it to one or more S3 buckets. Then:

  1. You create a file share on the gateway: an NFS export (NFS v3 or v4.1) or an SMB share (SMB v2 or v3). Each share maps to one bucket, or to a prefix in a bucket. One gateway can hold up to 50 shares.
  2. Servers mount the share like any NAS. When an application writes a file, the gateway stores it on the local cache and confirms at once, so the application gets local-disk speed.
  3. In the background the gateway uploads the file over HTTPS (on the internet, or over Direct Connect or a VPN from lesson 10.10) to S3. One file becomes one object; the path becomes the key, and file metadata such as owner, permissions and timestamps is kept in the object’s metadata.
  4. Reads of recently used files are served from the cache (a cache hit). Reads of anything else are fetched from S3 (a cache miss), then cached.
  5. The gateway writes into S3 Standard, Standard-IA, One Zone-IA or Intelligent-Tiering. It cannot write directly into the Glacier classes; a normal lifecycle rule on the bucket (lesson 10.5) moves old objects there. An object that has gone to Glacier Flexible Retrieval or Deep Archive must be restored before the share can read it again.
  6. The gateway reaches the bucket with an IAM role you give it (one per share). For SMB shares it can join your Active Directory, so users log in with their normal domain accounts and Windows permissions apply.

Because the data lands as plain objects, everything in AWS can use it: analytics with Athena, machine learning, replication to another Region, versioning, Object Lock. That is the big advantage over a NAS: the files are on premises for the people who need them and in the cloud for everything else.

Limits and numbers. Cache from 150 GiB up to 64 TiB per gateway. Up to 50 file shares per gateway. Maximum file size 5 TiB (an S3 object’s limit). Cost: about $0.01 per GB written to AWS, capped at $125 per gateway per month (the first 100 GB per account are free), plus normal S3 storage and request charges; the VM runs on your own hardware, or you pay for the EC2 instance.

Watch a write, a cache hit and a miss, the lifecycle move, the Active Directory check and a broken internet link:

S3 File Gateway: a file share backed by a bucket

A hospital’s scan archive writes to an ordinary NFS share. The share is a gateway VM that keeps recent files in a local cache and stores every file as an object in S3.

Scenario

The archive saves a new scan to what it thinks is a local file server.

Hospital data centre
us-east-1
Scan archivePACS
Doctor’s PC
Active Directory
File Gateway
S3 bucket
Deep Archive
NFS write
Step 1 of 4
1

The scan archive mounts the share over NFS, exactly as it would mount a NAS. It has no idea S3 exists, and it does not need an SDK or any code change. That is the whole point of a file gateway.

#The rest of the Storage Gateway family

S3 File Gateway is one of four gateway types. You need the others only as context for this lesson; chapter 12 covers them in depth.

  • Volume Gateway presents block storage (disks, not files) to your servers over iSCSI, backed by S3 and backed up as EBS snapshots. Two modes:
    • Cached volumes: the full data lives in AWS; only frequently used data is cached on premises for low latency. Good when the primary copy should be in the cloud.
    • Stored volumes: the full data stays on premises for the lowest latency, and the gateway sends asynchronous point-in-time backups to AWS as EBS snapshots. Good when everything must stay local but you want off-site backup.
  • Tape Gateway replaces physical tapes (see the analogy below). It is a VTL, a virtual tape library, that your backup software sees over iSCSI as tape drives and a media changer. Virtual tapes are stored in S3; when the backup software ejects (archives) a tape, it moves to S3 Glacier Flexible Retrieval or Glacier Deep Archive.
  • Amazon FSx File Gateway gave offices a local cache in front of an FSx for Windows File Server share. It has been closed to new customers since 28 October 2024; existing customers can keep using it.
Exam phraseGateway type
On-premises servers need NFS or SMB access to files stored in S3S3 File Gateway
Keep all data on premises for low latency, back it up to AWSVolume Gateway, stored volumes
Primary data in AWS, only frequently used data on premisesVolume Gateway, cached volumes
Replace physical backup tapes, keep the existing backup softwareTape Gateway
Windows file shares with Active Directory, but no S3 in the questionFSx for Windows File Server (chapter 12), not a gateway

#Path 7: AWS DataSync

#What it is, and how it moves data

AWS DataSync is a managed service for copying large amounts of data over the network, quickly and safely, once or on a schedule. Think of it as a professional removal company: it packs, moves, checks nothing is broken, and leaves. Unlike a gateway, it does not stay in the data path.

  • Sources and destinations. On premises: NFS, SMB, HDFS (Hadoop) and self-managed object storage that speaks the S3 API. In AWS: S3 (any storage class, including Glacier Flexible Retrieval and Deep Archive), EFS, and every FSx file system. Other clouds: Google Cloud Storage, Azure Blob Storage and Azure Files, and S3-compatible stores such as Wasabi, Oracle, Cloudflare R2 and others.
  • The agent. For on-premises storage you deploy a DataSync agent: a VM (VMware, Hyper-V, KVM) or an EC2 instance that sits near the storage and reads it over NFS, SMB or HDFS. Nothing is installed on the file server itself. No agent is needed between AWS storage services in the same account, or between S3 and object storage in another cloud.
  • The transfer. A task names a source location, a destination location and options. The agent reads in parallel and sends data over TLS with AWS’s own protocol, which compresses in flight, detects sparse files and keeps a long, busy link full. One task can use about 10 Gbps; several agents and tasks can run in parallel for bigger jobs.
  • Incremental. By default a run copies only data that is new or has changed since the last run, so the first run moves everything and later runs move only the difference.
  • Scheduling. Tasks can run hourly, daily or weekly (the notes’ example: every night at 12).
  • Bandwidth limit. You can cap a task’s bandwidth, for example during office hours.
  • Verification. DataSync compares checksums of what it read with what it wrote and reports any file that does not match. You can verify only what was transferred, the whole destination, or skip it.
  • Metadata. File permissions and metadata (NFS POSIX ownership and permissions, SMB attributes and ACLs, timestamps) are preserved, which matters when you copy into EFS or FSx and users must keep their permissions.
  • Filters and reports. Include or exclude patterns, and task reports that list every file transferred, skipped or failed.

Cost. You pay per GB copied: $0.0125 per GB in Basic mode, or $0.015 per GB plus $0.55 per task execution in Enhanced mode, a newer mode for S3, other-cloud and NFS/SMB-to-S3 transfers that scales to virtually unlimited numbers of files. Normal S3 request charges and any data transfer out of the source also apply. There is no charge for the agent itself.

DataSync: copy, verify, repeat

A newspaper moves 200 TB of photo archives from its NAS to S3. An agent VM next to the NAS reads the files; the DataSync service writes them into AWS and checks every byte.

Scenario

Newspaper office
us-east-1
Photo NAS
Agent VM
DataSync
S3 bucket
EFS
mount
Step 1 of 4
1

You deploy the DataSync agent as a VM on the office’s hypervisor and activate it. The agent mounts the NAS read-only over NFS or SMB: no software goes on the NAS itself.

#DataSync or Storage Gateway?

This is the pair the owner’s notes warn about, and the difference is in one word: copy versus extension.

DataSync: a removal companyStorage Gateway: a permanent bridge

  • Copies data from A to B, once or on a schedule, then steps out of the way
  • Applications switch to the new location after the migration
  • Agent reads the source; tasks, schedules, verification, reports
  • Moves on premises → AWS, AWS → AWS (S3 ↔ EFS ↔ FSx), other clouds → AWS
  • Writes into any S3 class, including Glacier, directly
  • Exam words: migrate, scheduled transfer, sync, agent, one-time
  • Extends on-premises storage into AWS, permanently; it stays in the data path
  • Applications keep using a local NFS, SMB or iSCSI target every day
  • Local cache for low-latency access to recent data
  • Only on premises ↔ AWS
  • File Gateway writes into Standard, IA or Intelligent-Tiering; lifecycle to Glacier
  • Exam words: hybrid, on-premises access, local cache, NFS/SMB to S3, tapes

They are often used together: DataSync for the big initial copy of years of files, then a File Gateway for the applications that keep writing new files.

Self-checkQuestion

A company must move 50 TB of files from an on-premises NFS server to Amazon EFS over its existing Direct Connect link within two weeks, keep POSIX permissions, and decommission the server afterwards. Which service fits best?

#Path 8: AWS Snowball

#Why ship data on a truck

Sometimes the network is the problem. The link is slow, shared with the whole office, expensive, unstable, or simply does not exist (a ship, a mine, a disaster zone). Then it is faster to put the data on a device and post it. That is AWS Snowball: secure, rugged, portable devices for offline data migration into and out of AWS, and for computing at the edge. It handles terabytes to petabytes.

The current devices are Snowball Edge:

DeviceStorageComputeUse
Snowball Edge Storage Optimized210 TB NVMe104 vCPU, 416 GBLarge data migrations; up to about 1.5 GB/s transfer
Snowball Edge Compute Optimized28 TB NVMe104 vCPUs, 416 GB memoryEdge computing: run EC2 instances or Lambda functions on site

How a job works:

  1. In the console you create a job: import into S3, export from S3, or local compute and storage only. You choose the bucket and the delivery address.
  2. AWS ships the device. Its E Ink screen shows the shipping label.
  3. You connect it to your local network (10 to 100 Gb Ethernet), unlock it with the job’s manifest file and unlock code, and copy data with the S3-compatible endpoint or an NFS mount. At local network speed even 200 TB fills in a day or two.
  4. You power it off; the E Ink label switches to the return address; the courier collects it.
  5. AWS imports the data into your bucket, then wipes the device to NIST standards. You follow the job status in the console.

Security. Everything is encrypted with 256-bit keys managed in AWS KMS; the keys are never stored on the device. The device is tamper-resistant, and losing it in transit exposes nothing without the manifest and unlock code.

Cost. A service fee per job (which includes a number of days on site), a daily fee for extra days, and shipping. Importing data into S3 has no per-GB transfer charge; exporting out of S3 is charged per GB.

#How long would the network take?

The calculation is simple and appears in many exam questions:

time = data in bits ÷ usable link speed

1 TB is 8,000 gigabits (decimal units, as networks count them). Links rarely run at 100% because other traffic shares them, so plan for about 80%:

Data100 Mbps1 Gbps10 Gbps
10 TB11.6 days1.2 days (28 hours)2.8 hours
100 TB116 days11.6 days1.2 days
1 PB3.2 years116 days11.6 days

Worked example: 100 TB over 1 Gbps = 800,000 gigabits ÷ 0.8 Gbps = 1,000,000 seconds = 11.6 days (9.3 days even at a perfect 100%). With Snowball: 2 to 3 days for the device to arrive, about a day to fill at local speed, 2 to 3 days to ship back, a day to import: about a week, whatever the size up to a device’s 210 TB, because several devices fill in parallel.

The rule of thumb: if it takes more than about a week over the network, use Snowball.

Self-checkQuestion

A company has 300 TB to move to S3 once, over a 500 Mbps internet link that it can use at about 80%. Roughly how long would the network transfer take, and what does the exam expect?

#Snowball cannot import straight into Glacier

A favourite exam trap. You have a pile of old data that only needs to be archived. Can the Snowball job put it directly into S3 Glacier or Glacier Deep Archive? No. A Snowball import always lands in S3 (Standard by default). The solution:

  1. Import into S3 Standard as usual.
  2. Before you ship the device back, add an S3 lifecycle rule on the bucket (or prefix) that transitions objects to Glacier Flexible Retrieval or Deep Archive after 0 days.
  3. Objects move to the archive class within a day or two of import, so you pay for Standard storage only briefly.

Compare DataSync, which can write into the Glacier classes directly. And remember S3 File Gateway writes only into Standard, IA or Intelligent-Tiering. In both offline and gateway cases, the lifecycle rule is the bridge to Glacier.

#Edge computing with Snowball Edge

Edge computing means processing data where it is created, at a location with limited internet and no computing power: a truck on the road, a ship at sea, a mining station underground. A Snowball Edge (Compute Optimized is built for it; Storage Optimized can also run small workloads) runs EC2 instances and Lambda functions on site, for example to pre-process sensor data, run machine learning inference or transcode video, and later ships the results back to AWS.

Snowball: when the road is faster than the wire

A film studio has 400 TB of raw footage and a 1 Gbps link shared by the whole office. AWS posts a rugged storage device; the studio fills it and posts it back.

Scenario

Film studio
us-east-1
Console
Footage NAS
Courier
Snow service
S3 Standard
Deep Archive
import job
Step 1 of 5
1

In the console the studio creates an import job: which bucket, which address. AWS prepares a device with the job’s encryption keys, which are managed in KMS and never stored on the device.

#The Snow Family today, and what the exam expects

The Snow Family has changed a lot, and the notes and many practice questions describe the old line-up:

  • AWS Snowcone (the small, portable device: 8 TB HDD or 14 TB SSD, for drones, fire stations, factories and places with little space or no internet) was discontinued on 12 November 2024.
  • AWS Snowmobile (a shipping-container truck holding up to 100 PB, for exabyte-scale moves) was retired in 2024.
  • The older Snowball Edge models (80 TB Storage Optimized, 52-vCPU and GPU Compute Optimized) were discontinued in November 2024 and replaced by the 210 TB and 28 TB models above.
  • Since 7 November 2025, Snowball Edge is available only to existing customers: AWS no longer offers any Snow device to new customers. AWS points new customers to DataSync for online transfers (with a temporary hosted Direct Connect connection for speed), AWS Data Transfer Terminal (physical AWS locations where you bring your own drives and upload at high speed) or partner offline services, and to AWS Outposts for edge computing.

#Path 9: S3 Transfer Acceleration

#Taking the AWS motorway early

Over long distances the public internet is slow for big uploads. Each packet crosses many networks; the round trip from Sydney to Virginia is about 200 ms; some packets are lost; and TCP slows down every time it sees loss. A fast office line may deliver only a fraction of its speed to a faraway bucket.

S3 Transfer Acceleration fixes the long part. You enable it on the bucket, and clients use a special endpoint, bucketname.s3-accelerate.amazonaws.com (or bucketname.s3-accelerate.dualstack.amazonaws.com for IPv6). DNS sends each client to the nearest CloudFront edge location, a short hop away. From there the data travels to the bucket’s Region over the AWS private backbone, on optimised, already-warm connections. The bucket, its objects and its Region do not change; only the path does.

  • Requirements: the bucket name must be DNS-compliant with no periods (dots); requests must use virtual-hosted style; only the bucket owner (or someone with s3:PutAccelerateConfiguration) can enable it; it can take up to 20 minutes before speeds improve. It works on general purpose buckets in most commercial Regions.
  • Combine with multipart upload (lesson 10.12): every part rides the accelerated path in parallel. AWS reports gains from about 50% to several hundred percent for long distances.
  • Cost: on top of normal S3 charges, $0.04 per GB for uploads through edges in the United States, Europe and Japan, $0.08 per GB through all other edges; downloads $0.04 per GB. If AWS determines that an accelerated transfer was not faster than a normal one, you are not charged the acceleration fee for it. The Speed Comparison tool shows the expected gain from your location.
  • When it does not help: users close to the bucket’s Region, small objects where latency dominates, or a link that is simply too small.

Transfer Acceleration: get on the AWS network early

A photographer in Sydney uploads 2 GB of photos a day to a bucket in us-east-1. Two routes: across the public internet the whole way, or a short hop to the Sydney edge location and then AWS’s own backbone.

Scenario

us-east-1
Sydney user
Public internet
Sydney edge
S3 bucket
PUT
Step 1 of 2
1

Without acceleration the upload goes to bucket.s3.us-east-1.amazonaws.com. Every packet crosses the Pacific through many networks AWS does not control.

Transfer Acceleration uses CloudFront’s edge locations, but it is not CloudFront: CloudFront is a cache in front of content that many people download (chapter 11), while acceleration is mainly a faster on-ramp for uploads and downloads of individual objects. A related idea, Global Accelerator, routes any TCP or UDP traffic through the AWS backbone (also chapter 11). For performance tuning of S3 overall, see lesson 10.6.

#Path 10: AWS Transfer Family

#Managed file transfer servers

Thousands of business processes still exchange files with old, trusted protocols: a bank receiving payment files from partners, a retailer exchanging orders with suppliers, a hospital receiving lab results. The partners have scripts and tools that speak SFTP, FTPS or FTP, and changing them is out of the question. Traditionally you ran your own SFTP server: patch it, scale it, back it up, copy files from it into storage.

AWS Transfer Family is a fully managed file transfer service that puts a server endpoint in front of Amazon S3 or Amazon EFS:

  • Protocols: SFTP (over SSH), FTPS (FTP over TLS), FTP (unencrypted), and AS2 (a business-to-business standard with signed receipts, common in supply chains and payments). Since late 2024 there are also Transfer Family web apps, a browser-based way for business users to upload and download files in S3.
  • Endpoints: a public endpoint (SFTP only), or a VPC endpoint that is either internal or internet-facing with Elastic IP addresses and security groups. Plain FTP is allowed only on an internal VPC endpoint, never over the internet. You can put your own DNS name in front with Route 53. Endpoints run across up to three AZs and scale automatically.
  • Identity: service-managed users (SSH keys stored in the service; SFTP only), AWS Directory Service for Microsoft AD, or a custom identity provider through Lambda or API Gateway (LDAP, Okta, Amazon Cognito, anything else).
  • Access: each user has an IAM role that decides what the server may do in the bucket on that user’s behalf, an optional session policy, and a home directory. Logical directories map the user’s “/” to a prefix such as inbound/partner-a/ (a chroot), so users never see the bucket name or each other.
  • After the upload: the file is an ordinary object, so S3 event notifications (lesson 10.7) or Transfer Family managed workflows (copy, tag, decrypt, scan, or any Lambda step) can process it at once. SFTP connectors let AWS send files to a partner’s SFTP server too.
  • Cost: per hour for each protocol enabled on an endpoint (about $0.30 per hour in us-east-1, even when idle), plus about $0.04 per GB uploaded or downloaded; AS2 is billed per message, web apps per unit-hour.

Transfer Family: SFTP without an SFTP server

A bank receives payment files from partner companies over SFTP. The partners keep their scripts; the bank swaps its old SFTP server for a managed endpoint that writes into S3.

Scenario

us-east-1
Partner ASFTP
Partner BFTPS
Identity
Transfer Family
S3 bucket
EFS
SSH key
Step 1 of 4
1

Partner A’s nightly script connects to sftp.bank.example (a Route 53 name for the endpoint) and logs in with its SSH key, exactly as it did with the old server.

#The five side by side

S3 File GatewayDataSyncSnowball EdgeTransfer AccelerationTransfer Family
What it isLocal file share backed by S3Managed copy and sync servicePhysical device shipped to youFaster path for uploads via edge locationsManaged SFTP / FTPS / FTP / AS2 endpoint
Protocol inNFS, SMBNFS, SMB, HDFS, S3 API (agent reads)Copy to device locally (NFS, S3-compatible)HTTPS to the S3 APISFTP, FTPS, FTP, AS2, browser
How data reaches S3Gateway uploads over HTTPSAgent → DataSync over TLSCourier → AWS importNearest edge → AWS backboneEndpoint writes with the user’s IAM role
Online or offlineOnlineOnlineOfflineOnlineOnline
One-off or ongoingOngoing, permanentBoth: one-off or scheduledOne-off per jobOngoingOngoing
Local cacheYesNoThe device is the “cache” in transitNoNo
Stays in the data pathYesNoNoYes (per request)Yes
Into Glacier directlyNo (lifecycle)YesNo (lifecycle)Only if the client sets the storage classOnly through the IAM role’s writes (normally Standard, then lifecycle)
Other destinations—EFS, FSx, other cloudsExport from S3—EFS
Cost modelPer GB written (capped) + S3Per GB copiedPer job + extra days + shippingExtra per GB, only if fasterPer protocol-hour + per GB
Pick it whenOn-premises apps keep using a shareMigrate or sync servers, schedule, verifyNetwork would take more than about a weekFar-away users upload to one bucketOutsiders send files with file transfer protocols

Try the chooser: change the data size and the link to see where the network stops being the answer, and change who sends the data to see the protocol decide.

Which bulk path to S3?

Describe the transfer. The recommendation, the network-time estimate and the reason change as you click.

1. How much data?

2. Internet or Direct Connect link

3. One-off or ongoing?

4. Who sends the data?

5. Already a Snowball customer (ordered before November 2025)?

Over the network at 80%

23 days

By Snowball (≈)

9.3 days(1 devices)

AWS Snowball Edge (offline)

About 23 days over the network is longer than a week, so ship it: 1 Storage Optimized device(s) of 210 TB, filled in parallel, about 9.3 days door to door. Import lands in S3; add a lifecycle rule if it must end in Glacier.

Exam keywords: weeks or months over the network, limited bandwidth, offline, petabytes, remote site

#What goes wrong

  • Choosing by habit, not by question. “We need to move data” is not enough. Ask: online or offline? once or forever? do local apps need a share? who sends it? Each of the five answers a different combination.
  • Snowball straight to Glacier. Not possible; import to S3 and set a 0-day lifecycle rule before shipping the device back.
  • Deleting the source too early. Keep the original data until the import or the DataSync verification report confirms every file. A lost device or a failed file must be re-sent.
  • A gateway cache that is too small. Every read becomes a slow miss, and uploads that pile up during an outage can fill the cache and slow down writes. Size the cache for the weekly working set plus upload backlog.
  • Writing to the bucket behind the gateway’s back. If another process puts objects directly into the bucket, the gateway does not list them until its cache is refreshed (manually, on a schedule or by an automatic refresh setting). And never let two gateways write the same prefix: there is no file locking between them.
  • Archived objects behind a file share. Lifecycle moves to Glacier Flexible Retrieval or Deep Archive make files unreadable through the share until restored. Choose transition ages that match how people really read.
  • Copying files that are still changing. DataSync verification will flag them; run an incremental pass after users stop writing, and plan a short freeze for cut-over.
  • Saturating the office link. A DataSync task can use 10 Gbps; set a bandwidth limit or schedule it outside working hours.
  • Bucket names with dots and Transfer Acceleration. They cannot be accelerated. You cannot rename a bucket; create a new one and copy.
  • Expecting acceleration to fix a small link. It improves a long path; it cannot create bandwidth.
  • Plain FTP on the internet. Transfer Family does not allow it; use SFTP or FTPS, or an internal VPC endpoint over VPN or Direct Connect.
  • Idle Transfer Family endpoints. Each enabled protocol is billed every hour, even with no traffic. Delete test endpoints and unused protocols.

#In system design

  • Migration in two phases: a bulk move (DataSync over the network, or Snowball if the network would take weeks) followed by continuous sync (scheduled DataSync) until cut-over. For huge volumes, the link itself is often upgraded temporarily with Direct Connect (lesson 10.10, chapter 22).
  • Hybrid for the long term: S3 File Gateway for applications that must keep a file share on site; Volume Gateway for block volumes; Tape Gateway to retire the tape library. Put the gateway’s traffic on Direct Connect or VPN and reach S3 through a VPC Interface endpoint (lesson 10.10) if it must not cross the internet.
  • Ingestion from people and partners: Transfer Family for partners and legacy protocols, Transfer Acceleration plus multipart upload for users far away, pre-signed URLs (lesson 10.12) for browser uploads from your own app.
  • Archive cheaply: whatever brings the data in, a lifecycle rule moves it to Glacier Flexible Retrieval or Deep Archive. DataSync is the one bulk path that can write there directly.
  • Process on arrival: every path ends with objects in S3, so S3 events or EventBridge (lesson 10.7) can start processing as soon as a file lands.

#Try it yourself

S3 File Gateway (after deploying and activating the gateway VM):

# Create an NFS share on the gateway that maps to a bucket, writing as S3 Standard
aws storagegateway create-nfs-file-share \
  --client-token pacs-share-1 \
  --gateway-arn arn:aws:storagegateway:us-east-1:111122223333:gateway/sgw-12A3456B \
  --role arn:aws:iam::111122223333:role/FileGatewayS3Access \
  --location-arn arn:aws:s3:::hospital-scans-example \
  --default-storage-class S3_STANDARD \
  --client-list 10.20.0.0/16

# Make the gateway notice objects written to the bucket by something else
aws storagegateway refresh-cache --file-share-arn <share-arn>

# On a Linux server on premises: mount it like any NFS share
sudo mount -t nfs -o nolock,hard 10.20.1.15:/hospital-scans-example /mnt/scans

A lifecycle rule that archives the share’s files (also the Snowball-to-Glacier trick, with Days 0):

aws s3api put-bucket-lifecycle-configuration --bucket hospital-scans-example \
  --lifecycle-configuration '{"Rules":[{"ID":"archive-scans","Status":"Enabled",
    "Filter":{"Prefix":"scans/"},
    "Transitions":[{"Days":180,"StorageClass":"DEEP_ARCHIVE"}]}]}'

DataSync, from an on-premises NFS server to S3, nightly, with verification:

aws datasync create-location-nfs --server-hostname 10.20.1.40 \
  --subdirectory /photos --on-prem-config AgentArns=<agent-arn>
aws datasync create-location-s3 --s3-bucket-arn arn:aws:s3:::news-photos-example \
  --s3-config BucketAccessRoleArn=arn:aws:iam::111122223333:role/DataSyncS3Access
aws datasync create-task --name photos-nightly \
  --source-location-arn <nfs-location-arn> --destination-location-arn <s3-location-arn> \
  --schedule ScheduleExpression="cron(0 0 * * ? *)" \
  --options VerifyMode=ONLY_FILES_TRANSFERRED,TransferMode=CHANGED,BytesPerSecond=37500000
aws datasync start-task-execution --task-arn <task-arn>
aws datasync describe-task-execution --task-execution-arn <execution-arn>

(BytesPerSecond=37500000 is 300 Mbps.)

Snowball (existing customers):

aws snowball create-job --job-type IMPORT \
  --resources '{"S3Resources":[{"BucketArn":"arn:aws:s3:::studio-footage-example"}]}' \
  --address-id <address-id> --role-arn arn:aws:iam::111122223333:role/SnowballImport \
  --snowball-type V3_5S --snowball-capacity-preference T240 --shipping-option SECOND_DAY
aws snowball describe-job --job-id <job-id>
aws snowball get-job-unlock-code --job-id <job-id>

Transfer Acceleration:

aws s3api put-bucket-accelerate-configuration --bucket media-central-example \
  --accelerate-configuration Status=Enabled
aws s3api get-bucket-accelerate-configuration --bucket media-central-example

# Upload through the accelerated endpoint (the CLI uses multipart for big files)
aws s3 cp ./shoot-0425.zip s3://media-central-example/sydney/ \
  --endpoint-url https://s3-accelerate.amazonaws.com
# or set it once: aws configure set default.s3.use_accelerate_endpoint true

Transfer Family, an SFTP endpoint with one partner locked into its own prefix:

aws transfer create-server --protocols SFTP --identity-provider-type SERVICE_MANAGED \
  --endpoint-type PUBLIC --domain S3
aws transfer create-user --server-id s-1234567890abcdef0 --user-name partner-a \
  --role arn:aws:iam::111122223333:role/TransferPartnerAccess \
  --home-directory-type LOGICAL \
  --home-directory-mappings '[{"Entry":"/","Target":"/bank-inbound-example/inbound/partner-a"}]' \
  --ssh-public-key-body "ssh-ed25519 AAAAC3Nza... partner-a"

# What the partner runs (unchanged from before)
sftp -i partner-a.key partner-a@s-1234567890abcdef0.server.transfer.us-east-1.amazonaws.com

#Summary

Key takeaways

  • The 13 paths to S3: network routes (10.10), bulk paths (this lesson), application paths (10.12). The five bulk paths answer different questions; ask online or offline, once or ongoing, file share or not, who sends.
  • S3 File Gateway: an NFS (v3, v4.1) or SMB (v2, v3) share on premises, run as a VM or EC2 instance; local cache for recent files; one file = one object; Standard / IA / One Zone-IA / Intelligent-Tiering, then lifecycle to Glacier; IAM role per share; AD for SMB. Permanent, stays in the data path.
  • Other gateways: Volume (iSCSI block, cached = data in AWS, stored = data on premises, EBS snapshots), Tape (VTL over iSCSI, tapes in S3, ejected to Glacier/Deep Archive), FSx File (closed to new customers since October 2024).
  • DataSync: agent-based (none needed AWS-to-AWS or S3-to-other-cloud) copy and sync; NFS, SMB, HDFS, object storage, other clouds → S3 (any class, Glacier directly), EFS, FSx; about 10 Gbps per task; schedules, incremental, bandwidth limit, verification, metadata kept; per-GB pricing. Copy, not extension.
  • Snowball Edge: offline devices (210 TB Storage Optimized, 28 TB Compute Optimized); order → ship → copy locally → return → import; KMS-encrypted; free import into S3. More than a week over the network → ship it. Cannot import into Glacier: import to S3, lifecycle rule at 0 days. Snowcone and Snowmobile are retired, Snowball is for existing customers only since November 2025, but the exam still expects these answers.
  • Transfer time = bits ÷ usable speed: 100 TB at 1 Gbps and 80% ≈ 11.6 days.
  • Transfer Acceleration: s3-accelerate endpoint → nearest edge location → AWS backbone; no dots in the bucket name; +$0.04 or $0.08 per GB, not charged when not faster; best with multipart for long distances.
  • Transfer Family: managed SFTP, FTPS, FTP, AS2 (and web apps) into S3 or EFS; FTP only inside a VPC; service-managed (SFTP), AD or custom identity; IAM role + logical home directory per user; billed per protocol-hour + per GB.