10.11
Paths to S3 (2/3): Moving Data in Bulk
S3 File Gateway, DataSync, Snowball, Transfer Acceleration and Transfer Family: which one fits which migration or daily feed.
30 min Reading + animations
Tip: tap or hover any dotted-underlined term to see what it means.
By the end of this lesson you will be able to
- Place the five bulk paths of this lesson on the 13-path map of how data reaches S3, and say in one sentence what each one is for.
- Explain how S3 File Gateway turns an NFS or SMB share into objects in a bucket, with a local cache, Active Directory and lifecycle rules, and recognise the other Storage Gateway types (Volume, Tape, FSx File).
- Describe how DataSync migrates and synchronises data with an agent, schedules, incremental copies and verification, including AWS-to-AWS and other-cloud transfers.
- Calculate how long a transfer takes over a network link, decide when shipping Snowball devices is faster, and use a lifecycle rule to land Snowball data in Glacier.
- Explain how S3 Transfer Acceleration uses edge locations, its requirements and price, and when it does not help.
- Set up a managed SFTP, FTPS, FTP or AS2 endpoint with AWS Transfer Family, map each partner to its own prefix, and choose the right identity provider.
- Choose between the five for a scenario, using the questions that separate them: one-off or ongoing, online or offline, file protocol or not, who sends the data.
#Where this lesson sits on the map
The owner asked for one place where every way data moves into or out of S3 is laid out side by side, because these services are easy to confuse. There are 13 paths, split over three lessons. Lesson 10.10 covered the private network routes (1 Gateway endpoint, 2 Interface endpoint, 3 VPC Peering, 4 Transit Gateway, 5 Site-to-Site VPN or Direct Connect): they decide which road the packets take. This lesson covers the bulk paths (6 S3 File Gateway, 7 AWS DataSync, 8 AWS Snowball, 9 S3 Transfer Acceleration, 10 AWS Transfer Family): services that move a lot of data for you, each in its own way. Lesson 10.12 covers the application paths (11 multipart upload, 12 pre-signed URLs, 13 Batch Operations) and ends with a chooser across all 13.
The five in this lesson answer different questions, and that is the key to never mixing them up:
| # | Path | The question it answers | Online or offline | One-off or ongoing |
|---|---|---|---|---|
| 6 | S3 File Gateway | “My servers on site must keep using a file share, but I want the data in S3.” | Online | Ongoing (it stays in the data path) |
| 7 | DataSync | “Copy these terabytes from here to there, quickly, safely, maybe every night.” | Online | One-off migration or scheduled |
| 8 | Snowball | “The network would take weeks or months. Can I post the data?” | Offline | One-off (per job) |
| 9 | Transfer Acceleration | “Users far from the bucket upload slowly over the internet.” | Online | Ongoing |
| 10 | Transfer Family | “Partners send files with SFTP, FTPS, FTP or AS2, and I don’t want to run that server.” | Online | Ongoing |
Every one of these services is taught in more depth later in the course: Snowball, Storage Gateway, Transfer Family and DataSync in chapter 12 (Snowball Edge, Storage Gateway, Transfer Family, DataSync and choosing), and large migrations in chapter 23 (moving large amounts of data). Transfer Acceleration also appears as a performance tool in lesson 10.6. This lesson is complete on its own: what each one is, how data flows, its limits and price, a real example, and why you would pick it over the other four.
#Path 6: Amazon S3 File Gateway
#Why hybrid storage needs a translator
Many companies run hybrid cloud: part of their systems in AWS, part on premises. The reasons are ordinary ones: a migration that takes years, security or compliance rules that keep some systems in the building, or simply an IT strategy. The servers that stay on premises still need storage, and S3 is attractive: unlimited, eleven nines of durability, cheap archive classes.
But S3 is object storage with its own HTTPS API (PUT, GET, list requests), not a file system. A Linux server mounts file shares with NFS, a Windows server with SMB, and old applications have no idea how to call an S3 API. Unlike EFS, which speaks NFS natively (lesson 5.4), S3 needs a translator. That translator is AWS Storage Gateway: a bridge between on-premises data and cloud storage, used for backup and restore, disaster recovery, tiered storage and a low-latency local cache. It comes in several types; the one that puts files into S3 is Amazon S3 File Gateway.
#How a file becomes an object
S3 File Gateway is a virtual appliance you run on premises: a VM on VMware, Microsoft Hyper-V, Linux KVM or Nutanix, or an EC2 instance from an AMI. You give it a cache disk and connect it to one or more S3 buckets. Then:
- You create a file share on the gateway: an NFS export (NFS v3 or v4.1) or an SMB share (SMB v2 or v3). Each share maps to one bucket, or to a prefix in a bucket. One gateway can hold up to 50 shares.
- Servers mount the share like any NAS. When an application writes a file, the gateway stores it on the local cache and confirms at once, so the application gets local-disk speed.
- In the background the gateway uploads the file over HTTPS (on the internet, or over Direct Connect or a VPN from lesson 10.10) to S3. One file becomes one object; the path becomes the key, and file metadata such as owner, permissions and timestamps is kept in the object’s metadata.
- Reads of recently used files are served from the cache (a cache hit). Reads of anything else are fetched from S3 (a cache miss), then cached.
- The gateway writes into S3 Standard, Standard-IA, One Zone-IA or Intelligent-Tiering. It cannot write directly into the Glacier classes; a normal lifecycle rule on the bucket (lesson 10.5) moves old objects there. An object that has gone to Glacier Flexible Retrieval or Deep Archive must be restored before the share can read it again.
- The gateway reaches the bucket with an IAM role you give it (one per share). For SMB shares it can join your Active Directory, so users log in with their normal domain accounts and Windows permissions apply.
Because the data lands as plain objects, everything in AWS can use it: analytics with Athena, machine learning, replication to another Region, versioning, Object Lock. That is the big advantage over a NAS: the files are on premises for the people who need them and in the cloud for everything else.
Limits and numbers. Cache from 150 GiB up to 64 TiB per gateway. Up to 50 file shares per gateway. Maximum file size 5 TiB (an S3 object’s limit). Cost: about $0.01 per GB written to AWS, capped at $125 per gateway per month (the first 100 GB per account are free), plus normal S3 storage and request charges; the VM runs on your own hardware, or you pay for the EC2 instance.
Watch a write, a cache hit and a miss, the lifecycle move, the Active Directory check and a broken internet link:
S3 File Gateway: a file share backed by a bucket
A hospital’s scan archive writes to an ordinary NFS share. The share is a gateway VM that keeps recent files in a local cache and stores every file as an object in S3.
Scenario
The archive saves a new scan to what it thinks is a local file server.
The scan archive mounts the share over NFS, exactly as it would mount a NAS. It has no idea S3 exists, and it does not need an SDK or any code change. That is the whole point of a file gateway.
#The rest of the Storage Gateway family
S3 File Gateway is one of four gateway types. You need the others only as context for this lesson; chapter 12 covers them in depth.
- Volume Gateway presents block storage (disks, not files) to your servers over iSCSI, backed by S3 and backed up as EBS snapshots. Two modes:
- Cached volumes: the full data lives in AWS; only frequently used data is cached on premises for low latency. Good when the primary copy should be in the cloud.
- Stored volumes: the full data stays on premises for the lowest latency, and the gateway sends asynchronous point-in-time backups to AWS as EBS snapshots. Good when everything must stay local but you want off-site backup.
- Tape Gateway replaces physical tapes (see the analogy below). It is a VTL, a virtual tape library, that your backup software sees over iSCSI as tape drives and a media changer. Virtual tapes are stored in S3; when the backup software ejects (archives) a tape, it moves to S3 Glacier Flexible Retrieval or Glacier Deep Archive.
- Amazon FSx File Gateway gave offices a local cache in front of an FSx for Windows File Server share. It has been closed to new customers since 28 October 2024; existing customers can keep using it.
| Exam phrase | Gateway type |
|---|---|
| On-premises servers need NFS or SMB access to files stored in S3 | S3 File Gateway |
| Keep all data on premises for low latency, back it up to AWS | Volume Gateway, stored volumes |
| Primary data in AWS, only frequently used data on premises | Volume Gateway, cached volumes |
| Replace physical backup tapes, keep the existing backup software | Tape Gateway |
| Windows file shares with Active Directory, but no S3 in the question | FSx for Windows File Server (chapter 12), not a gateway |
#Path 7: AWS DataSync
#What it is, and how it moves data
AWS DataSync is a managed service for copying large amounts of data over the network, quickly and safely, once or on a schedule. Think of it as a professional removal company: it packs, moves, checks nothing is broken, and leaves. Unlike a gateway, it does not stay in the data path.
- Sources and destinations. On premises: NFS, SMB, HDFS (Hadoop) and self-managed object storage that speaks the S3 API. In AWS: S3 (any storage class, including Glacier Flexible Retrieval and Deep Archive), EFS, and every FSx file system. Other clouds: Google Cloud Storage, Azure Blob Storage and Azure Files, and S3-compatible stores such as Wasabi, Oracle, Cloudflare R2 and others.
- The agent. For on-premises storage you deploy a DataSync agent: a VM (VMware, Hyper-V, KVM) or an EC2 instance that sits near the storage and reads it over NFS, SMB or HDFS. Nothing is installed on the file server itself. No agent is needed between AWS storage services in the same account, or between S3 and object storage in another cloud.
- The transfer. A task names a source location, a destination location and options. The agent reads in parallel and sends data over TLS with AWS’s own protocol, which compresses in flight, detects sparse files and keeps a long, busy link full. One task can use about 10 Gbps; several agents and tasks can run in parallel for bigger jobs.
- Incremental. By default a run copies only data that is new or has changed since the last run, so the first run moves everything and later runs move only the difference.
- Scheduling. Tasks can run hourly, daily or weekly (the notes’ example: every night at 12).
- Bandwidth limit. You can cap a task’s bandwidth, for example during office hours.
- Verification. DataSync compares checksums of what it read with what it wrote and reports any file that does not match. You can verify only what was transferred, the whole destination, or skip it.
- Metadata. File permissions and metadata (NFS POSIX ownership and permissions, SMB attributes and ACLs, timestamps) are preserved, which matters when you copy into EFS or FSx and users must keep their permissions.
- Filters and reports. Include or exclude patterns, and task reports that list every file transferred, skipped or failed.
Cost. You pay per GB copied: $0.0125 per GB in Basic mode, or $0.015 per GB plus $0.55 per task execution in Enhanced mode, a newer mode for S3, other-cloud and NFS/SMB-to-S3 transfers that scales to virtually unlimited numbers of files. Normal S3 request charges and any data transfer out of the source also apply. There is no charge for the agent itself.
DataSync: copy, verify, repeat
A newspaper moves 200 TB of photo archives from its NAS to S3. An agent VM next to the NAS reads the files; the DataSync service writes them into AWS and checks every byte.
Scenario
You deploy the DataSync agent as a VM on the office’s hypervisor and activate it. The agent mounts the NAS read-only over NFS or SMB: no software goes on the NAS itself.
#DataSync or Storage Gateway?
This is the pair the owner’s notes warn about, and the difference is in one word: copy versus extension.
DataSync: a removal companyStorage Gateway: a permanent bridge
- Copies data from A to B, once or on a schedule, then steps out of the way
- Applications switch to the new location after the migration
- Agent reads the source; tasks, schedules, verification, reports
- Moves on premises → AWS, AWS → AWS (S3 ↔ EFS ↔ FSx), other clouds → AWS
- Writes into any S3 class, including Glacier, directly
- Exam words: migrate, scheduled transfer, sync, agent, one-time
- Extends on-premises storage into AWS, permanently; it stays in the data path
- Applications keep using a local NFS, SMB or iSCSI target every day
- Local cache for low-latency access to recent data
- Only on premises ↔ AWS
- File Gateway writes into Standard, IA or Intelligent-Tiering; lifecycle to Glacier
- Exam words: hybrid, on-premises access, local cache, NFS/SMB to S3, tapes
They are often used together: DataSync for the big initial copy of years of files, then a File Gateway for the applications that keep writing new files.
Self-checkQuestion
A company must move 50 TB of files from an on-premises NFS server to Amazon EFS over its existing Direct Connect link within two weeks, keep POSIX permissions, and decommission the server afterwards. Which service fits best?
#Path 8: AWS Snowball
#Why ship data on a truck
Sometimes the network is the problem. The link is slow, shared with the whole office, expensive, unstable, or simply does not exist (a ship, a mine, a disaster zone). Then it is faster to put the data on a device and post it. That is AWS Snowball: secure, rugged, portable devices for offline data migration into and out of AWS, and for computing at the edge. It handles terabytes to petabytes.
The current devices are Snowball Edge:
| Device | Storage | Compute | Use |
|---|---|---|---|
| Snowball Edge Storage Optimized | 210 TB NVMe | 104 vCPU, 416 GB | Large data migrations; up to about 1.5 GB/s transfer |
| Snowball Edge Compute Optimized | 28 TB NVMe | 104 vCPUs, 416 GB memory | Edge computing: run EC2 instances or Lambda functions on site |
How a job works:
- In the console you create a job: import into S3, export from S3, or local compute and storage only. You choose the bucket and the delivery address.
- AWS ships the device. Its E Ink screen shows the shipping label.
- You connect it to your local network (10 to 100 Gb Ethernet), unlock it with the job’s manifest file and unlock code, and copy data with the S3-compatible endpoint or an NFS mount. At local network speed even 200 TB fills in a day or two.
- You power it off; the E Ink label switches to the return address; the courier collects it.
- AWS imports the data into your bucket, then wipes the device to NIST standards. You follow the job status in the console.
Security. Everything is encrypted with 256-bit keys managed in AWS KMS; the keys are never stored on the device. The device is tamper-resistant, and losing it in transit exposes nothing without the manifest and unlock code.
Cost. A service fee per job (which includes a number of days on site), a daily fee for extra days, and shipping. Importing data into S3 has no per-GB transfer charge; exporting out of S3 is charged per GB.
#How long would the network take?
The calculation is simple and appears in many exam questions:
time = data in bits ÷ usable link speed
1 TB is 8,000 gigabits (decimal units, as networks count them). Links rarely run at 100% because other traffic shares them, so plan for about 80%:
| Data | 100 Mbps | 1 Gbps | 10 Gbps |
|---|---|---|---|
| 10 TB | 11.6 days | 1.2 days (28 hours) | 2.8 hours |
| 100 TB | 116 days | 11.6 days | 1.2 days |
| 1 PB | 3.2 years | 116 days | 11.6 days |
Worked example: 100 TB over 1 Gbps = 800,000 gigabits ÷ 0.8 Gbps = 1,000,000 seconds = 11.6 days (9.3 days even at a perfect 100%). With Snowball: 2 to 3 days for the device to arrive, about a day to fill at local speed, 2 to 3 days to ship back, a day to import: about a week, whatever the size up to a device’s 210 TB, because several devices fill in parallel.
The rule of thumb: if it takes more than about a week over the network, use Snowball.
Self-checkQuestion
A company has 300 TB to move to S3 once, over a 500 Mbps internet link that it can use at about 80%. Roughly how long would the network transfer take, and what does the exam expect?
#Snowball cannot import straight into Glacier
A favourite exam trap. You have a pile of old data that only needs to be archived. Can the Snowball job put it directly into S3 Glacier or Glacier Deep Archive? No. A Snowball import always lands in S3 (Standard by default). The solution:
- Import into S3 Standard as usual.
- Before you ship the device back, add an S3 lifecycle rule on the bucket (or prefix) that transitions objects to Glacier Flexible Retrieval or Deep Archive after 0 days.
- Objects move to the archive class within a day or two of import, so you pay for Standard storage only briefly.
Compare DataSync, which can write into the Glacier classes directly. And remember S3 File Gateway writes only into Standard, IA or Intelligent-Tiering. In both offline and gateway cases, the lifecycle rule is the bridge to Glacier.
#Edge computing with Snowball Edge
Edge computing means processing data where it is created, at a location with limited internet and no computing power: a truck on the road, a ship at sea, a mining station underground. A Snowball Edge (Compute Optimized is built for it; Storage Optimized can also run small workloads) runs EC2 instances and Lambda functions on site, for example to pre-process sensor data, run machine learning inference or transcode video, and later ships the results back to AWS.
Snowball: when the road is faster than the wire
A film studio has 400 TB of raw footage and a 1 Gbps link shared by the whole office. AWS posts a rugged storage device; the studio fills it and posts it back.
Scenario
In the console the studio creates an import job: which bucket, which address. AWS prepares a device with the job’s encryption keys, which are managed in KMS and never stored on the device.
#The Snow Family today, and what the exam expects
The Snow Family has changed a lot, and the notes and many practice questions describe the old line-up:
- AWS Snowcone (the small, portable device: 8 TB HDD or 14 TB SSD, for drones, fire stations, factories and places with little space or no internet) was discontinued on 12 November 2024.
- AWS Snowmobile (a shipping-container truck holding up to 100 PB, for exabyte-scale moves) was retired in 2024.
- The older Snowball Edge models (80 TB Storage Optimized, 52-vCPU and GPU Compute Optimized) were discontinued in November 2024 and replaced by the 210 TB and 28 TB models above.
- Since 7 November 2025, Snowball Edge is available only to existing customers: AWS no longer offers any Snow device to new customers. AWS points new customers to DataSync for online transfers (with a temporary hosted Direct Connect connection for speed), AWS Data Transfer Terminal (physical AWS locations where you bring your own drives and upload at high speed) or partner offline services, and to AWS Outposts for edge computing.
#Path 9: S3 Transfer Acceleration
#Taking the AWS motorway early
Over long distances the public internet is slow for big uploads. Each packet crosses many networks; the round trip from Sydney to Virginia is about 200 ms; some packets are lost; and TCP slows down every time it sees loss. A fast office line may deliver only a fraction of its speed to a faraway bucket.
S3 Transfer Acceleration fixes the long part. You enable it on the bucket, and clients use a special endpoint, bucketname.s3-accelerate.amazonaws.com (or bucketname.s3-accelerate.dualstack.amazonaws.com for IPv6). DNS sends each client to the nearest CloudFront edge location, a short hop away. From there the data travels to the bucket’s Region over the AWS private backbone, on optimised, already-warm connections. The bucket, its objects and its Region do not change; only the path does.
- Requirements: the bucket name must be DNS-compliant with no periods (dots); requests must use virtual-hosted style; only the bucket owner (or someone with
s3:PutAccelerateConfiguration) can enable it; it can take up to 20 minutes before speeds improve. It works on general purpose buckets in most commercial Regions. - Combine with multipart upload (lesson 10.12): every part rides the accelerated path in parallel. AWS reports gains from about 50% to several hundred percent for long distances.
- Cost: on top of normal S3 charges, $0.04 per GB for uploads through edges in the United States, Europe and Japan, $0.08 per GB through all other edges; downloads $0.04 per GB. If AWS determines that an accelerated transfer was not faster than a normal one, you are not charged the acceleration fee for it. The Speed Comparison tool shows the expected gain from your location.
- When it does not help: users close to the bucket’s Region, small objects where latency dominates, or a link that is simply too small.
Transfer Acceleration: get on the AWS network early
A photographer in Sydney uploads 2 GB of photos a day to a bucket in us-east-1. Two routes: across the public internet the whole way, or a short hop to the Sydney edge location and then AWS’s own backbone.
Scenario
Without acceleration the upload goes to bucket.s3.us-east-1.amazonaws.com. Every packet crosses the Pacific through many networks AWS does not control.
Transfer Acceleration uses CloudFront’s edge locations, but it is not CloudFront: CloudFront is a cache in front of content that many people download (chapter 11), while acceleration is mainly a faster on-ramp for uploads and downloads of individual objects. A related idea, Global Accelerator, routes any TCP or UDP traffic through the AWS backbone (also chapter 11). For performance tuning of S3 overall, see lesson 10.6.
#Path 10: AWS Transfer Family
#Managed file transfer servers
Thousands of business processes still exchange files with old, trusted protocols: a bank receiving payment files from partners, a retailer exchanging orders with suppliers, a hospital receiving lab results. The partners have scripts and tools that speak SFTP, FTPS or FTP, and changing them is out of the question. Traditionally you ran your own SFTP server: patch it, scale it, back it up, copy files from it into storage.
AWS Transfer Family is a fully managed file transfer service that puts a server endpoint in front of Amazon S3 or Amazon EFS:
- Protocols: SFTP (over SSH), FTPS (FTP over TLS), FTP (unencrypted), and AS2 (a business-to-business standard with signed receipts, common in supply chains and payments). Since late 2024 there are also Transfer Family web apps, a browser-based way for business users to upload and download files in S3.
- Endpoints: a public endpoint (SFTP only), or a VPC endpoint that is either internal or internet-facing with Elastic IP addresses and security groups. Plain FTP is allowed only on an internal VPC endpoint, never over the internet. You can put your own DNS name in front with Route 53. Endpoints run across up to three AZs and scale automatically.
- Identity: service-managed users (SSH keys stored in the service; SFTP only), AWS Directory Service for Microsoft AD, or a custom identity provider through Lambda or API Gateway (LDAP, Okta, Amazon Cognito, anything else).
- Access: each user has an IAM role that decides what the server may do in the bucket on that user’s behalf, an optional session policy, and a home directory. Logical directories map the user’s “/” to a prefix such as
inbound/partner-a/(a chroot), so users never see the bucket name or each other. - After the upload: the file is an ordinary object, so S3 event notifications (lesson 10.7) or Transfer Family managed workflows (copy, tag, decrypt, scan, or any Lambda step) can process it at once. SFTP connectors let AWS send files to a partner’s SFTP server too.
- Cost: per hour for each protocol enabled on an endpoint (about $0.30 per hour in us-east-1, even when idle), plus about $0.04 per GB uploaded or downloaded; AS2 is billed per message, web apps per unit-hour.
Transfer Family: SFTP without an SFTP server
A bank receives payment files from partner companies over SFTP. The partners keep their scripts; the bank swaps its old SFTP server for a managed endpoint that writes into S3.
Scenario
Partner A’s nightly script connects to sftp.bank.example (a Route 53 name for the endpoint) and logs in with its SSH key, exactly as it did with the old server.
#The five side by side
| S3 File Gateway | DataSync | Snowball Edge | Transfer Acceleration | Transfer Family | |
|---|---|---|---|---|---|
| What it is | Local file share backed by S3 | Managed copy and sync service | Physical device shipped to you | Faster path for uploads via edge locations | Managed SFTP / FTPS / FTP / AS2 endpoint |
| Protocol in | NFS, SMB | NFS, SMB, HDFS, S3 API (agent reads) | Copy to device locally (NFS, S3-compatible) | HTTPS to the S3 API | SFTP, FTPS, FTP, AS2, browser |
| How data reaches S3 | Gateway uploads over HTTPS | Agent → DataSync over TLS | Courier → AWS import | Nearest edge → AWS backbone | Endpoint writes with the user’s IAM role |
| Online or offline | Online | Online | Offline | Online | Online |
| One-off or ongoing | Ongoing, permanent | Both: one-off or scheduled | One-off per job | Ongoing | Ongoing |
| Local cache | Yes | No | The device is the “cache” in transit | No | No |
| Stays in the data path | Yes | No | No | Yes (per request) | Yes |
| Into Glacier directly | No (lifecycle) | Yes | No (lifecycle) | Only if the client sets the storage class | Only through the IAM role’s writes (normally Standard, then lifecycle) |
| Other destinations | — | EFS, FSx, other clouds | Export from S3 | — | EFS |
| Cost model | Per GB written (capped) + S3 | Per GB copied | Per job + extra days + shipping | Extra per GB, only if faster | Per protocol-hour + per GB |
| Pick it when | On-premises apps keep using a share | Migrate or sync servers, schedule, verify | Network would take more than about a week | Far-away users upload to one bucket | Outsiders send files with file transfer protocols |
Try the chooser: change the data size and the link to see where the network stops being the answer, and change who sends the data to see the protocol decide.
Which bulk path to S3?
Describe the transfer. The recommendation, the network-time estimate and the reason change as you click.
1. How much data?
2. Internet or Direct Connect link
3. One-off or ongoing?
4. Who sends the data?
5. Already a Snowball customer (ordered before November 2025)?
Over the network at 80%
23 days
By Snowball (≈)
9.3 days(1 devices)
AWS Snowball Edge (offline)
About 23 days over the network is longer than a week, so ship it: 1 Storage Optimized device(s) of 210 TB, filled in parallel, about 9.3 days door to door. Import lands in S3; add a lifecycle rule if it must end in Glacier.
Exam keywords: weeks or months over the network, limited bandwidth, offline, petabytes, remote site
#What goes wrong
- Choosing by habit, not by question. “We need to move data” is not enough. Ask: online or offline? once or forever? do local apps need a share? who sends it? Each of the five answers a different combination.
- Snowball straight to Glacier. Not possible; import to S3 and set a 0-day lifecycle rule before shipping the device back.
- Deleting the source too early. Keep the original data until the import or the DataSync verification report confirms every file. A lost device or a failed file must be re-sent.
- A gateway cache that is too small. Every read becomes a slow miss, and uploads that pile up during an outage can fill the cache and slow down writes. Size the cache for the weekly working set plus upload backlog.
- Writing to the bucket behind the gateway’s back. If another process puts objects directly into the bucket, the gateway does not list them until its cache is refreshed (manually, on a schedule or by an automatic refresh setting). And never let two gateways write the same prefix: there is no file locking between them.
- Archived objects behind a file share. Lifecycle moves to Glacier Flexible Retrieval or Deep Archive make files unreadable through the share until restored. Choose transition ages that match how people really read.
- Copying files that are still changing. DataSync verification will flag them; run an incremental pass after users stop writing, and plan a short freeze for cut-over.
- Saturating the office link. A DataSync task can use 10 Gbps; set a bandwidth limit or schedule it outside working hours.
- Bucket names with dots and Transfer Acceleration. They cannot be accelerated. You cannot rename a bucket; create a new one and copy.
- Expecting acceleration to fix a small link. It improves a long path; it cannot create bandwidth.
- Plain FTP on the internet. Transfer Family does not allow it; use SFTP or FTPS, or an internal VPC endpoint over VPN or Direct Connect.
- Idle Transfer Family endpoints. Each enabled protocol is billed every hour, even with no traffic. Delete test endpoints and unused protocols.
#In system design
- Migration in two phases: a bulk move (DataSync over the network, or Snowball if the network would take weeks) followed by continuous sync (scheduled DataSync) until cut-over. For huge volumes, the link itself is often upgraded temporarily with Direct Connect (lesson 10.10, chapter 22).
- Hybrid for the long term: S3 File Gateway for applications that must keep a file share on site; Volume Gateway for block volumes; Tape Gateway to retire the tape library. Put the gateway’s traffic on Direct Connect or VPN and reach S3 through a VPC Interface endpoint (lesson 10.10) if it must not cross the internet.
- Ingestion from people and partners: Transfer Family for partners and legacy protocols, Transfer Acceleration plus multipart upload for users far away, pre-signed URLs (lesson 10.12) for browser uploads from your own app.
- Archive cheaply: whatever brings the data in, a lifecycle rule moves it to Glacier Flexible Retrieval or Deep Archive. DataSync is the one bulk path that can write there directly.
- Process on arrival: every path ends with objects in S3, so S3 events or EventBridge (lesson 10.7) can start processing as soon as a file lands.
#Try it yourself
S3 File Gateway (after deploying and activating the gateway VM):
# Create an NFS share on the gateway that maps to a bucket, writing as S3 Standard
aws storagegateway create-nfs-file-share \
--client-token pacs-share-1 \
--gateway-arn arn:aws:storagegateway:us-east-1:111122223333:gateway/sgw-12A3456B \
--role arn:aws:iam::111122223333:role/FileGatewayS3Access \
--location-arn arn:aws:s3:::hospital-scans-example \
--default-storage-class S3_STANDARD \
--client-list 10.20.0.0/16
# Make the gateway notice objects written to the bucket by something else
aws storagegateway refresh-cache --file-share-arn <share-arn>
# On a Linux server on premises: mount it like any NFS share
sudo mount -t nfs -o nolock,hard 10.20.1.15:/hospital-scans-example /mnt/scans
A lifecycle rule that archives the share’s files (also the Snowball-to-Glacier trick, with Days 0):
aws s3api put-bucket-lifecycle-configuration --bucket hospital-scans-example \
--lifecycle-configuration '{"Rules":[{"ID":"archive-scans","Status":"Enabled",
"Filter":{"Prefix":"scans/"},
"Transitions":[{"Days":180,"StorageClass":"DEEP_ARCHIVE"}]}]}'
DataSync, from an on-premises NFS server to S3, nightly, with verification:
aws datasync create-location-nfs --server-hostname 10.20.1.40 \
--subdirectory /photos --on-prem-config AgentArns=<agent-arn>
aws datasync create-location-s3 --s3-bucket-arn arn:aws:s3:::news-photos-example \
--s3-config BucketAccessRoleArn=arn:aws:iam::111122223333:role/DataSyncS3Access
aws datasync create-task --name photos-nightly \
--source-location-arn <nfs-location-arn> --destination-location-arn <s3-location-arn> \
--schedule ScheduleExpression="cron(0 0 * * ? *)" \
--options VerifyMode=ONLY_FILES_TRANSFERRED,TransferMode=CHANGED,BytesPerSecond=37500000
aws datasync start-task-execution --task-arn <task-arn>
aws datasync describe-task-execution --task-execution-arn <execution-arn>
(BytesPerSecond=37500000 is 300 Mbps.)
Snowball (existing customers):
aws snowball create-job --job-type IMPORT \
--resources '{"S3Resources":[{"BucketArn":"arn:aws:s3:::studio-footage-example"}]}' \
--address-id <address-id> --role-arn arn:aws:iam::111122223333:role/SnowballImport \
--snowball-type V3_5S --snowball-capacity-preference T240 --shipping-option SECOND_DAY
aws snowball describe-job --job-id <job-id>
aws snowball get-job-unlock-code --job-id <job-id>
Transfer Acceleration:
aws s3api put-bucket-accelerate-configuration --bucket media-central-example \
--accelerate-configuration Status=Enabled
aws s3api get-bucket-accelerate-configuration --bucket media-central-example
# Upload through the accelerated endpoint (the CLI uses multipart for big files)
aws s3 cp ./shoot-0425.zip s3://media-central-example/sydney/ \
--endpoint-url https://s3-accelerate.amazonaws.com
# or set it once: aws configure set default.s3.use_accelerate_endpoint true
Transfer Family, an SFTP endpoint with one partner locked into its own prefix:
aws transfer create-server --protocols SFTP --identity-provider-type SERVICE_MANAGED \
--endpoint-type PUBLIC --domain S3
aws transfer create-user --server-id s-1234567890abcdef0 --user-name partner-a \
--role arn:aws:iam::111122223333:role/TransferPartnerAccess \
--home-directory-type LOGICAL \
--home-directory-mappings '[{"Entry":"/","Target":"/bank-inbound-example/inbound/partner-a"}]' \
--ssh-public-key-body "ssh-ed25519 AAAAC3Nza... partner-a"
# What the partner runs (unchanged from before)
sftp -i partner-a.key partner-a@s-1234567890abcdef0.server.transfer.us-east-1.amazonaws.com
#Summary
Key takeaways
- The 13 paths to S3: network routes (10.10), bulk paths (this lesson), application paths (10.12). The five bulk paths answer different questions; ask online or offline, once or ongoing, file share or not, who sends.
- S3 File Gateway: an NFS (v3, v4.1) or SMB (v2, v3) share on premises, run as a VM or EC2 instance; local cache for recent files; one file = one object; Standard / IA / One Zone-IA / Intelligent-Tiering, then lifecycle to Glacier; IAM role per share; AD for SMB. Permanent, stays in the data path.
- Other gateways: Volume (iSCSI block, cached = data in AWS, stored = data on premises, EBS snapshots), Tape (VTL over iSCSI, tapes in S3, ejected to Glacier/Deep Archive), FSx File (closed to new customers since October 2024).
- DataSync: agent-based (none needed AWS-to-AWS or S3-to-other-cloud) copy and sync; NFS, SMB, HDFS, object storage, other clouds → S3 (any class, Glacier directly), EFS, FSx; about 10 Gbps per task; schedules, incremental, bandwidth limit, verification, metadata kept; per-GB pricing. Copy, not extension.
- Snowball Edge: offline devices (210 TB Storage Optimized, 28 TB Compute Optimized); order → ship → copy locally → return → import; KMS-encrypted; free import into S3. More than a week over the network → ship it. Cannot import into Glacier: import to S3, lifecycle rule at 0 days. Snowcone and Snowmobile are retired, Snowball is for existing customers only since November 2025, but the exam still expects these answers.
- Transfer time = bits ÷ usable speed: 100 TB at 1 Gbps and 80% ≈ 11.6 days.
- Transfer Acceleration:
s3-accelerateendpoint → nearest edge location → AWS backbone; no dots in the bucket name; +$0.04 or $0.08 per GB, not charged when not faster; best with multipart for long distances. - Transfer Family: managed SFTP, FTPS, FTP, AS2 (and web apps) into S3 or EFS; FTP only inside a VPC; service-managed (SFTP), AD or custom identity; IAM role + logical home directory per user; billed per protocol-hour + per GB.
Answer on your own first, then read the explanation, even when you got it right.
Self-checkQuestion
Which service lets on-premises servers keep reading and writing files over NFS or SMB while the files are stored as objects in an S3 bucket?
Self-checkQuestion
What is the main purpose of the local cache in an S3 File Gateway?
Self-checkQuestion
Which storage classes can an S3 File Gateway write objects into directly?
Self-checkQuestion
A company wants to keep its entire 12 TB database volume on premises for the lowest latency and send point-in-time backups to AWS. Which option fits?
Self-checkQuestion
A bank uses Veeam with a physical tape library and trucks tapes to an off-site vault. It wants to stop handling tapes without changing its backup software. What should it use?
Self-checkQuestion
Which two statements about AWS DataSync are true?
Choose 2
Self-checkQuestion
A media company must copy its on-premises SMB file server to Amazon FSx for Windows File Server, keep Windows ACLs, and then sync changes every hour until cut-over. Which service fits?
Self-checkQuestion
What is the most accurate one-line difference between DataSync and Storage Gateway?
Self-checkQuestion
An on-premises NFS share holds 30 TB. Over a 1 Gbps link at 80% efficiency, about how long does a full DataSync copy take?
Self-checkQuestion
A company has 2 PB to move to S3 and a 10 Gbps Direct Connect link it can use at 80%. Roughly how long would the network take?
Self-checkQuestion
A research lab must archive 150 TB of instrument data in S3 Glacier Deep Archive using Snowball Edge. What is the correct approach?
Self-checkQuestion
Which two situations make Snowball Edge the expected exam answer?
Choose 2
Self-checkQuestion
How is data on a Snowball Edge protected if the device is stolen in transit?
Self-checkQuestion
Which statement about the Snow Family in late 2026 is correct?
Self-checkQuestion
The owner’s notes list Snowball Edge Storage Optimized at 80 TB and Compute Optimized at 42 TB. What are the current models?
Self-checkQuestion
Which endpoint must a client use to upload through S3 Transfer Acceleration to the bucket media-central-example?
Self-checkQuestion
A team cannot enable Transfer Acceleration on the bucket photos.example.com. Why?
Self-checkQuestion
Users in Sydney upload 5 GB files to a bucket in us-east-1. Uploads are slow and often restart. Which two actions improve them most directly?
Choose 2
Self-checkQuestion
An office in Virginia uploads to a bucket in us-east-1 through the accelerate endpoint. What happens with the acceleration fee?
Self-checkQuestion
Which statement about S3 Transfer Acceleration is NOT true?
Self-checkQuestion
A retailer receives order files from 200 suppliers over SFTP to a self-managed EC2 server. It wants a managed, highly available replacement that stores files in S3 without suppliers changing their clients. What should it use?
Self-checkQuestion
Which two statements about AWS Transfer Family are true?
Choose 2
Self-checkQuestion
A partner must only see and write its own folder in a shared bucket through a Transfer Family SFTP endpoint. What achieves this most simply?
Self-checkQuestion
Two retail partners must exchange signed purchase orders and receive signed delivery receipts as required by their contracts. Which Transfer Family protocol fits?
Self-checkQuestion
A company with an existing Snowball account needs to move 1.5 PB once from a data centre with a 1 Gbps link. Which two choices are correct?
Choose 2
Self-checkQuestion
Which pair is matched correctly?
Self-checkQuestion
A company used S3 File Gateway for years. A new analytics job now writes result files directly into the same bucket with the S3 API, but users of the share cannot see them. What is the cause and fix?
Self-checkQuestion
During a DataSync migration, verification reports three files that do not match the source. What is the most likely cause, and the right next step?
Self-checkQuestion
An S3 File Gateway’s users complain that almost every file open is slow, even for files used last week. CloudWatch shows a very low cache hit ratio. What should you do first?
Self-checkQuestion
Which two statements are myths?
Choose 2