Transfer Family, DataSync and the Full Storage Map

By the end of this lesson you will be able to

• Explain what AWS Transfer Family replaces, and follow one session from login through the identity provider, the IAM role and the home directory into S3 or EFS.
• Choose the right endpoint type (public, VPC internet-facing, VPC internal) for SFTP, FTPS, FTP and AS2, and the right identity provider for a set of users.
• Process files after they land with managed workflows, and estimate a Transfer Family bill from endpoint-hours and gigabytes.
• Build a DataSync migration from its parts (agent, locations, task, schedule, execution), explain each phase of a run, and pick task options, task mode and network endpoint.
• Calculate how long a DataSync copy and its nightly runs take, and what they cost, and decide when the network is no longer the answer.
• Tell DataSync from Storage Gateway, Snowball, Transfer Family and S3 Transfer Acceleration in one sentence each, and choose between them from the words of a question.
• Place every storage service of the course (S3, the Glacier classes, EBS, instance store, EFS, the four FSx file systems, Storage Gateway, Transfer Family, DataSync, Snowball, and databases) on one map, and pick the right one for a scenario.

#Where this lesson fits

Lesson 10.11 introduced DataSync as “Path 7” and Transfer Family as “Path 10” of the ways data reaches S3: what each one is, a real example, and a five-way comparison with File Gateway, Snowball and Transfer Acceleration. This lesson is the deep dive. It goes inside a Transfer Family session and a DataSync task, covers the options, limits, failure cases and prices you need in a real project, and then closes chapter 12 with the full storage map of the course.

The other lessons of this chapter: 12.1 Snowball and edge computing, 12.2 the FSx family and 12.3 Storage Gateway. The storage services from chapter 5 (EBS, instance store, EFS) and chapter 10 (S3 and its storage classes) come back in the final map.

#AWS Transfer Family

#Why companies still run file transfer servers

File transfer protocols are older than the cloud and they are everywhere. Banks receive salary and settlement files from employers and wallets, garment exporters exchange purchase orders with European retailers, hospitals receive lab results, government offices publish datasets. The other side has scripts, schedulers and software that speak SFTP, FTPS or FTP, or the business-to-business standard AS2, and nobody will rewrite them for you.

So you run a file transfer server. That means an EC2 instance or a box in the data centre with an SSH daemon, patching every month, a disk that fills up, users and keys in a text file, a script that copies new files into storage, and a single point of failure that wakes someone up at night.

AWS Transfer Family is the managed version: a fully managed service for file transfers into and out of Amazon S3 or Amazon EFS over those protocols. There is no server or software for you to maintain. As the slides put it, the infrastructure is managed, scalable, reliable and highly available (it runs in up to three Availability Zones), you pay per provisioned endpoint per hour plus per gigabyte transferred, users’ credentials can be stored in the service, and it integrates with existing authentication systems.

The slides list the usual uses: sharing files, publishing public datasets, and exchanging data with CRM and ERP systems.

#One session, step by step

Everything in Transfer Family hangs on four objects: a server (the endpoint, with its protocols and endpoint type), an identity provider (who may log in), a user with an IAM role (what that user may do), and a home directory (where the user lands).

1. The client connects to the server’s hostname: either the AWS name s-1234abcd.server.transfer.ap-southeast-1.amazonaws.com or your own name, such as sftp.bank.example, pointed at it with Route 53 (the slides mark Route 53 as optional). An SFTP server proves who it is with its host key; when you migrate from an old server you can import its host key, so clients see the same fingerprint and nothing breaks.
2. The server asks the identity provider whether the username and the key or password are valid.
3. The answer includes the user’s IAM role, an optional session policy and the home directory.
4. The server assumes the role for this session. Every read and write is done with that role’s permissions. The role must trust transfer.amazonaws.com.
5. Uploads are written straight into S3 as objects (or into EFS as files). There is no server disk between the client and the storage.
6. When an upload completes, a managed workflow can process the file, and S3 events (lesson 10.7) can start anything else.

Watch the whole chain, and three ways it breaks:

#Endpoint types: who can reach the server

The endpoint type decides who can reach the server and which protocols you may enable.

Three rules worth remembering:

• Plain FTP is allowed only on an internal VPC endpoint. It sends passwords in clear text, so AWS never exposes it to the internet. The slides draw it the same way: FTP “only within VPC”.
• Partners who allowlist addresses in their firewalls need an internet-facing VPC endpoint with Elastic IPs, which you can also lock down with a security group. A public endpoint cannot do either.
• Security groups need port 22 for SFTP (also 2222 or 22000 if you choose them), 21 for the FTP or FTPS control channel and 8192–8200 for the data channel. (An older endpoint type, VPC_ENDPOINT, can no longer be created; use VPC.)

#Identity providers: who may log in

These are the slides’ “Microsoft Active Directory, LDAP, Okta, Amazon Cognito, custom” list: AD is built in; the others go through a custom identity provider. With a custom provider an empty answer from the function means “not authenticated”. Transfer Family also refuses usernames of one or two characters and the name root, because password scanners try them constantly.

#Home directories, roles and the storage behind the server

• S3 backend. The user’s IAM role decides which bucket actions are allowed. A session policy narrows it per user, so one role can serve many partners. Logical directories map what the user sees to real locations: the partner sees /, which is really payroll-example/inbound/acme/; they never see the bucket name or each other’s folders (a “chroot”).
• EFS backend. Access is controlled with POSIX user and group IDs given to each user, exactly as on a Linux server, plus the role. Use EFS when the files must be read by applications that mount a file system, for example a legacy batch job on EC2.
• In both cases the file is ordinary data afterwards. In S3 you can version it, replicate it, lifecycle it to Glacier (lesson 10.5) or query it with Athena.

#After the file lands: workflows, connectors, AS2 and web apps

• Managed workflows. A workflow attached to the server runs after each upload (and, if you want, after a partial upload when the connection drops). Steps: copy, tag, delete, decrypt (PGP) and custom (any Lambda function: malware scan, validation, loading into a database). If a step fails, the workflow’s exception handler runs instead, for example to move the file to quarantine/ and alert someone.
• SFTP connectors. Transfer Family can also send files from S3 to a partner’s external SFTP server, or fetch files from it, so you do not need an outbound server either.
• AS2. For trading partners that require AS2: messages are signed and encrypted, the receiver sends a signed receipt, and files are stored in S3.
• Web apps. A managed web page for business users to browse, upload and download files in S3, with sign-in through IAM Identity Center and permissions from S3 Access Grants.

#Price and availability

Worked example: the distributor above has one server with SFTP and FTPS: 2 × $0.30 × 720 hours = $432 a month, even if nobody connects, plus 500 GB a month × $0.04 = $20. The endpoint-hours dominate: delete test servers and disable unused protocols. Choosing up to three AZs costs nothing extra, and S3, EFS, Lambda and CloudWatch are billed separately as usual.

#AWS DataSync

#What DataSync is for

AWS DataSync moves large amounts of data over the network, quickly and safely, once or on a schedule: from on-premises (or another cloud) to AWS, from AWS back out, or between AWS storage services such as EFS to S3. The notes list its three traits: an agent is installed on premises, it uses its own fast transfer protocol, and it can be scheduled (for example every night at 12).

It is the removal company of AWS storage: it packs, moves, checks and leaves. It never becomes the place where your applications read or write.

#The building blocks

Where data can go, the slides’ list made precise:

• Agent needed: on-premises or other-cloud storage (NFS, SMB, HDFS, self-managed object storage with an S3 API, Google Cloud Storage, Azure Files) to or from AWS.
• No agent: between AWS storage services (S3, EFS, FSx), including across accounts and Regions, and between S3 and object storage in other clouds in Enhanced mode.
• Destinations in AWS: S3 in any storage class, including Glacier Instant Retrieval, Flexible Retrieval and Deep Archive; EFS; FSx for Windows File Server, Lustre, NetApp ONTAP and OpenZFS.

Activation and network. You deploy the agent, open its port 80 once from your browser to get an activation key (or fetch it from the local console), and activate it in your Region. After that the agent only makes outbound HTTPS (port 443) connections, so no inbound firewall rule is needed. Its service endpoint can be public, FIPS, or a VPC endpoint (AWS PrivateLink), which keeps the transfer on your VPN or Direct Connect and off the internet (lesson 10.10). Data is encrypted in transit with TLS 1.3.

#What happens in one run

A task execution moves through these statuses:

The first run copies everything; every later run copies only what is new or changed (this is incremental by default). So a migration is: one big run, then nightly runs that shrink to minutes, then a final run after a short freeze on cut-over day.

#Task options that matter

• What to copy. Transfer mode: only data that changed (default) or everything. Include and exclude filters (for example skip *.tmp), or a manifest: an exact list of files to copy.
• Deleted files. By default a file deleted at the source is kept at the destination. For a true mirror, set the option to remove it. For S3 classes with a minimum storage duration (Standard-IA, Glacier), overwrites and deletions can trigger early-deletion charges, so think before syncing a constantly changing folder into Glacier.
• Metadata. Preserves POSIX ownership, permissions and timestamps (NFS), NTFS ACLs and owners (SMB, FSx for Windows), and object tags and metadata for object storage. This is the slides’ “file permissions and metadata are preserved”.
• Verification. Verify only the transferred data, or the whole destination, or skip it. Use “only transferred data” when the destination is an archive class.
• Bandwidth. One task can use up to 10 Gbps with an agent (5 Gbps for agentless transfers). You can set a bandwidth limit, and change it while an execution is running.
• Schedule. Hourly, daily, weekly or a cron or rate expression (at most once an hour), and you can pause a schedule.
• Reports and logs. Task reports list every file transferred, skipped, verified or failed; CloudWatch Logs and EventBridge events let you alarm on failures.

#Basic mode and Enhanced mode

Since late 2024 a task has a task mode, chosen when the task is created:

The agent itself costs nothing. You still pay normal S3 request and storage charges, and any data transfer out of the source (for example from another cloud).

#How long, and how much?

The same formula as in lesson 10.11: time = data in bits ÷ usable speed, with 1 TB = 8,000 gigabits.

Worked example, the garments group: 30 TB = 240,000 Gb. At 300 Mbps for 10 office hours and 800 Mbps (80% of 1 Gbps) for 14 night hours, a day moves 0.3 × 36,000 + 0.8 × 50,400 = 51,120 Gb, about 6.4 TB. So the first copy takes about 4.7 days. Nightly changes of 20 GB = 160 Gb take 200 seconds at 0.8 Gbps. Cost in Basic mode: (30,000 + 30 × 20) GB × $0.0125 = $382.50 for the first month.

Try your own numbers. When the first copy goes past a week or a month, the verdict changes:

#DataSync and the Snow Family, then and now

Two links between DataSync and the Snow devices of lesson 12.1 are history: Snowcone shipped with a DataSync agent installed, and DataSync could copy from the S3-compatible storage on a Snowball Edge. DataSync stopped supporting Snow devices on 12 November 2024, when Snowcone was discontinued. Today the pattern is simply: Snowball Edge for the bulk (where you still can order it), DataSync over the network for the changes afterwards. Another 2025 retirement: DataSync Discovery, which analysed on-premises storage and recommended AWS targets, ended on 20 May 2025.

#DataSync or Storage Gateway?

The notes draw the line in one sentence: Storage Gateway is a continuous storage extension that works as part of your local storage; DataSync is a tool to copy or permanently migrate data from one place to another.

They are often used together: DataSync moves ten years of history into a bucket, then the applications that keep producing files are pointed at a File Gateway share on the same bucket. After DataSync writes behind the gateway’s back, refresh the gateway’s cache so the share lists the new objects (lesson 12.3).

#Choosing a transfer service

The five transfer services (and the network links under them) answer different questions. Ask them in this order:

1. Do applications on premises need to keep using the data every day? Yes → Storage Gateway (File for NFS/SMB, Volume for iSCSI disks, Tape for backup software). It is the only one that stays in the path.
2. Who sends the data? Outside companies or people using SFTP, FTPS, FTP or AS2 → Transfer Family. Individual users far from the bucket uploading over HTTPS → S3 Transfer Acceleration.
3. Your own servers or storage, and the network is enough? Copy, migrate or sync, with schedules and verification → DataSync.
4. The network would take weeks or months? → Snowball Edge (offline), then DataSync for the changes. Where Snow devices are not available to you: a bigger or temporary Direct Connect link, AWS Data Transfer Terminal (bring your drives to an AWS location), or a partner service.

Two neighbours that are not storage services but appear in the same questions: Site-to-Site VPN and Direct Connect (lesson 10.10) are the roads, not the trucks: DataSync and Storage Gateway run over them. AWS DMS moves databases and AWS Application Migration Service moves whole servers (both in chapter 23, moving large amounts of data and its neighbours).

#The full storage map

#Three kinds of storage, plus the services that move data

Every storage service in this course is one of three kinds (lesson 5.4 introduced them):

• Block storage is a raw disk. One operating system formats it with a file system and owns it: EBS, instance store, and the iSCSI volumes of Volume Gateway and FSx for NetApp ONTAP.
• File storage is a shared tree of folders reached over a network protocol (NFS, SMB, Lustre): EFS and the four FSx file systems.
• Object storage is a flat bucket of whole objects reached over an HTTPS API: S3 and its Glacier classes.

Around them sit the hybrid and transfer services that connect your building to AWS: Storage Gateway, Transfer Family, DataSync and Snowball. And when the data must be searched, indexed and updated by field, the answer is not storage at all but a database.

#All of it in one table

The notes’ comparison table and the slides’ storage summary, completed with everything chapters 5, 10 and 12 taught:

For databases the course has its own chapters: RDS and Aurora in chapter 7, DynamoDB in chapter 15, and how to choose a database in chapter 16. The slides end their storage summary the same way: databases are for specific workloads, usually with indexing and querying.

Walk your own scenarios through the questions:

#What goes wrong

#Transfer Family and DataSync in system design

• Partner ingestion pipeline. Transfer Family (internet-facing VPC endpoint, Elastic IPs, custom identity provider) → S3 with a prefix per partner → managed workflow (decrypt, scan, tag) → EventBridge or S3 events start processing (lesson 10.7) → lifecycle to Glacier. Partners change nothing; you run no servers.
• High availability. A Transfer Family server already spans up to three AZs. For Regional failover, run a second server in another Region with replicated buckets and switch with Route 53 health checks. Global Accelerator in front of Transfer Family can improve long-distance throughput (chapter 11).
• Private file exchange. Internal VPC endpoint reached over Direct Connect or VPN for your own branches, including legacy FTP devices.
• Migration in phases. Discovery and planning, then DataSync for the bulk (or Snowball Edge if the network would take weeks), nightly incremental runs, a write freeze, the final run and the verification report, then cut-over and decommissioning. Use a VPC endpoint for DataSync so the copy never crosses the internet.
• Copies between AWS services. DataSync without an agent: EFS → S3 for a cheaper second copy, S3 → EFS for an application that needs files, FSx → FSx in another Region, or another account.
• Hybrid for good. When the applications stay on premises, Storage Gateway, not DataSync; DataSync may still do the first big copy.
• Pick the storage last. Decide first how the data is accessed (block, file, object, query), by whom (one host, many, the whole internet), with which protocol, and for how long. The table above then gives one answer.

#Try it yourself

Transfer Family: an internet-facing SFTP server in a VPC, a service-managed user with a logical home directory, then an upload.

# 1. Create the server: SFTP on a VPC endpoint, CloudWatch logging
aws transfer create-server --protocols SFTP --identity-provider-type SERVICE_MANAGED \
  --endpoint-type VPC \
  --endpoint-details VpcId=vpc-0a1b2c3d4example,SubnetIds=subnet-0aa11example,SecurityGroupIds=sg-0cc33example \
  --logging-role arn:aws:iam::111122223333:role/AWSTransferLoggingAccess
# { "ServerId": "s-01234567890abcdef" }

#    Make it internet-facing: Elastic IPs are attached with update-server, while it is stopped
aws transfer stop-server --server-id s-01234567890abcdef
aws transfer update-server --server-id s-01234567890abcdef \
  --endpoint-details VpcId=vpc-0a1b2c3d4example,SubnetIds=subnet-0aa11example,AddressAllocationIds=eipalloc-0bb22example
aws transfer start-server --server-id s-01234567890abcdef

# 2. Add a partner: role, logical home directory (chroot to their prefix), SSH public key
aws transfer create-user --server-id s-01234567890abcdef --user-name acme \
  --role arn:aws:iam::111122223333:role/TransferPartnerS3 \
  --home-directory-type LOGICAL \
  --home-directory-mappings '[{"Entry":"/","Target":"/payroll-example/inbound/acme"}]' \
  --ssh-public-key-body "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIExampleKey acme@example.com"

# 3. Check that the identity setup returns what you expect
aws transfer test-identity-provider --server-id s-01234567890abcdef --user-name acme \
  --server-protocol SFTP --source-ip 203.0.113.10
# { "StatusCode": 200, "Message": "", "Url": "..." }

# 4. As the partner: connect and upload, then see the object
sftp -i acme_key acme@198.51.100.20
# sftp> put salary-oct.csv
aws s3 ls s3://payroll-example/inbound/acme/
# 2026-10-26 09:14:02     184213 salary-oct.csv

DataSync: an agent, an SMB source, an FSx for Windows destination, a nightly task with a bandwidth limit, one run.

# 1. Activate the agent VM (activation key from the agent's local console)
aws datasync create-agent --agent-name head-office-agent --activation-key ABCDE-12345-FGHIJ-67890-KLMNO
# { "AgentArn": "arn:aws:datasync:ap-south-1:111122223333:agent/agent-0b0addbeef44baca3" }

# 2. Source: the old Windows file server over SMB
aws datasync create-location-smb --server-hostname 10.20.1.40 --subdirectory /shares \
  --user svc-datasync --domain CORP --password '<from your vault>' \
  --agent-arns arn:aws:datasync:ap-south-1:111122223333:agent/agent-0b0addbeef44baca3

# 3. Destination: FSx for Windows File Server
aws datasync create-location-fsx-windows \
  --fsx-filesystem-arn arn:aws:fsx:ap-south-1:111122223333:file-system/fs-0123456789example \
  --security-group-arns arn:aws:ec2:ap-south-1:111122223333:security-group/sg-0dd44example \
  --user svc-datasync --domain CORP --password '<from your vault>'

# 4. The task: changed files only, copy NTFS permissions, verify, 300 Mbps cap, every night at 1 AM
aws datasync create-task --name shares-to-fsx --task-mode BASIC \
  --source-location-arn <smb-location-arn> --destination-location-arn <fsx-location-arn> \
  --options TransferMode=CHANGED,VerifyMode=ONLY_FILES_TRANSFERRED,SecurityDescriptorCopyFlags=OWNER_DACL,BytesPerSecond=37500000,LogLevel=TRANSFER \
  --schedule ScheduleExpression="cron(0 1 * * ? *)"

# 5. Run it now and watch the phases
aws datasync start-task-execution --task-arn <task-arn>
aws datasync describe-task-execution --task-execution-arn <execution-arn>
# "Status": "TRANSFERRING", "FilesTransferred": 1204533, "BytesTransferred": 8812004376011,
# "BytesCompressed": 7105227001338, "EstimatedFilesToTransfer": 4100000

# 6. Lift the cap for the night without stopping the execution
aws datasync update-task-execution --task-execution-arn <execution-arn> --options BytesPerSecond=-1

BytesPerSecond=37500000 is 300 Mbps and -1 means no limit. Delete test Transfer Family servers when you are done: they bill every hour.

#Summary

Key takeaways

• Transfer Family is a fully managed SFTP, FTPS, FTP and AS2 endpoint (plus browser web apps) in front of S3 or EFS. Exam: SFTP/FTP/FTPS + “without managing servers”.
• A session: hostname (your own name through Route 53, imported host key) → identity provider → IAM role + session policy + logical home directory → objects written straight into S3 → managed workflow (copy, tag, delete, PGP decrypt, Lambda; exception handler).
• Endpoint types: public = SFTP only, changing IPs, no security group; VPC internet-facing = SFTP, FTPS, AS2 with Elastic IPs and security groups; VPC internal = all four, and the only place plain FTP is allowed.
• Identity: service managed (SSH keys), AWS Directory Service (Microsoft AD), or a custom identity provider through Lambda or API Gateway (LDAP, Okta, Cognito…).
• Price: $0.30 per hour per enabled protocol, plus $0.04 per GB; idle servers still cost about $216 a month per protocol.
• DataSync copies large data over the network, once or scheduled (hourly, daily, weekly): agent (a VM near the source, outbound 443 only) for on-premises and other clouds, no agent between AWS services. Destinations: S3 in any class including Glacier, EFS, all four FSx.
• A run: queued → launching → preparing → transferring (TLS, compressed, metadata preserved) → verifying. Incremental by default, up to 10 Gbps per task, bandwidth limits, filters, manifests, reports.
• Basic mode: $0.0125 per GB, 50 million files per run; Enhanced mode: parallel, virtually unlimited files, $0.015 per GB + $0.55 per run.
• DataSync copies and leaves; Storage Gateway stays in the path as a local NFS, SMB, iSCSI or tape interface with a cache. Often both: DataSync for the past, a gateway for the future.
• Choosing: apps keep using it on site → Storage Gateway; outsiders with file protocols → Transfer Family; far users uploading → Transfer Acceleration; own storage over the network → DataSync; weeks over the network → Snowball Edge.
• The map: block (EBS, instance store), file (EFS, FSx for Windows, Lustre, NetApp ONTAP, OpenZFS), object (S3, Glacier classes), hybrid and transfer (Storage Gateway, Transfer Family, DataSync, Snowball), and databases when you need to query.

Self-check: Transfer Family, DataSync and the storage map30 questions