High Performance Computing on AWS

By the end of this lesson you will be able to

• Explain why the cloud suits HPC (thousands of cores in minutes, faster results by adding nodes, paying only for the hours used) and name the typical workloads.
• Tell a loosely coupled job from a tightly coupled one, and design each: Batch and Spot for independent tasks, Hpc instances with EFA in a cluster placement group for MPI.
• Choose how to move terabytes or petabytes in (DataSync, Direct Connect, Snowball Edge), and know what happened to Snowmobile and the Snow Family.
• Explain enhanced networking (SR-IOV, ENA, the legacy Intel 82599 VF), what EFA adds (OS-bypass through libfabric, SRD, MPI and NCCL), and EFA's limits (Linux, one AZ, not routable).
• Pick HPC storage: EBS io2 Block Express, instance store, S3, EFS and FSx for Lustre linked to S3 (scratch or persistent, lazy load, export).
• Automate the cluster with AWS Batch (array and multi-node parallel jobs) or AWS ParallelCluster (YAML, Slurm, EFA, Lustre), and know where AWS Parallel Computing Service fits.
• Put it all together in a reference architecture, and recognise the exam keywords that pick each service.

#Why HPC belongs in the cloud

High performance computing (HPC) means solving very large problems, huge datasets or very heavy mathematics, quickly, by running one job across hundreds or thousands of CPU cores or GPUs at the same time. Supercomputers do this; so does a rack of servers in a university basement.

Owning that hardware is hard. The cluster is idle most of the year and too small in the one week everybody needs it, and it is out of date in four or five years. The cloud changes the economics (slide p818):

• Huge capacity in minutes: start 10,000 cores for a job, not after a six-month purchase.
• Faster time to results: if a job scales, adding nodes shortens it. 100 nodes for 1 hour often cost about the same as 10 nodes for 10 hours, but the answer arrives nine hours sooner.
• Pay only for what you use: when the job ends, the nodes go away and so does the bill.
• Pick the right machine per job: CPU types for one model, GPUs for the next, without buying either.

Typical workloads: genomics, computational chemistry (drug and material design), financial risk modelling (Monte Carlo), weather and climate prediction, machine learning and deep learning training, autonomous driving simulation, computational fluid dynamics (CFD) for cars, ships and aircraft, seismic processing for oil and gas, and film rendering.

This lesson is a synthesis. Every building block was taught earlier: instance families (lesson 4.1), Spot (lesson 4.4), placement groups, ENA and EFA (lesson 4.5), EBS and instance store (lessons 5.2 and 5.3), Snowball, FSx and DataSync (lessons 12.1, 12.2, 12.4). Here you wire them together and learn which combination each kind of HPC job needs. The slides split the topic into four layers, and so does this lesson: data management and transfer, compute and networking, storage, automation and orchestration.

#Two kinds of HPC job

Before choosing any service, ask one question: do the nodes need to talk to each other while they work?

A checkpoint is a snapshot of a running simulation's state written to durable storage every so often. A tightly coupled job that loses a node restarts from the last checkpoint instead of from zero; how often you checkpoint is a trade-off between the time spent writing and the work you can lose.

#Data management and transfer

HPC starts with data: terabytes of sequencer output, petabytes of seismic surveys, years of weather observations. It must reach AWS before the first core starts, and results often go back. The slides (p819) list three ways; all were taught in detail earlier, so here is the HPC view:

The arithmetic that decides between online and offline: 1 Gbps fully used moves about 10.8 TB a day; 10 Gbps about 108 TB a day. A 500 TB dataset therefore needs about 46 days on a 1 Gbps line, but under 5 days on a 10 Gbps Direct Connect link. More examples and the formula are in lesson 23.5.

Once in AWS, the data's long-term home is S3: durable, cheap, and readable by every later step. The fast file system the cluster needs is created in front of S3 for each run (the storage section below).

#Compute: the right instances

#Instance families for HPC

Recap from lesson 4.1: the slides (p820) say “CPU optimized, GPU optimized”, and the notes say C family or P/G family. In more detail:

The current HPC-optimised types, verified in November 2026:

Two facts that matter in a design: Hpc instances are sold On-Demand and with Savings Plans only, not as Spot, and they exist in only a few Regions and AZs (Hpc8a launched in US East Ohio and Europe Stockholm). To reach the full 300 Gbps on Hpc7a, attach interfaces to both of its two network cards.

#Spot, Spot Fleet and Auto Scaling for cost

Slide p820 suggests Spot Instances and Spot Fleets for cost savings, plus Auto Scaling (lesson 4.4). That advice is right for loosely coupled work: a Spot Fleet or an AWS Batch compute environment draws from many instance types and AZs, so a reclaimed instance costs one task. For tightly coupled MPI jobs it is a trap: one interruption stops the whole job. Auto Scaling in HPC usually means the scheduler scaling the compute fleet with the job queue (Batch compute environments, ParallelCluster's Slurm queues), down to zero when the queue is empty.

#Networking: placement, enhanced networking and EFA

The notes put it simply: HPC servers exchange a lot of data with each other, so an ordinary network is not enough. Three tools, from the outside in.

#Cluster placement group

Recap from lesson 4.5: a cluster placement group packs instances onto nearby hardware in one Availability Zone, in the same high-bandwidth network segment. Inside it, a single flow between two instances can reach 10 Gbps (5 Gbps outside), and latency is the lowest EC2 offers. Slide p820 shows it as “same rack, same AZ, low latency, 10 Gbps network”.

In an HPC design it brings three consequences:

• Shared fate: instances close together can fail together (a rack switch, a power feed). In an MPI job one failure stops all ranks, so checkpoint.
• Capacity: the group needs all its instances close together. Launch them in one request, of one type, and for deadline work reserve them with an On-Demand Capacity Reservation in the placement group, or you may meet an insufficient capacity error.
• One AZ only: EFA traffic cannot leave the AZ anyway.

The other two strategies are the wrong tools for MPI: spread (separate racks, at most 7 running instances per AZ per group) is for a few critical servers, and partition (groups of racks that fail separately) is for big distributed systems such as HDFS, HBase or Cassandra.

#Enhanced networking: SR-IOV and ENA

Without enhanced networking, every packet of a virtual machine passes through the hypervisor's software. Enhanced networking uses SR-IOV (single root I/O virtualization): the physical network card presents many virtual functions, and each instance gets one directly, so packets skip the hypervisor. Slide p821 lists the result: higher bandwidth, higher PPS (packets per second) and lower latency, at no extra cost.

The EC2 guide still says ENA “supports network speeds of up to 100 Gbps”, which is the slide's number. Instances with several network cards go further in total (Hpc7a reaches 300 Gbps with interfaces on its two cards, and the largest GPU instances far more), but each ENA interface stays the building block. For the exam: enhanced networking = SR-IOV = ENA, the Intel VF is the legacy answer.

#Elastic Fabric Adapter (EFA)

An EFA is an ENA plus an extra device for HPC and machine learning. With an EFA interface (“EFA with ENA”), the instance gets two things: the normal ENA device for IP traffic (SSH, S3, the file system) and an EFA device that applications can drive directly from user space.

How the fast path works (slide p821: “bypasses the underlying Linux OS to provide low-latency, reliable transport”):

1. The application uses an MPI library (Open MPI 4.1 or later, Intel MPI) or, for GPUs, NCCL.
2. That library sits on libfabric, an open-source networking API that runs in user space.
3. libfabric places messages straight into the EFA device's queues: no system call, no kernel copy, no TCP stack. This is OS-bypass.
4. On the wire, EFA uses SRD (Scalable Reliable Datagram): packets of one message are sprayed over many network paths and lost ones are retransmitted in microseconds by the Nitro hardware, so latency is low and, just as important, consistent.
5. On newer instances (Nitro v4 and later) EFA supports RDMA read and write: data lands directly in the other instance's memory, or a GPU's memory.

The rules that turn into exam traps:

• Linux for OS-bypass. On Windows, an EFA interface works as a plain ENA (the only exception is the AWS Cloud Digital Interface SDK for live video). Slide p821 says “only works for Linux”.
• Not routable, one AZ. EFA traffic cannot cross Availability Zones or VPCs; the ENA part of the interface remains routable as usual.
• Supported instance types only: Hpc, the larger C, M, R sizes, P and G GPU types and Trainium, among others. Check with describe-instance-types and the network-info.efa-supported filter.
• One EFA per network card on instances with several cards, one per instance otherwise. There are also EFA-only interfaces (no IP address at all) for extra network cards.
• No extra charge. EFA is a free option on supported instances.
• The security group must allow all traffic to and from itself, because MPI ranks talk on many ports.

#Storage for HPC

Slide p822 splits storage in two: storage attached to the instance and network storage shared by many instances. Each was taught earlier; here is what each one means for an HPC job:

Slide p822 describes EFS as scaling “IOPS based on total size, or use provisioned IOPS”. That describes the older bursting and provisioned throughput modes; today Elastic throughput is the default and scales automatically with the workload, and you pay for what you use.

#FSx for Lustre in an HPC run

The notes call FSx for Lustre the most important HPC storage, and the exam agrees. Lustre (from Linux + cluster) stripes every file across many storage servers, and every client talks to all of them at once, so throughput grows with the file system. Recap of what lesson 12.2 taught, in the HPC order of events:

1. Create a file system for the run. Scratch (scratch 2) is cheapest, has no replication and bursts to about six times its baseline: right for a job whose input is in S3 and which can be re-run. Persistent (persistent 2) replicates within the AZ and replaces failed servers: right for long-running work, shared home directories and data that must survive a server failure.
2. Link it to S3 with a data repository association: a directory such as /fsx/input maps to an S3 prefix (up to 8 per file system; not on the old scratch 1 type). Only the metadata is imported, so millions of objects appear as files in minutes.
3. Lazy load: the first read of a file pulls its content from S3; every later read by any node comes from Lustre. Preload files before the job if the first read must not slow the run.
4. Run the job: hundreds of nodes read and write /fsx as a normal POSIX file system.
5. Export: with automatic export, or an export data repository task at the end, new and changed files go back to S3.
6. Delete the file system. S3 holds input and output.

#Automation and orchestration

Building an HPC environment by hand again and again is slow and error-prone (the notes' point), and a cluster built by hand tends to be left running because nobody wants to build it again. Two AWS tools automate it (slide p823), and a third arrived in 2024.

#AWS Batch

AWS Batch runs batch jobs (containers) on a managed compute environment of EC2 (On-Demand or Spot) or Fargate. You submit jobs to a job queue; Batch works out how many vCPUs and how much memory the queue needs, launches instances to fit, retries failed jobs, and scales the environment back to zero when the queue is empty. The full service gets its own lesson in lesson 25.5; for HPC, two job types matter:

• Array jobs: one submission creates up to 10,000 child jobs, each with its own index. The natural way to run loosely coupled work on Spot.
• Multi-node parallel jobs (slide p823): one job that spans several EC2 instances, for MPI or distributed training. Batch launches a main node first, then the child nodes, all at once (gang scheduling); the job ends when the main node exits. AWS recommends a cluster placement group in the compute environment, and multi-node jobs are not supported on Spot compute environments or on Fargate.

#AWS ParallelCluster

AWS ParallelCluster is an open-source cluster management tool from AWS. You describe the cluster in one YAML file, and the pcluster command line (or the ParallelCluster UI, a web console you deploy) builds it through CloudFormation:

• the head node, with the Slurm scheduler (the scheduler most HPC centres already use);
• queues of compute nodes, each with instance types, minimum and maximum counts (minimum 0 means no compute cost while the queue is empty), On-Demand or Spot, EFA on and a placement group;
• shared storage: EBS, EFS, FSx for Lustre (new or existing, linked to S3), FSx for ONTAP or OpenZFS;
• networking in an existing VPC and subnets (the pcluster configure wizard can create a VPC and subnets for you);
• custom bootstrap scripts, monitoring dashboards in CloudWatch, login nodes.

Users then log in to the head node and submit jobs with Slurm's sbatch. Slurm, through ParallelCluster's plugin, launches compute nodes for the queued jobs and terminates them after they sit idle (10 minutes by default).

#A reference architecture: the cyclone forecast cluster

Now wire the four layers together. A weather centre in Dhaka runs a WRF-style forecast model (WRF is the widely used open-source weather model) whenever a cyclone forms:

1. Data in: observation data and global model boundary conditions arrive every few hours by DataSync over a Direct Connect link into an S3 bucket. Years of archive came in earlier, on Snowball Edge devices.
2. Storage: a scratch FSx for Lustre file system is linked to the S3 prefixes for input and output.
3. Cluster: ParallelCluster builds a head node with Slurm and a queue of Hpc7a instances, On-Demand, EFA enabled, in a cluster placement group in one AZ, minimum 0 and maximum 64 nodes. A capacity reservation in the placement group covers the cyclone season.
4. Run: the forecaster submits the model with sbatch. Slurm launches the nodes, the model reads its input through Lustre (lazy-loaded from S3), the ranks exchange boundaries over EFA every time step, and a checkpoint is written every simulated hour.
5. Results out: forecast files are exported to S3; Athena and QuickSight (lesson 17.5) or a web map show the expected track and storm surge to disaster-management officials.
6. Back to zero: idle compute nodes terminate; between seasons the whole cluster is deleted and rebuilt from the same YAML file.

#Build one yourself

Choose a workload and the five building blocks, then run the design. Try a tightly coupled job on Spot, a genome alignment in a spread placement group, or EFA on T3.

#What goes wrong

• MPI job slower on 64 nodes than on 16: no EFA, or no placement group, so latency dominates. Check fi_info -p efa on the nodes and the placement group of each instance.
• EFA attached but the job uses TCP: the MPI library was not built with libfabric, the security group blocks traffic to itself, or the instance runs Windows.
• A rank in another AZ or VPC cannot be reached over EFA: EFA traffic is not routable. Keep the whole job in one AZ.
• Insufficient capacity when the job starts: the placement group cannot fit all nodes. Launch in one request with one type, reserve capacity in the placement group, or try another AZ.
• The whole simulation restarts from zero after a node loss: no checkpoints, or checkpoints written to the scratch file system that was lost with the node. Write checkpoints to persistent storage or export them to S3.
• Batch multi-node job stuck in RUNNABLE: the compute environment uses Spot (not supported), lacks enough vCPUs, or the account's instance quota is lower than the nodes requested.
• The first minutes of every run are slow: lazy loading from S3. Preload the input before the job starts.
• Results lost when the cluster was deleted: output stayed on the scratch Lustre file system and was never exported to S3.
• A loosely coupled job costs a fortune: it runs On-Demand on Hpc instances in a cluster placement group. Use Batch with Spot across many instance types and AZs.
• The bill keeps running after the forecast: compute queues with a minimum above zero, or a hand-built cluster nobody switched off.
• Data transfer planned on Snowball in 2026: check whether your account can still order devices before 31 December 2026; plan DataSync, Direct Connect or Data Transfer Terminal.

#HPC in system design

• Loosely coupled pipeline: S3 (input) → AWS Batch array job on Spot across many instance types and AZs, with retries → S3 (results) → Athena or QuickSight. Add EventBridge to start the pipeline when a sequencer upload lands in S3.
• Tightly coupled cluster: S3 data lake → FSx for Lustre linked to S3 → ParallelCluster (Slurm) with Hpc instances, EFA, a cluster placement group in one AZ, On-Demand or reserved → checkpoints and results exported to S3.
• Distributed ML training: S3 dataset → FSx for Lustre → GPU instances (P family) with EFA and NCCL in a cluster placement group, orchestrated by Slurm (ParallelCluster or PCS) or Batch multi-node jobs.
• Cloud bursting: an on-premises Slurm cluster keeps running; peaks go to AWS over Direct Connect, with FSx for Lustre (or DataSync) bringing the data close to the cloud nodes.
• Cost: Spot for independent tasks, Savings Plans for steady HPC use, capacity reservations for deadlines, scale to zero always.

#Try it yourself

These commands use the AWS CLI v2 and the ParallelCluster CLI (pip install aws-parallelcluster). Hpc7a nodes and Lustre file systems are expensive: read before you run, and delete everything you create.

# 1. Which instance types in this Region support EFA, and how fast are they?
aws ec2 describe-instance-types --region us-east-2 \
  --filters Name=network-info.efa-supported,Values=true Name=instance-type,Values='hpc*' \
  --query "InstanceTypes[].[InstanceType,NetworkInfo.NetworkPerformance,NetworkInfo.MaximumNetworkCards]" \
  --output text
# hpc6a.48xlarge   100 Gigabit   1
# hpc7a.96xlarge   300 Gigabit   2
# hpc8a.96xlarge   300 Gigabit   2

# 2. A cluster placement group, and four instances with EFA launched into it in one request
aws ec2 create-placement-group --group-name cyclone-pg --strategy cluster --region us-east-2
aws ec2 run-instances --region us-east-2 --count 4 --instance-type hpc7a.96xlarge \
  --image-id ami-0abcdef1234567890 --key-name forecast-key \
  --placement GroupName=cyclone-pg \
  --network-interfaces "NetworkCardIndex=0,DeviceIndex=0,InterfaceType=efa,SubnetId=subnet-0bbb2222example,Groups=sg-0ccc3333example"

# 3. On a node: is the EFA provider visible to libfabric?
fi_info -p efa -t FI_EP_RDM
# provider: efa
#     fabric: efa
#     domain: rdmap0s6-rdm
#     version: 122.0
#     type: FI_EP_RDM
#     protocol: FI_PROTO_EFA

# 4. A scratch FSx for Lustre file system, linked to S3 for input and output
aws fsx create-file-system --region us-east-2 --file-system-type LUSTRE \
  --storage-capacity 2400 --subnet-ids subnet-0bbb2222example \
  --lustre-configuration DeploymentType=SCRATCH_2
# { "FileSystem": { "FileSystemId": "fs-0123456789abcdef0", "Lifecycle": "CREATING", ... } }

aws fsx create-data-repository-association --region us-east-2 \
  --file-system-id fs-0123456789abcdef0 --file-system-path /input \
  --data-repository-path s3://forecast-example/input/ --batch-import-meta-data-on-create \
  --s3 "AutoImportPolicy={Events=[NEW,CHANGED,DELETED]}"
aws fsx create-data-repository-association --region us-east-2 \
  --file-system-id fs-0123456789abcdef0 --file-system-path /output \
  --data-repository-path s3://forecast-example/output/ \
  --s3 "AutoExportPolicy={Events=[NEW,CHANGED,DELETED]}"

# Mount it on a node, and preload the input so the run does not wait on lazy loading
sudo mount -t lustre -o relatime,flock fs-0123456789abcdef0.fsx.us-east-2.amazonaws.com@tcp:/abcdefgh /fsx
find /fsx/input -type f -print0 | xargs -0 -n 1 sudo lfs hsm_restore

A ParallelCluster configuration, saved as cluster.yaml:

Region: us-east-2
Image:
  Os: alinux2023
HeadNode:
  InstanceType: c7i.2xlarge
  Networking:
    SubnetId: subnet-0aaa1111example
  Ssh:
    KeyName: forecast-key
Scheduling:
  Scheduler: slurm
  SlurmSettings:
    ScaledownIdletime: 10
  SlurmQueues:
    - Name: cyclone
      CapacityType: ONDEMAND
      Networking:
        SubnetIds:
          - subnet-0bbb2222example
        PlacementGroup:
          Enabled: true
      ComputeResources:
        - Name: hpc7a
          InstanceType: hpc7a.96xlarge
          MinCount: 0
          MaxCount: 64
          Efa:
            Enabled: true
SharedStorage:
  - MountDir: /fsx
    Name: forecast-fsx
    StorageType: FsxLustre
    FsxLustreSettings:
      StorageCapacity: 2400
      DeploymentType: SCRATCH_2
      ImportPath: s3://forecast-example/input
      ExportPath: s3://forecast-example/output

# 5. Build the cluster, check it, submit the model
pcluster create-cluster --cluster-name cyclone --cluster-configuration cluster.yaml --region us-east-2
pcluster describe-cluster --cluster-name cyclone --region us-east-2 --query clusterStatus
# "CREATE_COMPLETE"
pcluster ssh --cluster-name cyclone -i ~/.ssh/forecast-key.pem
sbatch --nodes=64 --ntasks-per-node=192 --exclusive run-wrf.sh
# Submitted batch job 41
squeue
#  JOBID PARTITION     NAME  USER ST  TIME  NODES NODELIST(REASON)
#     41   cyclone  run-wrf  ec2-u CF  0:42     64 cyclone-dy-hpc7a-[1-64]
#  (CF = configuring: Slurm is launching the 64 nodes)

# 6. Loosely coupled work in AWS Batch: one array job of 10,000 alignments on a Spot queue
aws batch submit-job --job-name align-rice --job-queue spot-queue \
  --job-definition bwa-align:3 --array-properties size=10000 \
  --retry-strategy attempts=3
# { "jobName": "align-rice", "jobId": "8a1b2c3d-EXAMPLE" }

# 7. An MPI job in AWS Batch: a multi-node parallel job definition with 8 nodes (On-Demand queue)
aws batch submit-job --job-name cfd-hull --job-queue ondemand-mnp-queue \
  --job-definition openfoam-mnp:1 --node-overrides numNodes=8

# Clean up
pcluster delete-cluster --cluster-name cyclone --region us-east-2
aws fsx delete-file-system --file-system-id fs-0123456789abcdef0 --region us-east-2

#Summary

Key takeaways

• The cloud suits HPC: thousands of cores in minutes, faster results by adding nodes, pay only for the hours used. Workloads: genomics, computational chemistry, financial risk, weather, ML and deep learning, autonomous driving, CFD.
• Loosely coupled (independent tasks) → AWS Batch array jobs on Spot, no placement group, S3. Tightly coupled (MPI) → Hpc instances On-Demand, EFA, cluster placement group in one AZ, FSx for Lustre, ParallelCluster, checkpoints. Hpc instances are not sold as Spot; Batch multi-node jobs do not run on Spot.
• Data in: Direct Connect (private, steady, up to 400 Gbps), DataSync (online, verified, to S3, EFS or any FSx), Snowball Edge (offline, 210 TB per device; existing customers only, support ends 31 December 2026). Snowmobile was retired in 2024, but older exam questions still use it for exabytes. 1 Gbps ≈ 10.8 TB a day.
• Cluster placement group: one AZ, nearby hardware, lowest latency, 10 Gbps per flow; shared fate and capacity limits. Very low latency, not zero; not guaranteed one rack.
• Enhanced networking = SR-IOV: higher bandwidth, higher PPS, lower latency. ENA up to 100 Gbps per the guide (more per instance with several network cards); Intel 82599 VF up to 10 Gbps, legacy. ENA Express: SRD for ordinary TCP, 25 Gbps per flow.
• EFA = ENA + OS-bypass through libfabric for MPI and NCCL, carried by SRD, RDMA on newer instances. Linux only for OS-bypass (Windows: plain ENA), cannot cross AZs or VPCs, supported types only, free.
• Storage: io2 Block Express 256,000 IOPS per volume; instance store millions of IOPS but ephemeral; S3 is not a file system but is the data lake; EFS for shared home folders; FSx for Lustre for hundreds of GB/s, millions of IOPS, sub-ms latency, scratch or persistent, linked to S3 by a data repository association with lazy load and export.
• Orchestration: AWS Batch (array jobs; multi-node parallel jobs span instances), AWS ParallelCluster (open source, YAML, pcluster, Slurm, EFA, placement group, Lustre, scale to zero), AWS PCS (managed Slurm, 2024; the exam's answer is still ParallelCluster).

Self-check: High Performance Computing on AWS31 questions