Placement Groups, ENIs and Hibernate

By the end of this lesson you will be able to

• Choose between cluster, spread and partition placement groups from a scenario, and explain what each one protects against.
• Know the hard numbers: 7 running instances per AZ in a spread group, 7 partitions per AZ in a partition group, one AZ for a cluster group.
• Explain what “insufficient capacity” means in a placement group and how to retry, and when a precision time placement group or Amazon Time Sync is the answer.
• Describe an elastic network interface (ENI), its attributes and its limits, and use a secondary ENI to fail a private IP over to a standby instance.
• Tell ENI, ENA and EFA apart.
• Explain how EC2 Hibernate keeps RAM across a stop, list every condition it needs, and pick it for slow-starting applications.

#Why you would care where an instance lands

When you launch an instance, EC2 picks a physical server for it somewhere in the Availability Zone. Normally that is exactly what you want: AWS spreads instances across its hardware so that one broken server or rack hurts as few customers as possible.

Sometimes the default is wrong for your application:

• A high performance computing (HPC) job or a tightly coupled big data job wants its instances as close as possible, so they talk to each other with the lowest latency and the highest bandwidth.
• A handful of critical servers (two database nodes, three domain controllers) must never share a rack, so one hardware fault cannot take out two of them.
• A large distributed system such as Kafka, Cassandra or HDFS wants its nodes in groups of racks, and wants to know which group each node is in, so it can keep copies of the data in different groups.

A placement group is how you tell EC2 which of these you want. You create the group with a strategy, then launch instances into it. Placement groups are free; you pay only for the instances.

#Placement groups

#Cluster: as close as possible

A cluster placement group packs instances close together inside one Availability Zone, in the same high-bandwidth segment of the AWS network (often the same rack or neighbouring racks).

• Gain: the lowest latency and the highest packet rate between the instances. With enhanced networking, a single TCP flow between two instances in the group can reach 10 Gbps; outside a cluster group a single flow is limited to 5 Gbps. Total bandwidth depends on the instance type (100 Gbps and more on the largest types).
• Risk: everything sits in one place. A failure of the shared hardware, and certainly a failure of the AZ, can stop every instance at the same time.
• Use it for: HPC, tightly coupled big data jobs that must finish fast, and any application where most traffic goes between the instances in the group.

How to get the best out of it:

• Launch all the instances in one request, of one instance type. Adding instances later, or mixing types, raises the chance of an insufficient capacity error.
• Use an instance type with enhanced networking (ENA, or EFA for HPC, both covered below). Burstable types such as T2/T3 are not supported in a cluster group.
• If you need guaranteed room, create an On-Demand Capacity Reservation in the cluster placement group.
• The speed between two instances is limited by the slower of the two.

#Spread: each instance on its own hardware

A spread placement group puts every instance on distinct hardware: a different rack, each with its own network and power source.

• Gain: one hardware failure can affect at most one instance of the group.
• Limit: 7 running instances per Availability Zone per group. A spread group can span several AZs of the Region, so in a Region with three AZs one group can hold 21 running instances (7 in each).
• Use it for: a small number of critical instances that must be kept apart: a primary and a standby database, a cluster of three or five coordination nodes, a pair of license servers.
• You may mix instance types and add instances over time, because each new instance only needs one free rack.
• Spread groups do not support Dedicated Instances. (On AWS Outposts there is also a host-level spread; in Regions it is always rack-level.)

#Partition: groups of racks that fail separately

A partition placement group divides its instances into partitions. Each partition gets its own set of racks, and no two partitions share a rack. A hardware failure can take down many instances in one partition but never touches the other partitions.

• Limit: up to 7 partitions per AZ. The group can span several AZs of the same Region. The number of instances is limited only by your account’s EC2 quotas, so hundreds of instances are normal.
• Visibility: each instance can read its partition number from the instance metadata (placement/partition-number), and the API shows which instance is in which partition.
• Use it for: large distributed and replicated systems that understand topology: HDFS, HBase, Cassandra, Kafka. You configure the system to treat each partition as a “rack”, and it places the copies of each piece of data in different partitions.
• EC2 tries to spread instances evenly across the partitions you asked for, but does not guarantee it. You can also launch into a specific partition (PartitionNumber=2).
• With Dedicated Instances, a partition group can have at most 2 partitions.

#The three strategies side by side

#Precision time placement groups and Amazon Time Sync

Distributed databases, financial trading and event logs often order events by timestamp. If two servers disagree about the time, a later write can look earlier and win, and records end up in the wrong order.

Every EC2 instance can already use the Amazon Time Sync Service at no charge:

• It answers on the link-local address 169.254.169.123 (and fd00:ec2::123 over IPv6 on Nitro instances), so the instance needs no internet access to reach it. Amazon Linux uses it by default.
• Behind it is a fleet of satellite-connected and atomic reference clocks in every Region.
• On supported instance types there is also a PTP hardware clock exposed to Linux for microsecond-level accuracy.

The newest strategy, a precision time placement group, makes sure your instances land on hardware with direct access to these high-precision time sources. Instances in it get the enhanced local NTP source, the PTP hardware clock (Linux only) and hardware packet timestamping (Linux only). It costs nothing extra. A cluster group can name a precision time group as its parent, so a tightly coupled cluster also gets precise time.

#Rules that apply to every placement group

• An instance can be in one placement group at a time, and groups cannot be merged.
• To move an existing instance into (or out of) a group, stop it, change its placement (modify-instance-placement), and start it again.
• A group’s name is unique per account per Region.
• Placement groups are free.
• If you stop and start an instance in a placement group, it stays in the group, but the start fails if there is not enough suitable capacity at that moment.
• Capacity Reservations reserve room only in cluster groups, not in spread or partition groups. Zonal Reserved Instances cannot reserve capacity in a placement group at all.

#Insufficient capacity: what it means and what to do

A placement group adds a constraint to every launch. EC2 can have plenty of free servers in the AZ and still fail your request, because none of them meet the group’s rule:

The error you see is InsufficientInstanceCapacity. It is not a quota problem: raising your account limits does not fix it.

#Elastic network interfaces (ENI)

An elastic network interface (ENI) is a virtual network card that lives in a subnet of a VPC. Every instance has at least one; an instance’s networking settings are really the settings of its ENIs.

#What an ENI carries

#Primary and secondary ENIs

• The primary ENI (device index 0, usually eth0 in Linux) is created with the instance and can never be detached.
• Secondary ENIs (eth1, eth2…) can be created on their own, then attached and detached. You can attach while the instance is running (hot attach), stopped (warm attach) or launching (cold attach).
• The maximum number of ENIs, and of IP addresses per ENI, depends on the instance type. A small type may allow 2 or 3 ENIs; the largest types allow dozens.
• Amazon Linux and Windows Server configure a hot or warm attached ENI themselves; other operating systems may need you to bring the interface up and fix its routes.

#Where an ENI can go

• An ENI belongs to one subnet, so it is bound to that subnet’s Availability Zone. It can be attached only to an instance in the same AZ.
• The instance may be in a different subnet of that AZ, and even in a different VPC of the same account, as long as the AZ matches.
• Attaching a second ENI does not double bandwidth. Bandwidth is set by the instance type, not by the number of cards.
• If an instance has more than one ENI, its auto-assigned public IPv4 is not given back after a stop and start. Use an Elastic IP when a multi-ENI instance needs a fixed public address.

#What ENIs are used for

• Failover by moving a secondary ENI. Clients connect to a private IP (and maybe an Elastic IP) on a secondary ENI. If the instance fails, you detach the ENI and attach it to a standby in the same AZ. The standby now has the same private IP, the same MAC, the same security groups and the same Elastic IP, and clients need no change.
• Dual-homed instances. One instance with an ENI in a production subnet and another in a management subnet, each with its own security groups, for example a monitoring appliance or a server that admins reach only over the management network.
• Software licensed to a MAC address. Keep the license on a secondary ENI so it survives replacing the instance.
• Network and security appliances that need one interface per network.

#ENI, ENA and EFA

Three names that sound alike:

Exam shortcut: “HPC with MPI, inter-node communication at scale, OS bypass” → EFA (plus a cluster placement group). “Enhanced networking, more bandwidth, lower latency” on ordinary workloads → ENA.

#EC2 Hibernate

#What a normal stop and start does

When you stop an EC2 instance, its EBS volumes are kept, but everything in RAM is lost. When you start it again:

1. The instance usually lands on a new physical host.
2. The operating system boots from scratch.
3. Your application starts, connects to its dependencies and warms its caches (loads data into memory, compiles code, fills connection pools).

(On the very first start of a new instance, the user data script also runs before step 3; lesson 4.3 covers it.)

For many applications step 3 takes minutes, sometimes much longer, and the first users see a slow system.

#What hibernate does instead

With EC2 Hibernate, a stop works like hibernating a laptop:

1. You ask EC2 to stop with hibernation. The instance moves to stopping.
2. EC2 signals the operating system to suspend to disk: it freezes every process and writes the whole content of RAM to a file on the root EBS volume. Because the root volume is encrypted, the saved memory is encrypted too.
3. The operating system shuts down, and the instance becomes stopped. You pay for EBS storage (and any Elastic IP), not for the instance.
4. When you start it, the instance boots, the operating system reads the saved RAM back from the root volume and unfreezes the processes. Applications continue exactly where they were, with warm caches.

The operating system did not really restart, so its uptime keeps counting from where it was, and long-running processes are not broken. After a normal stop and start, uptime starts again from 0.

Across a hibernation, the instance keeps its private IPv4 and IPv6 addresses and its Elastic IPs. An auto-assigned public IPv4 is released and a new one is given at start, just like a normal stop. Data on instance store volumes is lost.

#Conditions for using hibernate

#When to use it

• Long-running processing that you want to pause and resume without losing progress held in memory.
• Services that take a long time to initialise: large in-memory caches, applications that pre-load data, Java services with long warm-up.
• Saving costs on a pre-warmed environment (for example a test or development instance) that you stop at night but want ready in seconds in the morning.

#Putting it together: the “big data fleet” scenario

A line that appears in many exam questions: “You have an application performing big data analysis hosted on a fleet of EC2 instances.” It means one server is not enough, so a group of instances shares the work, for example analysing the logs of millions of users, training a machine learning model or genomics research. The last line of the question tells you which part is being asked about:

#Try it yourself

# 1. One placement group of each strategy
aws ec2 create-placement-group --group-name hpc-pg   --strategy cluster
aws ec2 create-placement-group --group-name crit-pg  --strategy spread --spread-level rack
aws ec2 create-placement-group --group-name kafka-pg --strategy partition --partition-count 3
aws ec2 create-placement-group --group-name time-pg  --strategy precision-time
aws ec2 describe-placement-groups --query "PlacementGroups[].[GroupName,Strategy,PartitionCount]"

# 2. Launch into them: the whole cluster in ONE request, one instance type
aws ec2 run-instances --image-id $AMI --instance-type c7i.8xlarge --count 8 \
  --subnet-id $SUBNET_A --placement "GroupName=hpc-pg"
# ...and one Kafka broker into partition 2 of the partition group
aws ec2 run-instances --image-id $AMI --instance-type m7i.xlarge --count 1 \
  --subnet-id $SUBNET_A --placement "GroupName=kafka-pg,PartitionNumber=2"

# 3. On a broker: read the partition number from the instance metadata (IMDSv2)
TOKEN=$(curl -sX PUT http://169.254.169.254/latest/api/token \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/placement/partition-number    # → 2

# 4. Check that the clock uses Amazon Time Sync (169.254.169.123)
chronyc sources -v

# 5. A secondary ENI with a fixed private IP, attached as eth1, then moved to a standby
ENI=$(aws ec2 create-network-interface --subnet-id $SUBNET_A --private-ip-address 10.0.1.50 \
  --groups $SG_APP --query NetworkInterface.NetworkInterfaceId --output text)
aws ec2 attach-network-interface --network-interface-id $ENI --instance-id $PRIMARY --device-index 1
ATT=$(aws ec2 describe-network-interfaces --network-interface-ids $ENI \
  --query "NetworkInterfaces[0].Attachment.AttachmentId" --output text)
aws ec2 detach-network-interface --attachment-id $ATT        # add --force only as a last resort
aws ec2 attach-network-interface --network-interface-id $ENI --instance-id $STANDBY --device-index 1

# 6. Hibernate: enable at launch, with an encrypted root volume big enough for the RAM
aws ec2 run-instances --image-id $AL2023_AMI --instance-type m7i.large \
  --hibernation-options Configured=true \
  --block-device-mappings '[{"DeviceName":"/dev/xvda","Ebs":{"VolumeSize":30,"VolumeType":"gp3","Encrypted":true}}]'
aws ec2 stop-instances --instance-ids $ID --hibernate
aws ec2 start-instances --instance-ids $ID
uptime          # on the instance: the counter continued from before the hibernation

#Summary

Key takeaways

• A placement group tells EC2 how to place instances on hardware. It is free; pick the strategy from the scenario.
• Cluster: one AZ, packed close, lowest latency, 10 Gbps single flows, highest risk of correlated failure. Launch all instances at once, same type, enhanced networking. Keywords: HPC, low latency, high throughput.
• Spread: each instance on distinct hardware, 7 running instances per AZ per group, can span AZs. Keywords: a small number of critical instances kept apart.
• Partition: up to 7 partitions per AZ, each on its own racks, hundreds of instances, partition number in the metadata. Keywords: HDFS, HBase, Cassandra, Kafka.
• Insufficient capacity means no hardware meets the group’s rule right now: launch in one request, stop and start the whole cluster group and retry, use another AZ or group, or retry later. It is not a quota problem.
• Precision time groups place instances on hardware with direct access to high-precision time; every instance can use Amazon Time Sync at 169.254.169.123 for free.
• An ENI is a virtual network card: primary and secondary private IPs, one Elastic IP per private IP, a public IP, security groups, a MAC. It is bound to one AZ. The primary ENI never detaches; a secondary ENI can be moved to a standby for failover. The number of ENIs depends on the instance type.
• ENA = enhanced networking; EFA = ENA plus OS bypass for HPC/MPI, used with cluster groups.
• Hibernate writes RAM to the encrypted root EBS volume and restores it on start, so processes and caches survive and uptime continues. It must be enabled at launch; RAM under 150 GB (Linux), root volume big enough, no more than 60 days, On-Demand and Spot.

Self-check: placement groups, ENIs and hibernate30 questions