A Highly Available Single EC2 Instance

By the end of this lesson you will be able to

• Say what high availability really promises (less downtime, not zero) and why one EC2 instance in one Availability Zone can never provide it.
• Tell apart what EC2 automatic recovery fixes (a broken host) from what it cannot (a lost AZ, a hung application, an instance in an Auto Scaling group).
• Build pattern 1: an Elastic IP moved to a standby instance by a CloudWatch alarm or EventBridge rule and a Lambda function, and name its weak spots.
• Build pattern 2: an Auto Scaling group with min 1, max 1, desired 1 across two AZs whose user data re-attaches the Elastic IP through an instance role.
• Build pattern 3: carry an EBS volume to another AZ with terminate and launch lifecycle hooks and snapshots, with DLM snapshots as the safety net, and work out the data you lose when the AZ dies hard.
• Choose a better design when you can change the application: EFS Regional, RDS Multi-AZ, a load balancer, or a Route 53 failover record.
• Compare every option by failover time, data loss, cost and what breaks, and pick the exam answer from the question’s words.

#Why one instance is a problem

Most of this course makes applications highly available the easy way: several instances in several AZs behind a load balancer (lesson 6.1). That only works when the application is stateless, so any copy can serve any request, and when you are allowed to run several copies.

Some servers are neither. A licence bound to one host, an old desktop-era program with its database file on the local disk, a vendor appliance that refuses to run twice. You must run exactly one copy. This lesson is about making that one copy survive as much as possible, with the three designs on the slides and the better alternatives around them.

#High availability means less downtime, not none

The notes define high availability as a system that “never goes down”. That is not what the term means, and no design on AWS promises it. High availability means the system keeps downtime low: it is designed so that a single failure (an instance, a host, an AZ) causes at most a short interruption, usually measured as an availability percentage (99.9% allows about 8.8 hours of downtime a year, 99.99% about 53 minutes). Every pattern in this lesson has a few minutes of downtime when it fails over. A design with zero interruption and full capacity after a failure is fault tolerant, which costs more (lesson 6.1 draws the line).

What the notes get right: to be highly available on AWS, the system must be able to run in more than one AZ.

#One instance lives in one Availability Zone

An EC2 instance runs on one host in one data centre of one AZ. Its EBS volumes are locked to that AZ too (lesson 5.1). If the AZ loses power, cooling or networking, the instance and its data are inside the failure, and nothing on a one-instance diagram is outside it.

#What you get for free: automatic recovery

Before building anything, know what EC2 already does. When the system status check fails (the host’s hardware, power or network), simplified automatic recovery, on by default for supported instance types, moves the instance to a healthy host in the same AZ. A CloudWatch alarm on StatusCheckFailed_System with the recover action does the same, faster and with your own notification. The instance keeps its ID, private and public IPs, Elastic IP and EBS volumes; only RAM is lost. All of this is taught in lesson 19.3.

What it does not do:

• It cannot leave the AZ, so it does nothing for an AZ failure.
• It does not fix a hung application or a broken operating system (the instance status check needs the reboot action instead).
• It is not supported for instances in an Auto Scaling group (the group replaces them instead), nor with an Elastic Fabric Adapter, on Dedicated Hosts, or on unsupported types.

So automatic recovery is the right first layer for any single instance, and the patterns below add the missing piece: another AZ.

#Pattern 1: an Elastic IP and a standby instance

The first slide’s design keeps a second, stopped instance ready and moves the address to it:

1. A primary instance serves traffic through an Elastic IP (a static public IPv4 address you own in the Region, lesson 3.6).
2. A standby instance, built from the same AMI, sits stopped in another AZ. A stopped instance costs only its EBS storage.
3. Something detects the failure: a CloudWatch alarm on the status checks (or on an application metric), or an EventBridge rule on an event such as the instance’s state change. The slide says “CloudWatch Event”, which is EventBridge today (lesson 19.5).
4. The trigger invokes a Lambda function that calls StartInstances on the standby and then AssociateAddress to move the Elastic IP to it.
5. Users keep using the same address and land on the standby.

# The failover function (boto3), invoked by the alarm or the EventBridge rule
import boto3
ec2 = boto3.client("ec2")

def handler(event, context):
    ec2.start_instances(InstanceIds=["i-0standby0example"])
    ec2.get_waiter("instance_running").wait(InstanceIds=["i-0standby0example"])
    ec2.associate_address(AllocationId="eipalloc-0clinic0example",
                          InstanceId="i-0standby0example")  # moves it off the dead primary

#Details that decide whether it works

• Put the standby in another AZ. A standby next to the primary dies in the same AZ outage.
• Detect silence, not only failure. When a whole AZ goes down, the instance stops sending metrics instead of reporting a failed check. An alarm with the default missing-data setting would stay in INSUFFICIENT_DATA. Set treat missing data as breaching, or detect from outside with a Route 53 health check.
• Moving an address is one call. In today’s accounts AssociateAddress re-associates an address that is attached elsewhere automatically; --allow-reassociation is the old explicit flag, and --no-allow-reassociation makes the call fail instead. The Lambda function’s role needs ec2:StartInstances and ec2:AssociateAddress, nothing more.
• A stopped instance holds no capacity. In a large AZ event many customers start instances in the surviving AZs at once, and StartInstances can fail with InsufficientInstanceCapacity. An On-Demand Capacity Reservation in the standby’s AZ guarantees it, but you pay for it all the time (lesson 4.4).
• The standby has its own disk. It only has the data you copied to it. Without replication, the standby comes up with yesterday’s, or last month’s, data. It also drifts: patches and configuration applied to the primary must be applied to the standby.
• Fail back on purpose. When the old AZ returns, nothing moves back by itself. Sync the data and move the address back at a quiet hour.

#Pattern 2: an Auto Scaling group of exactly one

The second slide replaces the hand-made standby with an Auto Scaling group (lesson 6.5):

When the instance fails a health check, the group terminates it and launches a new one, in whichever of its AZs is healthy. Because max is 1, it terminates first and launches second: there is no overlap.

#The replacement has a new IP address

A new instance gets a new private IP and a new random public IP. Users, DNS records and partners know only 203.0.113.10. The ASG replaces instances; it does not move addresses. So the launch template’s user data does it at boot (lesson 4.3):

#!/bin/bash
# User data: find the clinic's Elastic IP by tag and attach it to this instance
TOKEN=$(curl -sX PUT http://169.254.169.254/latest/api/token -H "X-aws-ec2-metadata-token-ttl-seconds: 300")
ID=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id)
ALLOC=$(aws ec2 describe-addresses --filters Name=tag:app,Values=clinic \
          --query 'Addresses[0].AllocationId' --output text)
aws ec2 associate-address --allocation-id "$ALLOC" --instance-id "$ID" --allow-reassociation

For that to work, the launch template attaches an instance profile whose role allows the calls:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "ec2:DescribeAddresses", "Resource": "*" },
    {
      "Effect": "Allow",
      "Action": "ec2:AssociateAddress",
      "Resource": "*",
      "Condition": { "StringEquals": { "aws:ResourceTag/app": "clinic" } }
    }
  ]
}

(AssociateAddress is authorised against both the address and the instance; the tag condition above matches when the address carries app=clinic and the instance is launched with the same tag. If you do not tag instances, drop the condition and keep the action.) Finding the address by tag rather than hard-coding it lets the same template run in test and production, each with its own address.

#What counts as a failure: the health check type

• EC2 (the default): the instance is not running, or a status check fails. This sees a dead host or a dead AZ, but not a hung application: the operating system answers its checks while the web server returns errors. The group keeps a zombie.
• ELB: available only with a load balancer in front. It adds the load balancer’s check of the application.
• Custom: with no load balancer, a watchdog (a cron script on another machine, a CloudWatch alarm on an application metric, a Route 53 health check feeding a Lambda function) calls aws autoscaling set-instance-health --health-status Unhealthy, and the group replaces the instance.

#How long it takes, and what it costs

Downtime is detection + termination + launch + boot + user data: a few minutes of failed status checks, about a minute to launch, one to two minutes to boot and run the script. Typically 3 to 10 minutes. This is recovery, not seamless high availability, and the notes’ “if it crashes it starts automatically in another AZ” should be read that way.

The cost is the reason the exam likes it: an Auto Scaling group is free; you pay for the one instance and the Elastic IP (every public IPv4 address, Elastic or not, costs 0.005 USD an hour since February 2024, about 3.65 USD a month).

What it does not solve: the replacement boots from the AMI, with a fresh root volume. Anything the old instance wrote to its own disks is gone. That is pattern 3’s job.

#Pattern 3: an ASG that carries its EBS volume

The third slide adds the hard part: the instance keeps data on an EBS volume, and the replacement must have it.

#The problem: a volume cannot leave its AZ

An EBS volume in ap-southeast-1a can be attached only to instances in ap-southeast-1a. There is no move, and attaching it to an instance in 1b fails. The only road between AZs is the one from lesson 5.1: snapshot → create a volume from the snapshot in the target AZ → attach. It works because snapshots are Regional: they are stored by EBS (in S3 that AWS manages) for the whole Region, not inside one AZ.

#The terminate hook: snapshot on the way out

A terminating lifecycle hook pauses the old instance in Terminating:Wait before it is terminated (lesson 6.5 teaches hooks):

1. The hook sends an event to EventBridge, which invokes a Lambda function (or an SSM Automation document).
2. The function, ideally after stopping the application or freezing the file system with an SSM Run Command so the data is consistent, calls CreateSnapshot on the data volume and tags the snapshot (app=clinic, role=data).
3. When the snapshot is completed, it calls complete-lifecycle-action, and termination goes ahead.

#The launch hook: a volume in the new AZ

A launching lifecycle hook holds the replacement in Pending:Wait, before it counts as in service:

1. EventBridge invokes the function again, with the new instance’s ID.
2. By default the group launches the replacement while the old instance is still terminating, so the function first waits until the terminate hook’s final snapshot is completed (or that hook has given up). Then it finds the newest completed snapshot with the tags, and calls CreateVolume with the replacement’s AZ.
3. It waits for the volume to be available, calls AttachVolume, then complete-lifecycle-action with CONTINUE.
4. User data (or a systemd unit) waits for the device, mounts it, starts the application, and attaches the Elastic IP as in pattern 2.

The notes do the same work in user data alone: find the latest snapshot, create a volume in the instance’s own AZ, attach it. That works too (the instance role then needs ec2:DescribeSnapshots, ec2:CreateVolume, ec2:AttachVolume, and ec2:CreateTags if you tag the volume), and it is the answer the notes give for the exam question below. The hook version is cleaner: the instance is not counted as in service until its data is attached, and the snapshot on termination gives a fresher copy.

#When the AZ dies hard: DLM is the safety net

The terminate hook only helps when the old AZ is still reachable, for example when the instance itself failed. If the whole AZ is gone, the volume is unreachable, CreateSnapshot fails or never completes, and nothing new is saved. The launch hook then builds the volume from the newest snapshot that already exists.

That is why the notes’ part of the design stays essential: regular snapshots by Amazon Data Lifecycle Manager (DLM), on a schedule (every 1, 2, 3, 4, 6, 8, 12 or 24 hours, or a cron expression, but never more often than once an hour), targeting the volume by tag, with retention by count or age. In a hard AZ failure, the data you lose is everything written since the last scheduled snapshot. With hourly snapshots, up to an hour of bookings. DLM starts each run within an hour of the scheduled time, so plan for some slack. That gap is the recovery point of this design; lesson 23.1 is where the chapter on disaster recovery defines and compares these numbers. Need less loss? Snapshot more often (below one hour DLM cannot help; an EventBridge Scheduler schedule or rule can call the EBS CreateSnapshot action, say every 15 minutes, with your own clean-up), or move the data off the instance (next section).

Two things to make the hook function do in this case: fail fast (if the volume cannot be snapshotted, call complete-lifecycle-action at once instead of waiting), and keep the hook’s heartbeat timeout reasonable. It is 1 hour by default; if nobody completes the action, the hook waits that long before the default result applies, and the launch hook, which waits for that final snapshot, waits with it. A big first snapshot may need more time, which a script can buy with record-lifecycle-action-heartbeat (up to 48 hours or 100 heartbeats, whichever is shorter).

#Details that bite

• Restored volumes start slow. A volume created from a snapshot loads its blocks from S3 on first access, so the first reads after failover are slower. Fast Snapshot Restore on the snapshot (paid per AZ per hour) removes the penalty (lesson 5.1).
• Crash-consistent is not application-consistent. A snapshot of a running database file may need recovery on mount. Stop or flush the application in the terminate hook; use DLM pre and post scripts for the scheduled ones.
• Tags are the glue. The launch hook must find the right snapshot. Tag consistently, and sort by StartTime, keeping only completed snapshots.
• Same AZ is possible. After an instance failure the group may launch the replacement in the same AZ. If the old volume was kept (DeleteOnTermination false), re-attaching it is faster than restoring a snapshot; a good function checks the AZ first.
• Clean up. Every failover leaves an old volume or a new one; DLM retention removes old snapshots, but stray volumes cost money until you delete them.

#The better answer: keep the data off the instance

Patterns 2 and 3 work hard to carry state that should not be on the instance in the first place. If you can change where the data lives, the problem mostly disappears (lesson 9.2 shows the same idea for web apps):

• Files on Amazon EFS with the Regional file system type: the data is stored across at least three AZs, every AZ has a mount target, and the replacement simply mounts the same file system (lesson 5.4). The notes say EFS is “natively Multi-AZ”: true only for Regional. EFS One Zone keeps the data in one AZ and fails with it, exactly like the EBS volume.
• Database on RDS Multi-AZ: a synchronous standby in another AZ and automatic failover in about one to two minutes, with the same endpoint name (lesson 7.1).
• Objects (scans, PDFs) in S3, which is Regional.

The instance then holds only the operating system and the application, which the AMI already has. The ASG of one replaces it in minutes, and no data is lost. Downtime is still the minutes of a replacement, because one instance is still one instance; what you gain is that recovery no longer costs data.

#Other ways to keep one address

The Elastic IP is the slides’ answer for “users must find the new instance”. It is not the only one, and AWS itself suggests using Elastic IPs sparingly (lesson 3.6).

#A load balancer in front, even for one instance

Put the ASG of 1/1/1 behind an Application Load Balancer and register it as a target group:

• Users use the ALB’s DNS name (or your alias record to it). No address ever moves, so no user data, no role, no Elastic IP.
• The ELB health check type tests the application, so a hung app is replaced too. With a tuned health check (for example a 10-second interval and 3 failures), detection takes about half a minute instead of several minutes.
• The instance can sit in a private subnet, and the ALB adds TLS termination and WAF.
• Cost: an ALB is about 16 USD a month plus capacity units. For a small clinic that is the main price of the cleanest design.

If partners must allow-list fixed IPs, a Network Load Balancer can have one Elastic IP per AZ; Global Accelerator gives two static anycast IPs (lesson 6.3).

#A Route 53 failover record

Instead of moving an IP, point the name somewhere else: a failover routing record with a primary (the active instance, with a health check) and a secondary (lesson 8.3, lesson 8.4). When the health check fails (by default after 3 failed checks at 30-second intervals), Route 53 answers with the secondary. The catch is DNS caching: clients keep the old answer for the record’s TTL, so use a short TTL (for example 60 seconds), and remember that some clients cache longer. It also needs a secondary that is running, or a script that updates the record after the ASG launches the replacement.

#Comparing the patterns

#What goes wrong

#How it is used in system design

The clinic’s final design, built step by step as budget allowed:

1. Day one (free): simplified automatic recovery is already on; add an alarm on StatusCheckFailed_System with the recover action and an SNS email, and an alarm on StatusCheckFailed_Instance with reboot.
2. First real fix: an ASG with 1/1/1 across ap-southeast-1a and 1b, user data that attaches the tagged Elastic IP, and an instance role for it. Cost: nothing extra.
3. The data: the vendor’s database file stays on an EBS volume (the licence needs a local path). Lifecycle hooks carry it across AZs; a DLM policy snapshots it hourly and keeps 48; FSR is enabled on the latest snapshot in both AZs during clinic hours.
4. Next year: the scanned reports move to EFS Regional, and when the vendor supports it, the database moves to RDS Multi-AZ. The hooks and DLM become unnecessary, and an ALB replaces the Elastic IP.

The pattern generalises: a single license server for engineering software, a legacy reporting box, a small game server, a self-hosted CI runner with a cache. If it cannot be a fleet, make it a self-healing one: ASG of one across AZs, state outside the instance or carried by snapshots, and an address that follows it.

#Try it yourself

# 1. Allocate and tag the Elastic IP the instance will look for
aws ec2 allocate-address --domain vpc \
  --tag-specifications 'ResourceType=elastic-ip,Tags=[{Key=app,Value=clinic}]'
# {"AllocationId": "eipalloc-0a1b2c3d4e5f6a7b8", "PublicIp": "203.0.113.10", "Domain": "vpc"}

# 2. Move it by hand to another instance (what pattern 1's Lambda function does)
aws ec2 associate-address --allocation-id eipalloc-0a1b2c3d4e5f6a7b8 \
  --instance-id i-0standby0example --allow-reassociation
# {"AssociationId": "eipassoc-0f1e2d3c4b5a69788"}

# 3. Pattern 2: a group of exactly one across two AZs
aws autoscaling create-auto-scaling-group --auto-scaling-group-name clinic-asg \
  --launch-template LaunchTemplateName=clinic-lt,Version='$Latest' \
  --min-size 1 --max-size 1 --desired-capacity 1 \
  --vpc-zone-identifier "subnet-0aaa1111example,subnet-0bbb2222example" \
  --health-check-type EC2 --health-check-grace-period 300

# 4. Pattern 3: the two lifecycle hooks (EventBridge rules on these hook events invoke the function)
aws autoscaling put-lifecycle-hook --auto-scaling-group-name clinic-asg \
  --lifecycle-hook-name snapshot-on-terminate \
  --lifecycle-transition autoscaling:EC2_INSTANCE_TERMINATING \
  --heartbeat-timeout 900 --default-result CONTINUE
aws autoscaling put-lifecycle-hook --auto-scaling-group-name clinic-asg \
  --lifecycle-hook-name volume-on-launch \
  --lifecycle-transition autoscaling:EC2_INSTANCE_LAUNCHING \
  --heartbeat-timeout 600 --default-result ABANDON

# 5. What the launch hook function does: newest tagged snapshot -> volume in the new AZ -> attach
aws ec2 describe-snapshots --owner-ids self \
  --filters Name=tag:app,Values=clinic Name=status,Values=completed \
  --query 'sort_by(Snapshots,&StartTime)[-1].[SnapshotId,StartTime]' --output text
# snap-0123456789abcdef0	2026-11-01T09:00:12.000Z
aws ec2 create-volume --snapshot-id snap-0123456789abcdef0 \
  --availability-zone ap-southeast-1b --volume-type gp3 \
  --tag-specifications 'ResourceType=volume,Tags=[{Key=app,Value=clinic}]'
# {"VolumeId": "vol-0fedcba9876543210", "AvailabilityZone": "ap-southeast-1b", "State": "creating", ...}
aws ec2 attach-volume --volume-id vol-0fedcba9876543210 \
  --instance-id i-0replacement0example --device /dev/sdf
aws autoscaling complete-lifecycle-action --auto-scaling-group-name clinic-asg \
  --lifecycle-hook-name volume-on-launch --instance-id i-0replacement0example \
  --lifecycle-action-result CONTINUE

# 6. The safety net: an hourly DLM policy for volumes tagged app=clinic, keeping 48
aws dlm create-lifecycle-policy --description "clinic hourly" --state ENABLED \
  --execution-role-arn arn:aws:iam::111122223333:role/AWSDataLifecycleManagerDefaultRole \
  --policy-details '{"PolicyType":"EBS_SNAPSHOT_MANAGEMENT","ResourceTypes":["VOLUME"],
    "TargetTags":[{"Key":"app","Value":"clinic"}],
    "Schedules":[{"Name":"hourly","CopyTags":true,
      "CreateRule":{"Interval":1,"IntervalUnit":"HOURS","Times":["00:00"]},"RetainRule":{"Count":48}}]}'
# {"PolicyId": "policy-0a1b2c3d4e5f6a7b8"}

# 7. Test it: kill the instance on purpose and time the gap
aws autoscaling set-instance-health --instance-id i-0abc123example --health-status Unhealthy
aws autoscaling describe-scaling-activities --auto-scaling-group-name clinic-asg --max-items 2

Run step 7 before you trust the design: a failover nobody has tested is a hope, not a plan.

#Summary

Key takeaways

• High availability means little downtime, not none, and it needs more than one AZ. One instance in one AZ is never the HA answer.
• Automatic recovery fixes a broken host in the same AZ (same IPs, same volumes); it cannot survive an AZ failure, does not fix a hung app, and does not apply inside an ASG.
• Pattern 1: a stopped standby in another AZ; a CloudWatch alarm or EventBridge rule invokes Lambda, which starts it and moves the Elastic IP. Watch for missing-data alarms, capacity and a stale standby.
• Pattern 2: an ASG of 1/1/1 across at least two AZs, the cheapest automatic answer. The replacement gets a new IP, so user data attaches the Elastic IP by tag through an instance role allowed to call ec2:AssociateAddress. Expect minutes of downtime.
• Pattern 3: EBS is locked to its AZ but snapshots are Regional. A terminate hook snapshots on the way out; a launch hook creates the volume from the latest snapshot in the new AZ and attaches it. In a hard AZ failure the terminate hook cannot run, so DLM snapshots set the data you lose.
• Better still, keep the data off the instance: EFS Regional (not One Zone), RDS Multi-AZ, S3. Even better for the address: an ALB in front, or a Route 53 failover record.

Self-check: a highly available single EC2 instance30 questions