Health Checks and DNS Failover

By the end of this lesson you will be able to

• Explain why plain DNS keeps sending users to a dead server, and how Route 53 health checks turn DNS into automatic failover.
• Name the three health check types (endpoint, calculated, CloudWatch alarm) and pick the right one from an exam scenario.
• Describe how endpoint checks work: about 15 checkers in several Regions, 30 s or 10 s intervals, the failure threshold, the 18% rule, 2xx/3xx, string matching in the first 5,120 bytes, and the checker IP ranges your firewalls must allow.
• Build calculated health checks with AND, OR and “at least N of M”, and use them for maintenance.
• Monitor private resources (private EC2, RDS, DynamoDB, SQS, on-premises) with a CloudWatch alarm and a health check on that alarm.
• Predict what Route 53 answers when records are healthy, unhealthy, unchecked or all unhealthy, and how Evaluate Target Health works for Alias records.
• Calculate DNS failover time from the interval, threshold and TTL, and know when Application Recovery Controller is the better tool.

#Why DNS needs health checks

A DNS server does not know whether the server behind an IP address is alive. If www.example.com points to 198.51.100.10 and that server crashes, plain DNS keeps handing out 198.51.100.10 until someone edits the record by hand. Users get timeouts at three in the morning while the record looks perfectly fine. (For how resolvers, caching and TTLs work in general, see the System Design lesson DNS.)

A Route 53 health check is a small monitor that Route 53 runs for you. It keeps testing something (a web server, another health check, or a CloudWatch alarm) and holds a status: healthy or unhealthy. You then attach the health check to DNS records. When the status turns unhealthy, Route 53 stops answering with that record and answers with a healthy one instead. That is automated DNS failover: no human edits a record during the outage.

A typical picture: an Application Load Balancer and an Auto Scaling group in us-east-1, the same stack in eu-west-1, a health check on each, and records (latency, failover, weighted, geoproximity…) that use those health checks. If one Region’s stack stops answering, Route 53 sends everyone to the other.

#The three types of health check

Route 53 supports exactly three kinds of health check. Exam questions love to offer a fourth one that does not exist.

CloudWatch is AWS’s metrics and alarms service; it gets its own chapter later. For now: services publish numbers (metrics) to it every minute, and an alarm watches one metric and goes into the alarm state when it crosses a threshold.

#Endpoint health checks

#How the checkers decide

When you create an endpoint health check, Route 53 health checkers in data centres around the world start sending requests to your endpoint.

• About 15 checkers. AWS does not publish a fixed number; the course slides and most exam material say about 15, spread over the health-checker Regions (us-east-1, us-west-1, us-west-2, eu-west-1, ap-southeast-1, ap-southeast-2, ap-northeast-1, sa-east-1). The documentation says that at a 30-second interval your endpoint receives a request about every two seconds on average, which matches about 15 checkers.
• Interval: 30 seconds (standard) or 10 seconds (fast). Each checker waits this long between checks. Fast costs extra. You cannot change the interval after you create the health check. The checkers do not coordinate, so you may see bursts of several requests per second and then a quiet gap.
• Failure threshold: 1 to 10, default 3. The number of consecutive checks a checker must fail before it considers the endpoint unhealthy, and also the number it must pass to consider it healthy again.
• Protocols: HTTP, HTTPS and TCP.
  • TCP: the checker must open a TCP connection within 10 seconds.
  • HTTP / HTTPS: a TCP connection within 4 seconds, then an HTTP status code of 2xx or 3xx within 2 seconds. 4xx and 5xx fail. A redirect (301, 302) passes.
  • HTTPS checks do not validate the certificate, so an expired certificate does not fail the check. Turn on SNI if your endpoint needs the host name in the TLS handshake to pick the right certificate. The endpoint must support TLS 1.0, 1.1 or 1.2.
• What you point it at: an IP address (IPv4 or IPv6, public only; for an EC2 instance use an Elastic IP so the address never changes) or a domain name (Route 53 resolves it at every interval, IPv4 only). Add a port and, for HTTP(S), a path such as /health.
• Route 53 aggregates the checkers’ results. A new health check counts as healthy until there is enough data.

#The 18% rule

Each checker reaches its own verdict. Route 53 then counts them:

• If more than 18% of health checkers report the endpoint healthy, the health check is healthy.
• If 18% or fewer report healthy, it is unhealthy.

The bar is low on purpose. Route 53 wants checkers in several Regions to agree, but it must not fail over just because a network problem cut off one or two checker locations from your endpoint. With 15 checkers, 3 healthy (20%) keeps the check healthy; 2 healthy (13.3%) makes it unhealthy. AWS notes that the value might change in a future release, so do not build logic on the exact number.

#String matching

A status code can lie. A maintenance page, or a friendly “please try again later” page, often comes back as 200. With string matching (HTTP or HTTPS only, chosen when you create the check, extra cost), the checker also reads the response body and looks for a text you choose:

• The text must appear entirely within the first 5,120 bytes of the body.
• The body must arrive within 2 seconds of the status code.
• The search is case-sensitive, up to 255 characters.
• A compressed body is unpacked only if it uses gzip or deflate; with any other compression the search almost always fails.

A good pattern: a small /health page that really tests the dependencies (database, cache) and prints a marker such as "status":"ok" at the very start.

#Choosing which Regions check you

By default Route 53 uses checkers in all the recommended Regions. You can customise the list, for example when all your users are in Asia or when the checker traffic itself is a burden. You must keep at least three Regions. When you remove a Region, its checkers keep checking for up to an hour so that the endpoint is never left unchecked.

#Let the checkers in

Health checkers are ordinary clients on the internet. If anything between them and your endpoint drops their packets, every checker sees a timeout, the check turns unhealthy, and Route 53 fails away from a healthy application.

• AWS publishes the checker ranges in https://ip-ranges.amazonaws.com/ip-ranges.json under "service": "ROUTE53_HEALTHCHECKS". (ROUTE53 is the name servers, needed only if you block DNS queries; ROUTE53_HEALTHCHECKS_PUBLISHING is internal and needs no rule.) The older GetCheckerIpRanges API still returns the same list.
• In a security group, the easiest way is the AWS-managed prefix list com.amazonaws.<region>.route53-healthchecks (and com.amazonaws.<region>.ipv6.route53-healthchecks for IPv6), which Route 53 keeps up to date. The lists have existed since 2023.
• Open the same ranges on network ACLs (lesson 3.5), and on on-premises routers and firewalls for endpoints in your own data centre. Allow only the health check port.

#Watching health checks: CloudWatch metrics, latency graphs and alarms

Health checks are integrated with CloudWatch.

• Route 53 publishes health check metrics in the AWS/Route53 namespace, dimension HealthCheckId, and only in US East (N. Virginia), us-east-1. If you look for them in another Region’s CloudWatch console, they are not there.
• HealthCheckStatus is 1 (healthy) or 0 (unhealthy); HealthCheckPercentageHealthy is the share of checkers that say healthy (endpoint checks); ChildHealthCheckHealthyCount counts healthy children (calculated checks).
• Latency graphs (optional, extra cost, chosen at creation): ConnectionTime, SSLHandshakeTime and TimeToFirstByte from checkers in each Region, shown in the console.
• Notifications: create a CloudWatch alarm on HealthCheckStatus (Minimum < 1) that publishes to an SNS topic, which e-mails or pages your team. Because the metric lives in us-east-1, the alarm and its SNS topic are created there too. Several minutes can pass between the failure and the e-mail.
• The console shows each checker’s last failure reason (timeout, HTTP 503, string not found, connection refused), which is the fastest way to debug a red check.

SNS (Simple Notification Service) is AWS’s publish/subscribe messaging service; it gets its own chapter later.

#What health checks cost

Prices for us-east-1 at the time of writing (check the Route 53 pricing page before you rely on them):

The first 50 health checks of AWS endpoints are free. A disabled health check is still billed. Example: an HTTPS check with string matching and the fast interval on an ALB costs 0.50 + 3 × 1.00 = $3.50 a month.

#Calculated health checks

A calculated health check (the parent) monitors other health checks (the children) and turns their statuses into one:

• AND: healthy only when all children are healthy.
• OR: healthy when at least one child is healthy.
• At least N of M: healthy when N or more children are healthy. N = 0 means always healthy; N greater than the number of children means always unhealthy.
• NOT: invert the parent (or a child), so healthy becomes unhealthy and the other way round.

One parent can monitor up to 255 child health checks (the console and API accept 256). The children can be endpoint checks or CloudWatch alarm checks, but not other calculated checks: there is only one level.

The notes’ example: you have three web servers. You build a calculated check with the rule “healthy if at least 2 of the 3 are healthy” and attach it to the primary failover record. One server can die, or be taken down for maintenance, without the whole site failing over. That is the classic use case: maintenance without causing every health check to fail.

Two details that trap people:

• A disabled health check is always considered healthy, also as a child. If you want a child to count as unhealthy on purpose (to drain a server), invert it instead of disabling it.
• The children’s own thresholds still apply: a child turns unhealthy only after its own failure threshold, and then the parent recalculates.

#Private resources: health checks on CloudWatch alarms

Route 53 health checkers live outside your VPC, on the internet. They cannot reach private endpoints: an EC2 instance with only a private IP, an RDS database in a private subnet, an internal load balancer, or a server in your data centre reached over VPN or Direct Connect (private links that get their own chapter later). Route 53 even refuses to create an endpoint health check for a private, local or non-routable address.

The solution is a three-step chain:

1. Publish a metric

   The resource reports a number to CloudWatch from inside AWS, so no internet path is needed: the EC2 StatusCheckFailed metric, RDS ReplicaLag or FreeStorageSpace, DynamoDB throttled requests, SQS ApproximateNumberOfMessagesVisible, or a custom metric from your application or from the CloudWatch agent on an on-premises server.

2. Create a CloudWatch alarm on it

   For example: alarm when StatusCheckFailed is 1, or when the queue holds more than 10,000 messages for five minutes.

3. Create a health check that monitors the alarm

   Choose “State of CloudWatch alarm”. Alarm OK → healthy; alarm in the alarm state → unhealthy. Attach this health check to your records exactly like an endpoint check.

Rules for the alarm, from the Route 53 documentation:

• It must be in the same AWS account as the health check (no cross-account alarms).
• Standard-resolution metrics only, with a simple statistic (Average, Minimum, Maximum, Sum, SampleCount); no extended (percentile) statistics, no metric math and no “M out of N” alarms.
• Route 53 watches the alarm’s data stream, not the alarm state itself, so you cannot force a failover with the SetAlarmState API.
• If you change the alarm, update the health check.
• Insufficient data: you choose what the health check reports when CloudWatch lacks data: healthy, unhealthy or last known status (a new check with no history counts as healthy). AWS advises against last known status when the metric may be missing for hours.

The owner’s practice question, in full:

#How records use health checks

A health check on its own only holds a status (and can send alerts). It changes DNS answers only when you associate it with records.

Rules Route 53 follows when it chooses a record:

1. It never runs a check at query time. Checks run all the time in the background; a query reads the latest status.
2. A record without a health check is always healthy. In a group where some records have health checks and one does not, the unchecked one is always eligible. Put a health check on every non-Alias record of a group.
3. Alias records use Evaluate Target Health instead (lesson 8.2). With Evaluate Target Health = Yes, the Alias record inherits the health of its target:
   • ALB / NLB: healthy only if every target group that has targets contains at least one healthy target; a target group with no registered targets counts as unhealthy. These are the load balancer’s own health checks (lesson 6.2), free, and AWS says do not create Route 53 health checks for the instances behind a load balancer.
   • Another record in the same hosted zone: healthy if at least one record of the target group is healthy, which is how you build trees (latency Alias → weighted records → endpoint checks).
   • CloudFront: you cannot set Evaluate Target Health to Yes. S3 websites, API Gateway, VPC interface endpoints, Global Accelerator: allowed but pointless, because these services are highly available by design; use a Route 53 health check that tests your application through them instead.
4. If no record in the group is healthy, all of them are treated as healthy. Route 53 must answer something and has no basis to choose, so it routes normally as if every record were healthy. For a failover pair where both have health checks and both are unhealthy, it returns the primary.
5. Weight 0 records are used only when every nonzero-weight record is unhealthy.

#DNS failover timing

How long after a server dies do users reach the secondary? Add three pieces:

1. Detection: up to interval × failure threshold. Default: 30 s × 3 = 90 s (the server can die just after a successful check). The 18% rule flips the status when the slower checkers are still counting, so it is often a few seconds less.
2. Propagation: the new status reaches the Route 53 name servers, usually within seconds.
3. Caching: resolvers that cached the old answer keep it until the TTL expires. Worst case, a full TTL. Some clients and applications (old Java settings, browsers) cache longer than the TTL.

Worked example. A failover record pair with TTL 60 s and a default health check (30 s, threshold 3):

• detection ≤ 90 s, plus TTL ≤ 60 s, gives a worst case of about 150 s, two and a half minutes;
• with the fast interval and threshold 1 (10 s × 1 = 10 s) and TTL 60 s: about 70 s;
• with the default check and a TTL of one day (86,400 s): detection still takes 90 s, but some users keep the dead address for up to 24 hours. The health check worked; the TTL defeated it.

Alias records to AWS resources do not have a TTL you can set: Route 53 uses the target’s TTL (60 seconds for a load balancer, lesson 8.2). Lower thresholds and the fast interval detect sooner but can flap: a single slow response sends traffic away and back. The same threshold also slows failback: after the fix, the endpoint must pass that many checks in a row, and then Route 53 returns to the primary automatically.

#What goes wrong

#Beyond health checks: Application Recovery Controller (side notes)

Health checks make failover automatic, which is what you want for most outages. For large, regulated or very expensive applications, teams often want the opposite: a human-approved, predictable switch that does not depend on a probe deciding whether a Region is “bad enough”, and that keeps working even while parts of AWS are impaired.

• Amazon Application Recovery Controller (ARC) (formerly Route 53 ARC) offers routing controls: simple on/off switches grouped on control panels. Each routing control is tied to a special Route 53 health check of the recovery control type: switch on = healthy, off = unhealthy. Attach those health checks to failover records, and flipping a switch moves traffic.
• Routing controls run on a cluster of endpoints in five AWS Regions (a highly available data plane), so you can fail over even if one Region, or the control plane, is having trouble. Safety rules stop operators from switching everything off at once.
• Readiness checks (checking that the standby Region has the same capacity and configuration) are no longer open to new customers; AWS recommends ARC Region switch, a managed multi-Region recovery orchestrator (2025) with plan evaluation, instead. Zonal shift and zonal autoshift move traffic away from one AZ for load balancers.
• Route 53 accelerated recovery (November 2025) keeps a copy of a public hosted zone’s records in US West (Oregon) so that you can still change records within about an hour if us-east-1 is impaired. DNS answers were never affected; it protects the ability to edit.
• Other recent changes: a new health checks console (2024) and the managed prefix lists for checker IPs (2023), both covered above. The health check rules themselves (18%, 30/10 s, thresholds, 5,120 bytes) have not changed.

For the exam: automatic DNS failover → health checks + failover routing; “switch traffic between Regions manually and reliably without depending on health checks or control-plane changes” → ARC routing controls.

#Health checks in system design

• Active-passive DR (lesson 8.3’s failover policy): a primary with an endpoint or calculated check, a secondary that is also checked, TTL 60 s. Add a static S3 “sorry” page as a last resort for the case where everything is down.
• Active-active: latency or weighted records per Region, each an Alias to a load balancer with Evaluate Target Health = Yes, so a Region with no healthy targets drops out by itself.
• Check the right layer. Load balancer health checks decide which instances get requests (lesson 6.2); Route 53 health checks decide which Region or endpoint gets users. Do not duplicate them.
• Health of private tiers through CloudWatch alarms; combined health of a Region through a calculated check (web tier AND database alarm, or “at least 2 of 3” web servers).
• Alert on every health check with a CloudWatch alarm and SNS in us-east-1: a failover that nobody knows about is a second outage waiting for the secondary to fail.
• Test failover regularly (invert the primary’s check in a game day). Many DR plans fail because the secondary was never exercised.

#Try it yourself

# 1. An HTTPS endpoint check with string matching, threshold 3, standard 30 s interval
aws route53 create-health-check --caller-reference web-use1-$(date +%s) \
  --health-check-config '{
    "Type": "HTTPS_STR_MATCH",
    "FullyQualifiedDomainName": "us-east-1-www.example.com",
    "Port": 443, "ResourcePath": "/health",
    "SearchString": "\"status\":\"ok\"",
    "RequestInterval": 30, "FailureThreshold": 3,
    "EnableSNI": true, "MeasureLatency": true,
    "Regions": ["us-east-1", "eu-west-1", "ap-southeast-1"]
  }'

# 2. What each checker sees right now, and why it last failed
aws route53 get-health-check-status --health-check-id 1a2b3c4d-1111-2222-3333-444455556666
aws route53 get-health-check-last-failure-reason --health-check-id 1a2b3c4d-1111-2222-3333-444455556666

# 3. A calculated check: healthy if at least 2 of 3 children are healthy
aws route53 create-health-check --caller-reference web-tier-$(date +%s) \
  --health-check-config '{
    "Type": "CALCULATED", "HealthThreshold": 2,
    "ChildHealthChecks": ["<web-1-id>", "<web-2-id>", "<web-3-id>"]
  }'

# 4. A health check on a CloudWatch alarm (for a private resource)
aws route53 create-health-check --caller-reference db-alarm-$(date +%s) \
  --health-check-config '{
    "Type": "CLOUDWATCH_METRIC",
    "AlarmIdentifier": {"Region": "eu-west-1", "Name": "orders-db-replica-lag"},
    "InsufficientDataHealthStatus": "LastKnownStatus"
  }'

# 5. Attach a health check to the primary failover record
aws route53 change-resource-record-sets --hosted-zone-id Z0123456789EXAMPLE \
  --change-batch '{"Changes": [{"Action": "UPSERT", "ResourceRecordSet": {
    "Name": "www.example.com", "Type": "A", "TTL": 60,
    "SetIdentifier": "primary", "Failover": "PRIMARY",
    "HealthCheckId": "1a2b3c4d-1111-2222-3333-444455556666",
    "ResourceRecords": [{"Value": "198.51.100.10"}]}}]}'

# 6. Game day: invert the primary's check to force a failover, then put it back
aws route53 update-health-check --health-check-id 1a2b3c4d-1111-2222-3333-444455556666 --inverted
aws route53 update-health-check --health-check-id 1a2b3c4d-1111-2222-3333-444455556666 --no-inverted

# 7. Get an e-mail when the check turns unhealthy (metrics and alarm live in us-east-1)
aws cloudwatch put-metric-alarm --region us-east-1 --alarm-name www-primary-unhealthy \
  --namespace AWS/Route53 --metric-name HealthCheckStatus \
  --dimensions Name=HealthCheckId,Value=1a2b3c4d-1111-2222-3333-444455556666 \
  --statistic Minimum --period 60 --evaluation-periods 1 \
  --threshold 1 --comparison-operator LessThanThreshold \
  --alarm-actions arn:aws:sns:us-east-1:111122223333:oncall

Let the checkers in, and see their ranges:

# The managed prefix list for a security group (Region of the endpoint)
aws ec2 describe-managed-prefix-lists \
  --filters Name=prefix-list-name,Values=com.amazonaws.eu-west-1.route53-healthchecks
aws ec2 authorize-security-group-ingress --group-id sg-0web0123456789abc \
  --ip-permissions 'IpProtocol=tcp,FromPort=443,ToPort=443,PrefixListIds=[{PrefixListId=pl-0123456789abcdef0}]'

# The raw ranges from ip-ranges.json
curl -s https://ip-ranges.amazonaws.com/ip-ranges.json \
  | jq -r '.prefixes[] | select(.service=="ROUTE53_HEALTHCHECKS") | "\(.region) \(.ip_prefix)"'

Watch a failover from the outside (run it every few seconds during a game day):

dig +noall +answer www.example.com          # answer and the remaining TTL from your resolver
dig +noall +answer www.example.com @ns-123.awsdns-15.com   # straight from Route 53, no cache

The first command shows what your resolver has cached and how many seconds of TTL are left; the second asks a Route 53 name server of your zone directly and shows the new answer as soon as the health check flips.

#Summary

Key takeaways

• A health check holds a healthy/unhealthy status; associated with records, it gives automated DNS failover. Exactly three types: endpoint, calculated (monitors other health checks) and CloudWatch alarm. There is no SQS, RDS or DynamoDB health check.
• Endpoint checks: about 15 checkers in several Regions, HTTP, HTTPS or TCP, interval 30 s or 10 s (fast, extra cost), failure threshold 1–10, default 3, healthy if more than 18% of checkers say so, pass only on 2xx/3xx (TCP within 4 s, response within 2 s), optional string match in the first 5,120 bytes, choose at least 3 checker Regions, HTTPS does not validate certificates.
• Allow the checkers in: ROUTE53_HEALTHCHECKS ranges in ip-ranges.json, or the managed prefix list com.amazonaws.<region>.route53-healthchecks, in security groups, NACLs and on-premises firewalls.
• Calculated checks: up to 255 children (API 256), AND / OR / NOT / at least N of M, one level only, disabled children count healthy; use them for maintenance without failover.
• Private resources (private EC2, RDS, on-premises, private hosted zones, SQS backlog, DynamoDB throttles): checkers are outside the VPC, so use a CloudWatch metric → alarm → health check on the alarm (same account, standard resolution, choose the insufficient-data behaviour).
• Records: simple has no health checks; failover primary needs one; records without a check are always healthy; Alias uses Evaluate Target Health (ALB/NLB: every target group with targets needs a healthy target; not for CloudFront); all unhealthy → all treated healthy (failover: primary).
• Failover time ≈ interval × threshold + TTL (default 90 s + 60 s ≈ 2.5 min). Short TTLs on records that fail over; failback is automatic after the same number of passes.
• Metrics and alarms for health checks live in us-east-1; use SNS for alerts. For manual, reliable Region switches use ARC routing controls.

Self-check: Route 53 health checks and DNS failover31 questions