Alarms, Composite Alarms, Recovery and Synthetics

By the end of this lesson you will be able to

• Explain what a CloudWatch alarm is: a rule on one metric, a metric math expression or an anomaly detection band, with three states (OK, ALARM, INSUFFICIENT_DATA) and actions on each change of state.
• Set period, evaluation periods and datapoints to alarm (M out of N), choose how missing data is treated, and predict the alarm state for a given series of datapoints.
• Choose between a static threshold alarm, an anomaly detection alarm and a composite alarm, and use composite rules and action suppression to cut alarm noise.
• List the actions an alarm can take today (SNS, Lambda, EC2 stop, terminate, reboot and recover, Auto Scaling, Systems Manager items and incidents, investigations) and which of them a composite alarm can use.
• Protect a single EC2 instance with the recover action on the system status check, say what recovery keeps and loses, and when simplified automatic recovery or an Auto Scaling group is the better answer.
• Test an application from the outside with CloudWatch Synthetics canaries, and a hybrid network path with Network Synthetic Monitor, and know where Internet Monitor fits.

#Why alarms: nobody watches graphs at 3 am

Lesson 19.1 showed how CloudWatch collects metrics, and lesson 19.2 how it keeps logs. Both are useless at 3 am if somebody has to stare at a dashboard to notice a problem. A CloudWatch alarm is the part that watches for you and does something when a number goes wrong: it sends a message, runs a function, scales a group or repairs an instance.

#The problem

A bKash-style mobile wallet pays out salaries at 10:00 on the day before Eid. Millions of people open the app within minutes. If the API's slowest one percent of requests (the p99 latency) climbs above one second, people start retrying, the retries add more load, and within ten minutes the app is down. The operations team needs to know at minute two, not at minute ten, and it does not want to be woken every time a single minute is a little slow.

#What an alarm is

A CloudWatch alarm watches one metric, one metric math expression (for example an error rate computed from two metrics) or an anomaly detection band, compares each datapoint with a threshold and decides on a state:

On every change of state the alarm can run actions: notify an SNS topic, invoke a Lambda function, stop or recover an EC2 instance, scale an Auto Scaling group, and more (all listed below). Every alarm is regional: it lives in one Region and watches metrics of that Region (cross-account observability from lesson 19.1 lets a monitoring account alarm on metrics of its source accounts).

#How an alarm decides

#What an alarm can watch

• A single metric with a statistic: Average, Sum, Minimum, Maximum, SampleCount or a percentile such as p99 (lesson 19.1). Dimensions pick the exact metric: the CPU of one instance (InstanceId) or of a whole Auto Scaling group (AutoScalingGroupName).
• A metric math expression that combines metrics, for example 100 * errors / requests for an error rate, so one alarm watches a ratio instead of two raw counts.
• An anomaly detection band around a metric or an expression (see “Three kinds of alarm” below).
• A metric created from logs by a metric filter (lesson 19.2), so log lines can raise alarms.
• Side note: an alarm can also run a Metrics Insights query (SQL-like) that covers many resources at once, for example every DynamoDB table, and track each one as a separate contributor; and newer alarm mute rules silence actions during scheduled maintenance windows. The exam's alarm is the classic one on one metric.

#Period, evaluation periods and datapoints to alarm

Three settings turn a stream of numbers into a decision:

• Period: the length of time, in seconds, that is summarised into one datapoint with the chosen statistic. Usually 60 seconds or more; for high-resolution custom metrics also 10, 20 or 30 seconds (high-resolution alarms cost more).
• Evaluation periods (N): how many of the most recent datapoints the alarm looks at.
• Datapoints to alarm (M): how many of those N must breach to go to ALARM. They do not have to be consecutive. M = N means “every one of the last N”; M smaller than N is an M out of N alarm, which tolerates the odd good datapoint in a run of bad ones.

An alarm with a period of one minute or more is evaluated every minute on a sliding window; a 10, 20 or 30 second alarm is evaluated every 10 seconds. If N × period is longer than a day, the alarm is evaluated once an hour.

A worked example for the wallet: period 60 seconds, statistic p99, threshold greater than 1 second, 3 out of 3. The earliest the alarm can go to ALARM is when the third breaching minute has been recorded, about three minutes after latency went bad. With 3 out of 5 it also fires when latency flaps (bad, good, bad, good, bad), which a strict 3 out of 3 would never catch. With a period of 5 minutes and 2 out of 3, it needs two breaching 5-minute datapoints: at least 10 minutes.

Two rules of thumb:

• The period must not be shorter than the interval at which the metric is published. EC2 with basic monitoring sends metrics every 5 minutes; an alarm with a 1-minute period then sees four empty periods out of five. Use a 5-minute period, or turn on detailed monitoring (1-minute, paid, lesson 19.1).
• Actions run on state changes, not continuously. An SNS message is sent once when the alarm enters ALARM, not every minute while it stays there. The one exception: Auto Scaling actions are repeated while the alarm remains in ALARM.

#When datapoints are missing

Datapoints go missing when a server stops, an agent crashes, a network link breaks, or simply because some metrics are only published when something happens (DynamoDB publishes ThrottledRequests only when a request is throttled). Each alarm says how to treat a missing datapoint:

How CloudWatch applies it: it fetches a few more datapoints than N (the evaluation range). If it finds at least N real datapoints, it decides on the most recent N real ones and the missing-data setting is not used at all. Only when there are fewer than N real datapoints does it fill the gaps with your setting, and as few times as it can. Two special cases from the documentation: alarms on DynamoDB metrics default to ignoring missing data, and AWS recommends missing (and actions only on ALARM) for alarms that stop, terminate, reboot or recover EC2 instances, so a brief gap in reporting never stops a healthy instance.

The classic trap: an alarm whose only action is on ALARM, on a metric that stops arriving. With the default setting it goes to INSUFFICIENT_DATA, not ALARM, and nobody is told. Either treat missing data as breaching, or add an action on INSUFFICIENT_DATA as well.

Try it: set the minutes, M, N and the missing-data treatment, and press Evaluate.

#Three kinds of alarm

#Static threshold alarms

The usual alarm (the notes call it a standard alarm; AWS calls it a metric alarm): one metric or expression, one fixed threshold, a comparison (greater than, greater or equal, less than, less or equal). “CPU above 80%”, “p99 above 1 second”, “free storage below 10 GB”, “queue depth above 10,000 for five minutes”. Simple and predictable, and the right choice when you know the number that means trouble.

#Anomaly detection alarms

Some metrics have no single “bad” number. Requests per minute are low at night, high every evening, and huge on Eid salary day. A fixed threshold either fires on every normal peak (and is soon ignored) or is set so high that it misses a quiet failure.

Anomaly detection applies machine learning to the metric's history and builds a band of expected values that follows its hourly, daily and weekly pattern. Facts to know:

• The model takes up to two weeks of data to train well (the band appears within about three hours for a new model).
• The threshold of an anomaly detection alarm is the width of the band in standard deviations (2 is common): a bigger number, a wider band, fewer alarms.
• The alarm fires when the metric goes above the band, below the band, or either. “Below” catches failures that look like silence: a broken login page makes requests drop.
• You can exclude unusual periods (an outage, a holiday) from training, and it works on metric math expressions too.

#Composite alarms

A composite alarm watches the states of other alarms, not metrics. Its rule combines them with AND, OR and NOT, using the functions ALARM("name"), OK("name"), INSUFFICIENT_DATA("name") and the constants TRUE and FALSE:

ALARM("cpu-high") AND ALARM("disk-reads-high")
(ALARM("cpu-high") OR ALARM("disk-reads-high")) AND NOT ALARM("deploy-running")

• A rule can name up to 100 child alarms: metric alarms and other composite alarms.
• Why: to reduce alarm noise (alarm fatigue). Children usually have no actions; only the composite notifies, so one real problem produces one page.
• Action suppression: name an actions suppressor alarm (for example one that is in ALARM during deployments or maintenance). While it is in ALARM, the composite's actions are held back; a wait period and an extension period give it time to switch on and settle afterwards.
• What it can do: notify SNS, invoke Lambda, create a Systems Manager OpsItem or incident, start an investigation. It cannot take EC2 actions (stop, terminate, reboot, recover) or Auto Scaling actions: put those on a metric alarm.
• A composite alarm is in INSUFFICIENT_DATA only right after it is created.

#What an alarm can do: actions

Each state (OK, ALARM, INSUFFICIENT_DATA) has its own list of actions, up to five each, so an alarm can page on ALARM and send a “resolved” message on OK. Every state change is also published as an event to EventBridge (lesson 19.5), where a rule can start Step Functions, call an API or route to another account.

A CloudWatch alarm can also back a Route 53 health check, which is how private resources take part in DNS failover (lesson 8.4).

#Alarms on log data

An alarm cannot read log text. The chain from the notes, which the exam loves, is:

Log group → metric filter → CloudWatch metric → CloudWatch alarm → SNS

The metric filter (lesson 19.2) turns every matching line (say PAYMENT_FAILED) into a count; the alarm watches that count like any other metric. Remember that a metric filter only counts events that arrive after it is created, so an alarm on it never sees older failures. For an errors-only metric like this, treating missing data as notBreaching is usually right.

#Testing an alarm with set-alarm-state

How do you prove that the pager works without breaking production? Force the state:

aws cloudwatch set-alarm-state --alarm-name wallet-p99-high \
  --state-value ALARM --state-reason "testing the on-call SMS"

The alarm moves to ALARM and runs its ALARM actions exactly as in a real breach; at the next evaluation it returns to the state its real data supports. Use it to test SNS subscriptions, Lambda permissions and runbooks before the big day.

#EC2 instance recovery

#The three status checks

EC2 runs automated status checks every minute and publishes them as CloudWatch metrics (value 0 = passed, 1 = failed):

StatusCheckFailed (without a suffix) is 1 if any check fails. Status checks test the machine, not your application: a crashed web server with a healthy OS passes both checks (that is what load balancer health checks are for).

#The recover action

Create an alarm on StatusCheckFailed_System (maximum ≥ 1 for, say, 2 of 2 minutes) with the EC2 recover action, usually plus an SNS notification. When the host fails, EC2 migrates the instance to healthy hardware. To the operating system it looks like an unplanned reboot.

A recovered instance is identical to the original except for what was in volatile places:

Requirements and limits (they change, so check the current list): supported instance types only (most current general purpose, compute and memory optimised families, including metal sizes), shared or Dedicated Instance tenancy, and the instance must be running. Not supported: instances in an Auto Scaling group, on Dedicated Hosts (they have their own host recovery), with an Elastic Fabric Adapter, or during a scheduled maintenance event. With instance store volumes attached at launch only a few older families are supported, and their data is lost anyway. Recovery can also fail when there is no spare capacity or during a wider AWS event; then stop and start the instance yourself (a stop and start keeps the Elastic IP but gives a new public IPv4 address).

Recovery does not help an instance status check failure: moving a broken operating system to new hardware leaves it broken. For that, use a second alarm on StatusCheckFailed_Instance with the reboot action.

#Simplified automatic recovery

Since 2022, simplified automatic recovery is on by default for supported instances: when the system status check fails, EC2 tries the same migration with no alarm at all, at no extra cost, and posts an event to the AWS Health Dashboard. The CloudWatch recover action still has advantages: faster, minute-level response, your own SNS notification, and support for metal sizes and some instance store types. You can turn simplified recovery off per instance.

On the exam, the answer to “recover automatically from underlying hardware failure and keep the same IP addresses” is still a CloudWatch alarm on StatusCheckFailed_System with the EC2 recover action. For a fleet behind a load balancer, the better design is an Auto Scaling group, which replaces unhealthy instances (new ID, new IP) instead of recovering them.

#Synthetic monitoring: find faults before customers do

Metrics and logs describe what real traffic did. At 3 am there may be no real traffic, so nothing describes a broken payment page until the first customer meets it. Synthetic monitoring creates the traffic itself.

#CloudWatch Synthetics canaries

A canary is a configurable script that runs on a schedule and follows the same routes and actions as a customer: open the home page, log in, add to cart, pay. The name comes from the canaries coal miners carried as an early warning.

• Languages and runtimes: Node.js with Puppeteer or Playwright, Python with Selenium (headless Chrome, and Firefox on some runtimes), and Java for non-browser checks. Each canary runs as a Lambda function in your account, using a Synthetics layer.
• Schedule: run once, or on a rate or cron schedule, as often as once per minute, around the clock (the notes' “20/7” is a typo for 24/7).
• Blueprints: heartbeat (load a URL, keep a screenshot and an HTTP archive file), API canary (REST calls with headers and body, integrates with API Gateway), broken link checker (follow the links on a page and report 404s and timeouts), visual monitoring (compare screenshots with a baseline, fail above a difference threshold), canary recorder (a Chrome extension that records your clicks into a script), GUI workflow builder (click, type, verify text), and multi checks (up to 10 HTTP, DNS, SSL and TCP checks from a JSON configuration).
• Results: screenshots, HTTP archive files and logs go to an S3 bucket; metrics such as SuccessPercent and Duration go to CloudWatch (namespace CloudWatchSynthetics, dimension CanaryName, and StepName for steps), so alarms on canary metrics page the team.
• Private endpoints: a canary can run inside a VPC to test internal pages; it then needs a NAT gateway or VPC endpoints to reach S3 and CloudWatch.
• It also checks latency of each step and can detect unexpected content changes (for example injected scripts on a page).

#Network Synthetic Monitor

Network Synthetic Monitor (it launched as CloudWatch Network Monitor; the notes use the old name) measures the network path between your AWS workloads and your on-premises network over Direct Connect or Site-to-Site VPN (lessons 22.4 and 22.5). It is a different feature from Synthetics canaries, although both are “synthetic”.

• You choose a source (a subnet of your VPC) and destinations (on-premises IP addresses, protocol, port, packet size). AWS creates and manages the probes: no agent in the VPC and none on premises.
• Probes use ICMP (echo request and reply) or TCP (SYN and SYN-ACK to a port). IPv4 and IPv6 destinations are supported, in separate monitors (the slide's “IPv4/IPv4” means IPv4/IPv6).
• It publishes round-trip time and packet loss to CloudWatch every 30 or 60 seconds (the aggregation interval), so you can graph them, set static or anomaly detection alarms, and see degradation within minutes.
• The network health indicator metric (0 or 100) tells you whether the degradation is inside the AWS-controlled part of the path (to the Direct Connect location, or inter-Region paths). 0 means look at your side or your provider; 100 means AWS.
• It does not fail over traffic; routing (BGP, a backup VPN) does that. Each probe is charged.

#What goes wrong

• Silent alarm on a dead metric: the metric stops, the alarm goes to INSUFFICIENT_DATA, and its only action is on ALARM. Treat missing data as breaching for heartbeats, or add an INSUFFICIENT_DATA action.
• Alarm flapping on an errors-only metric: throttles or errors appear only when something happens, so the alarm jumps to INSUFFICIENT_DATA between events. Treat missing data as notBreaching.
• Period shorter than the metric's interval: a 1-minute alarm on EC2 basic monitoring (5-minute data). Use a 5-minute period or detailed monitoring.
• Too sensitive: 1 out of 1 on a noisy metric pages people for every blip; they learn to ignore it. Use M out of N, a longer period, a percentile, or anomaly detection.
• Too slow: a 5-minute period with 5 out of 5 takes 25 minutes to fire. Match N × period to how quickly you must react.
• Alarm storm: one root cause turns 30 alarms red. Use composite alarms and suppress actions during deployments.
• SNS subscription never confirmed: email subscriptions must be confirmed, so the alarm fires but nobody receives it. Test with set-alarm-state.
• Lambda action fails: the function has no resource-based policy that allows CloudWatch alarms to invoke it.
• Recover action not offered or not working: the alarm is not on a per-instance EC2 metric, the instance type is unsupported, the instance is in an Auto Scaling group, or capacity is short. The ASG's own health checks are the answer for fleets.
• Recover expected to fix a hung OS: recovery reacts only to the system status check; use a reboot action on the instance status check.
• Metric filter created after the incident: it counts only new log events, so the alarm shows nothing for the errors you are looking at.
• Canary fails for the wrong reason: in a private subnet without NAT or VPC endpoints, or with an expired test account password; the alarm fires while the site is fine.
• Composite rule never true: a child alarm was renamed or deleted, or two composites reference each other in a cycle, which stops their evaluation (change one rule to FALSE to break it).

#Alarms in system design

• Alarm on what users feel first: p99 latency, error rate (metric math), canary success, queue age. Resource alarms (CPU, memory) support them but should rarely page on their own.
• Pair every alarm with an action and a runbook: the alarm description can hold a link to the runbook; an OpsItem or a Lambda function can start the first steps automatically.
• Self-healing: alarm → Auto Scaling policy (capacity), alarm → EC2 reboot or recover (single instances), alarm → Lambda or EventBridge → Systems Manager Automation (restart a service, clear a disk).
• One page per real problem: many metric alarms without actions, composite alarms per service with actions, suppression during deployments.
• Hybrid and outside-in: canaries for the customer journey and public APIs, Network Synthetic Monitor for Direct Connect and VPN paths, Route 53 health checks for DNS failover.
• Central view: with cross-account observability (lesson 19.1) a monitoring account holds the alarms for all workload accounts.

#Try it yourself

These commands use the AWS CLI v2 in ap-southeast-1. Alarms, canaries and probes have small monthly charges; delete them afterwards.

# 1. An SNS topic for the on-call team (confirm the email subscription in your inbox)
aws sns create-topic --name oncall-alerts
aws sns subscribe --topic-arn arn:aws:sns:ap-southeast-1:111122223333:oncall-alerts \
  --protocol email --notification-endpoint oncall@example.com

# 2. A static threshold alarm: p99 latency of an ALB above 1 s for 3 of 5 minutes
aws cloudwatch put-metric-alarm --alarm-name wallet-p99-high \
  --namespace AWS/ApplicationELB --metric-name TargetResponseTime \
  --dimensions Name=LoadBalancer,Value=app/wallet-alb/50dc6c495c0c9188 \
  --extended-statistic p99 --period 60 --evaluation-periods 5 --datapoints-to-alarm 3 \
  --threshold 1 --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:ap-southeast-1:111122223333:oncall-alerts \
  --ok-actions arn:aws:sns:ap-southeast-1:111122223333:oncall-alerts

# 3. Recover an instance when the host fails, and reboot it when the OS hangs
aws cloudwatch put-metric-alarm --alarm-name results-recover \
  --namespace AWS/EC2 --metric-name StatusCheckFailed_System \
  --dimensions Name=InstanceId,Value=i-0a1b2c3d4e5f60718 \
  --statistic Maximum --period 60 --evaluation-periods 2 \
  --threshold 1 --comparison-operator GreaterThanOrEqualToThreshold \
  --alarm-actions arn:aws:automate:ap-southeast-1:ec2:recover \
                  arn:aws:sns:ap-southeast-1:111122223333:oncall-alerts
aws cloudwatch put-metric-alarm --alarm-name results-reboot \
  --namespace AWS/EC2 --metric-name StatusCheckFailed_Instance \
  --dimensions Name=InstanceId,Value=i-0a1b2c3d4e5f60718 \
  --statistic Maximum --period 60 --evaluation-periods 3 \
  --threshold 1 --comparison-operator GreaterThanOrEqualToThreshold \
  --alarm-actions arn:aws:automate:ap-southeast-1:ec2:reboot

# 4. A composite alarm that stays quiet during deployments
aws cloudwatch put-composite-alarm --alarm-name orders-really-broken \
  --alarm-rule 'ALARM("orders-cpu-high") AND ALARM("orders-disk-reads-high")' \
  --actions-suppressor deploy-running \
  --actions-suppressor-wait-period 120 --actions-suppressor-extension-period 300 \
  --alarm-actions arn:aws:sns:ap-southeast-1:111122223333:oncall-alerts

# 5. Test the notification path, then read what happened
aws cloudwatch set-alarm-state --alarm-name wallet-p99-high \
  --state-value ALARM --state-reason "testing the on-call SMS"
aws cloudwatch describe-alarm-history --alarm-name wallet-p99-high \
  --history-item-type StateUpdate --max-items 2 \
  --query 'AlarmHistoryItems[].[Timestamp,HistorySummary]' --output text
# 2026-11-01T04:12:31.204Z  Alarm updated from ALARM to OK
# 2026-11-01T04:11:58.871Z  Alarm updated from OK to ALARM

# 6. Let an alarm invoke Lambda directly (the function needs this permission)
aws lambda add-permission --function-name post-to-chat --statement-id cw-alarms \
  --action lambda:InvokeFunction --principal lambda.alarms.cloudwatch.amazonaws.com \
  --source-arn arn:aws:cloudwatch:ap-southeast-1:111122223333:alarm:wallet-p99-high

# 7. A heartbeat canary every minute (script zipped in S3, results to another bucket)
aws synthetics create-canary --name shop-checkout \
  --code S3Bucket=canary-code-example,S3Key=checkout.zip,Handler=checkout.handler \
  --artifact-s3-location s3://canary-results-example/shop-checkout/ \
  --execution-role-arn arn:aws:iam::111122223333:role/CanaryRole \
  --schedule Expression="rate(1 minute)" \
  --runtime-version syn-nodejs-puppeteer-9.1
aws synthetics start-canary --name shop-checkout

Also try in the console: open Alarms → Create alarm, pick a metric, choose Anomaly detection and watch the preview band; then select two alarms and choose Create composite alarm. In Application Signals → Synthetics canaries, create a canary from the Heartbeat monitoring blueprint and look at the screenshots of its first runs.

#Summary

Key takeaways

• A CloudWatch alarm watches one metric, a metric math expression or an anomaly detection band. States: OK, ALARM, INSUFFICIENT_DATA. Actions run on state changes (only Auto Scaling actions repeat).
• Period (60 s or more; 10, 20 or 30 s for high-resolution metrics), evaluation periods (N) and datapoints to alarm (M): M out of N tolerates noise and catches flapping. The period must not be shorter than the metric's interval (EC2 basic monitoring = 5 minutes).
• Missing data: missing (default, all missing → INSUFFICIENT_DATA), notBreaching (errors-only metrics), breaching (heartbeats), ignore (keep state). Used only when there are fewer than N real datapoints.
• Static threshold for known limits, anomaly detection for metrics with daily and weekly patterns, composite alarms (AND, OR, NOT over other alarms, up to 100 children, actions suppressor) to cut alarm noise. Composites cannot take EC2 or Auto Scaling actions.
• Actions: SNS (then anything), Lambda, EC2 stop / terminate / reboot / recover, Auto Scaling policies, Systems Manager Ops Center items and incidents, investigations. Every state change is also an EventBridge event. Test with set-alarm-state.
• Log → metric filter → metric → alarm → SNS for alerts on log text.
• EC2 recovery: alarm on StatusCheckFailed_System + recover → same instance ID, private / public / Elastic IPs, metadata, placement group, EBS; RAM and instance store lost. Instance status check failures → reboot. Not for instances in an Auto Scaling group. Simplified automatic recovery is on by default, but the exam answer is the alarm.
• Synthetics canaries: scripts (Node.js, Python, Java) on a schedule, as often as every minute, 24/7, testing UI flows, APIs, links and visual changes, results in S3, metrics for alarms.
• Network Synthetic Monitor: agentless probes from a VPC subnet to on-premises IPs over Direct Connect or VPN, round-trip time and packet loss, network health indicator. Internet Monitor watches the internet path to end users.

Self-check: CloudWatch alarms30 questions