CloudWatch Metrics and Metric Streams

By the end of this lesson you will be able to

• Place CloudWatch, CloudTrail, Config and EventBridge in the chapter’s exam theme, and say which question each one answers.
• Read a CloudWatch metric: namespace, metric name, dimensions, timestamp, unit, and the statistics (Average, Sum, Minimum, Maximum, SampleCount, percentiles such as p99) computed over a period.
• Explain EC2 basic monitoring (5 minutes, free) versus detailed monitoring (1 minute, paid), why memory and disk space used are missing, and how to get them.
• Publish custom metrics with PutMetricData, at standard resolution (1 minute) or high resolution (down to 1 second), and know the rules on namespaces, dimensions and timestamps.
• State the retention tiers correctly (under 60 seconds for 3 hours, 1 minute for 15 days, 5 minutes for 63 days, 1 hour for 455 days) and predict what detail is left for data of a given age.
• Use dashboards, metric math, anomaly detection and cross-account observability, and stream metrics out with Metric Streams through Amazon Data Firehose instead of polling GetMetricData.

#Why metrics: is the system healthy right now?

Chapter 19 is about four services that watch your AWS accounts. The exam rarely asks how to configure them; it asks you to pick the right one from the words in a scenario:

The full side-by-side comparison is in lesson 19.7. This lesson starts with the part of CloudWatch everything else is built on: metrics. Logs are lesson 19.2, alarms are 19.3, and the Insights features are 19.4.

#The problem

A bKash-style mobile wallet pays out salaries on the day before Eid. At 20:00 transfers jump from 300 to 6,000 a minute. The on-call engineer needs answers in seconds, not after reading a thousand log lines: Is CPU on the API servers high? Are transfers getting slower? How slow are the slowest ones? Is it worse than last Eid?

Every one of those questions is a number that changes over time. Collect such numbers at a regular rhythm, keep them, graph them, compare them and raise an alarm when they cross a line, and you have monitoring. In AWS that is CloudWatch Metrics.

#What a metric is

A metric is a time-ordered series of data points about one thing: the CPU utilisation of one EC2 instance, the size of one S3 bucket, the number of messages in one SQS queue. Think of it as a variable to watch, and each data point as its value at a moment.

• AWS services publish metrics for you, at no charge, for almost every resource: EC2, EBS, RDS, ELB, Lambda, DynamoDB, SQS, S3 and the rest. You do nothing to turn them on.
• You can publish your own (custom metrics): orders per minute, payment latency, active users, memory used.
• Metrics are regional: a metric lives in the Region where it was created.
• A metric cannot be deleted. It simply expires when no new data has arrived for 15 months.

#Anatomy of a metric

#Namespace, name and dimensions: the identity

Namespace + metric name + dimensions together are the metric. Every unique combination of dimension values is a separate metric, even with the same name. Latency with Service=transfer, Env=prod and Latency with Service=transfer, Env=staging are two metrics. You can only read back the combinations you actually published; CloudWatch does not add up custom metrics across dimensions for you (for some AWS namespaces such as AWS/EC2 it does offer pre-aggregated views, for example across all instances or per Auto Scaling group). To gather many metrics at once, use a metric math SEARCH expression.

#Timestamp, value and unit: the data point

• Timestamp: when the value was measured, ideally in UTC. If you leave it out, CloudWatch uses the time it received the point. A custom data point may be stamped up to two weeks in the past and up to two hours in the future; anything outside that window is rejected.
• Value: a number (no text, no not-a-number or infinity values).
• Unit: Percent, Count, Bytes, Seconds, Milliseconds, Count/Second and so on. Optional (it defaults to None), but points with different units are aggregated separately, so keep it consistent.

#Statistics and periods: how you read it

You rarely read raw points. You ask for a statistic over a period:

• Period: the length of each time bucket, in seconds: 1, 5, 10, 30 or any multiple of 60. The default is 60. Sub-minute periods only make sense for high-resolution metrics.
• Statistics: SampleCount (how many points), Sum, Average (Sum / SampleCount), Minimum, Maximum, and percentiles such as p50, p90, p99 (p99 = the value that 99% of points are below). CloudWatch also offers trimmed mean and related statistics, which ignore the extreme tails.

Why percentiles matter: the wallet’s average transfer latency at 20:05 is 180 ms and looks calm, but p99 is 2.4 seconds: one customer in a hundred waits more than two seconds, and on Eid night that is thousands of people. Averages hide the slow tail; percentiles show it. Percentiles need raw values (or the Values and Counts arrays); if you publish only a pre-summarised statistic set (Sum, SampleCount, Minimum, Maximum), percentiles are not available for it.

#Where metrics come from

#AWS service metrics, and EC2 basic versus detailed monitoring

Most AWS services publish their metrics every minute at no charge. EC2 is the classic exception:

Two consequences the exam likes. An Auto Scaling group that should react within a minute or two needs detailed monitoring, because a 5-minute metric makes every decision late (lesson 6.5). And an alarm with a 60-second period on a basic-monitoring metric has nothing to evaluate for four minutes out of five; use a period of at least 300 seconds, or turn on detailed monitoring (alarms are lesson 19.3).

#What EC2 cannot see: memory and disk space

EC2’s free metrics are measured by the hypervisor, from outside your operating system:

• Included: CPUUtilization; NetworkIn, NetworkOut, NetworkPacketsIn, NetworkPacketsOut; DiskReadOps, DiskWriteOps, DiskReadBytes, DiskWriteBytes (for instance store volumes); status checks (StatusCheckFailed, StatusCheckFailed_System, StatusCheckFailed_Instance, StatusCheckFailed_AttachedEBS); CPU credit metrics for burstable T instances; and EBS volume metrics for Nitro-based instances.
• Not included: memory (RAM) used, swap, disk space used or free on a file system, number of processes, application-level numbers. These live inside the guest OS, where the hypervisor cannot look.

To get them, run something inside the instance that publishes them as custom metrics: the unified CloudWatch agent (memory, disk, swap, netstat, per-process metrics, plus log files, on Linux and Windows, on EC2 and on premises; lesson 19.2 covers it), or your own script that calls PutMetricData. The notes call it a “custom agent”; the product is the CloudWatch agent.

#Custom metrics with PutMetricData

Your own numbers are published with the PutMetricData API (from the CLI, any SDK, the CloudWatch agent, or other tools):

• The first data point creates the metric. There is nothing to create in advance. A new metric can take up to 15 minutes to show up in list-metrics.
• Namespace: any name you like, up to 255 characters, but do not start it with AWS/ (that prefix belongs to AWS services).
• Standard resolution (the default): 1-minute granularity. High resolution: set StorageResolution to 1 and CloudWatch keeps 1-second granularity, readable at 1, 5, 10 or 30 seconds or any multiple of 60. Alarms on a high-resolution metric can use 10- or 30-second periods, at a higher alarm price.
• Many points per call: up to 1,000 metrics per request, and up to 150 values per metric with the Values and Counts arrays; a statistic set sends a pre-aggregated Sum, SampleCount, Minimum and Maximum instead (cheaper, but no percentiles).
• Timestamps: up to two weeks old or two hours ahead. Points stamped 3 to 24 hours in the past can take up to 2 hours to show in queries, and points a day old or more at least 48 hours.
• Cost: each custom metric (each unique combination of namespace, name and dimensions) is billed per month (0.30 USD for each of the first 10,000 in us-east-1, cheaper beyond), and PutMetricData calls are billed as API requests. The free tier includes 10 custom or detailed-monitoring metrics and 1 million API requests a month.

Because every dimension value is a separate billed metric, never put a high-cardinality value such as a user ID, an order ID or a request ID in a dimension: one million users means one million metrics. Put such details in logs instead.

#Resolution and retention: what survives as data ages

CloudWatch keeps fine detail only while data is young. As points age, they are rolled up into coarser points (Sum, SampleCount, Minimum and Maximum are kept, so averages stay correct), and after 15 months they expire:

Read it as a ladder: 1-minute data stays at 1 minute for 15 days; after that you can still read it, but only in 5-minute steps; after 63 days only hourly; after 455 days it is gone. And the ladder only goes down: data never collected at 1 minute (EC2 basic monitoring) never had a 1-minute step to begin with.

Two more details:

• Metrics with no new data for two weeks disappear from the console and list-metrics, but their data can still be read with get-metric-data or get-metric-statistics until it expires.
• If you need metrics for years (a bank’s audit, a 7-year retention rule), CloudWatch is not the archive. Copy them out with a metric stream to S3 (below), or a scheduled job.

Try it: pick how a metric was published and how old the data is, and see what is left.

#Looking at metrics: graphs, math, anomaly bands and dashboards

#Metric math, search and Metrics Insights

• Metric math combines metrics into a new time series with expressions: error rate = 100 * errors / requests, the total of several instances, the rate of change. Alarms can watch a metric math expression (lesson 19.3).
• SEARCH expressions gather every metric that matches, for example all CPUUtilization metrics of one Auto Scaling group, so a graph picks up new instances automatically.
• Metrics Insights is a SQL-like query language over metrics (SELECT AVG(CPUUtilization) FROM SCHEMA("AWS/EC2", InstanceId) GROUP BY InstanceId ORDER BY AVG() DESC LIMIT 10): the top 10 busiest instances in one query.

#Anomaly detection

Anomaly detection trains a machine-learning model on a metric’s history (with daily and weekly patterns) and draws an expected band. Traffic that is normal at 20:00 on a weekday can be abnormal at 04:00. An alarm can fire when the metric leaves the band instead of crossing a fixed number; that alarm type is lesson 19.3.

#Dashboards

A CloudWatch dashboard is a page of graphs, numbers, alarm states, log query results and text:

• Dashboards are global: one dashboard can show widgets from several Regions, and with cross-account access (below) from several accounts.
• Automatic dashboards for each service are free. For your own: 3 dashboards with up to 50 metrics each are free per month; after that each dashboard costs about 3 USD a month.
• Dashboards can be shared with people who have no AWS account (by a public link, by email address, or through single sign-on), refreshed automatically, and set to a time range and time zone.

#Cross-account observability

Large companies run one AWS account per team or application (lesson 20.1). To see them together, CloudWatch cross-account observability links them:

• One monitoring account creates a sink (one per Region) with a policy saying who may link. Each source account creates a link to it and chooses what to share: metrics (all or some namespaces), log groups, X-Ray traces, Application Signals, Application Insights applications and Internet Monitor monitors.
• The monitoring account then graphs, searches, alarms on and builds dashboards over source-account data without copying it, at no extra cost for metrics and logs.
• A monitoring account can have up to 100,000 source accounts; a source account can share with up to 5 monitoring accounts. With AWS Organizations, new accounts are linked automatically. The API behind it is the Observability Access Manager (OAM).
• It works per Region. The older cross-account cross-Region dashboard feature (sharing through IAM roles) still exists for viewing dashboards across Regions.

#Metric Streams: pushing metrics out

Many teams already use a third-party monitoring tool (Datadog, Dynatrace, New Relic, Splunk, Sumo Logic, Elastic) or want all metrics in a data lake. The old way was polling: a collector calls GetMetricData every few minutes for thousands of metrics. That is slow (the data is always one polling interval behind), costs money per metric requested (GetMetricData is not in the free API allowance), and runs into API throttling.

CloudWatch Metric Streams turn it around: CloudWatch pushes metric updates continuously, with near-real-time delivery and low latency, to a destination:

• The stream goes to an Amazon Data Firehose stream (formerly Kinesis Data Firehose; lesson 13.6), and Firehose delivers to S3, to Redshift (through S3), to OpenSearch, to any HTTP endpoint, or to a partner. Quick setups create everything for S3 and for partners (you give only the endpoint URL and API key).
• Output format: JSON, OpenTelemetry 1.0.0 or OpenTelemetry 0.7.0. Firehose can convert further, for example to Parquet for Athena (lesson 17.1).
• Filters: stream everything, or up to 1,000 filters that include or exclude namespaces or individual metrics. One stream uses either include filters or exclude filters, not both. New metrics that match are added automatically.
• Statistics: Minimum, Maximum, SampleCount and Sum are always included; extra statistics such as p99 can be added at extra cost.
• In a monitoring account, a stream can include the linked source accounts’ metrics.
• Price: per 1,000 metric updates (0.003 USD in us-east-1), plus Firehose. No limit on the number of streams.

#What goes wrong

• No memory or disk metric: it is not a default EC2 metric. Install the CloudWatch agent with an instance role that allows it to publish (lesson 19.2).
• A spike is invisible: a 5-minute basic metric or a 5-minute average swallowed it. Use detailed monitoring, high resolution, Maximum or p99.
• An alarm sits in INSUFFICIENT_DATA: a 60-second period on a 5-minute metric, or custom data stamped with the wrong time zone (CloudWatch alarms use UTC).
• Empty graph for a custom metric: wrong Region, a dimension combination that was never published, a different unit, or a brand-new metric that has not appeared yet (up to 15 minutes).
• PutMetricData rejected: timestamp more than two weeks old or two hours ahead, a namespace starting with AWS/, more than 30 dimensions, or a value that is not a number.
• The bill jumps: a high-cardinality dimension (user ID, request ID) created millions of metrics; or a tool polls GetMetricData for everything every minute.
• Old detail is gone: 1-minute data older than 15 days is only available at 5 minutes; nothing survives past 455 days unless you streamed it to S3.
• A partner tool misses metrics: the stream’s include filter lacks the namespace, or the Firehose delivery is failing (check its backup bucket and the stream’s own metrics).
• A dashboard in the monitoring account lacks a metric: the source account did not share that namespace, the link is in another Region, or the metric has not sent a new point since the link was made.

#Metrics in system design

Common patterns:

• Scale on what matters to users: publish a custom metric such as queue backlog per instance or active sessions, and use it in a target tracking policy instead of CPU (lesson 6.5 and lesson 13.1).
• Watch service level objectives with percentiles: p99 latency and error rate per API, not averages.
• Business numbers next to technical ones: orders per minute and payments succeeded on the same dashboard as CPU and 5xx, so a drop in sales is seen even when every server looks healthy.
• A central monitoring account for many accounts, through cross-account observability.
• Long-term analytics: metric streams to S3, queried with Athena, combined with cost data.
• Metrics feed alarms (19.3), which feed Auto Scaling, SNS, Lambda and EC2 actions. Metrics are the start of nearly every automated reaction in AWS.

#Try it yourself

These commands use the AWS CLI v2. Custom metrics and detailed monitoring cost a few cents; delete the stream at the end.

# 1. List the metrics of one EC2 instance (AWS/EC2 namespace, InstanceId dimension)
aws cloudwatch list-metrics --namespace AWS/EC2 \
  --dimensions Name=InstanceId,Value=i-0a1b2c3d4e5f60789 --query 'Metrics[].MetricName'
# [ "CPUUtilization", "NetworkIn", "NetworkOut", "NetworkPacketsIn", "StatusCheckFailed", ... ]
# (no memory metric: that needs the CloudWatch agent)

# 2. Turn on detailed monitoring (1-minute metrics, paid)
aws ec2 monitor-instances --instance-ids i-0a1b2c3d4e5f60789
# { "InstanceMonitorings": [ { "InstanceId": "i-0a1b2c3d4e5f60789", "Monitoring": { "State": "pending" } } ] }

# 3. Average and maximum CPU over the last hour in 5-minute periods
aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0a1b2c3d4e5f60789 \
  --start-time 2026-11-01T09:00:00Z --end-time 2026-11-01T10:00:00Z \
  --period 300 --statistics Average Maximum
# { "Label": "CPUUtilization", "Datapoints": [
#     { "Timestamp": "2026-11-01T09:05:00Z", "Average": 41.2, "Maximum": 63.8, "Unit": "Percent" }, ... ] }

# 4. Publish a high-resolution custom metric (storage resolution 1 second)
aws cloudwatch put-metric-data --namespace Wallet/Payments --metric-data \
  'MetricName=Latency,Dimensions=[{Name=Service,Value=transfer},{Name=Env,Value=prod}],Value=184,Unit=Milliseconds,StorageResolution=1'
# (no output on success; the first point creates the metric)

# 5. Read it back with a percentile, using get-metric-data
aws cloudwatch get-metric-data --start-time 2026-11-01T09:50:00Z --end-time 2026-11-01T10:00:00Z \
  --metric-data-queries '[{"Id":"p99","MetricStat":{"Metric":{"Namespace":"Wallet/Payments","MetricName":"Latency","Dimensions":[{"Name":"Service","Value":"transfer"},{"Name":"Env","Value":"prod"}]},"Period":60,"Stat":"p99"}}]'
# { "MetricDataResults": [ { "Id": "p99", "Label": "Latency", "Timestamps": [ "2026-11-01T09:59:00Z", ... ],
#     "Values": [ 2400.0, ... ], "StatusCode": "Complete" } ] }

# 6. Stream EC2, ALB and wallet metrics to an existing Firehose stream, OpenTelemetry 1.0 format
aws cloudwatch put-metric-stream --name wallet-to-partner \
  --firehose-arn arn:aws:firehose:ap-southeast-1:111122223333:deliverystream/metrics-to-partner \
  --role-arn arn:aws:iam::111122223333:role/MetricStreamToFirehose \
  --output-format opentelemetry1.0 \
  --include-filters Namespace=AWS/EC2 Namespace=AWS/ApplicationELB Namespace=Wallet/Payments
# { "Arn": "arn:aws:cloudwatch:ap-southeast-1:111122223333:metric-stream/wallet-to-partner" }
aws cloudwatch get-metric-stream --name wallet-to-partner --query '[State,OutputFormat]'
# [ "running", "opentelemetry1.0" ]

# 7. Clean up
aws cloudwatch delete-metric-stream --name wallet-to-partner
aws ec2 unmonitor-instances --instance-ids i-0a1b2c3d4e5f60789

Also try in the console: open All metrics, graph CPUUtilization for one instance with the Average and then the Maximum statistic, change the period from 5 minutes to 1 hour, and add the graph to a new dashboard. Then open Metrics, Streams and look at the quick partner setup list.

#Summary

Key takeaways

• Chapter theme: performance and logs → CloudWatch, react to events → EventBridge, who called which API → CloudTrail, what a resource looked like and is it compliant → Config.
• A metric = namespace + metric name + dimensions (up to 30); every new dimension combination is a separate metric. Data points have a timestamp, a value and a unit. Metrics are regional and expire after 15 months without data.
• Read metrics as statistics over a period: SampleCount, Sum, Average, Minimum, Maximum, percentiles (p99). Averages hide the slow tail.
• EC2 basic monitoring = 5 minutes, free; detailed = 1 minute, paid. Most other services publish every minute.
• EC2 default metrics (CPU, network, instance-store disk ops, status checks) do not include memory, swap or disk space used → unified CloudWatch agent or PutMetricData.
• Custom metrics with PutMetricData: standard resolution 1 minute, high resolution 1 second (StorageResolution=1); timestamps up to 2 weeks back, 2 hours ahead; avoid high-cardinality dimensions. EMF creates metrics from log lines.
• Retention: < 60 s for 3 hours, 1 min for 15 days, 5 min for 63 days, 1 hour for 455 days (15 months). Older data is rolled up, never made finer.
• Metric math, search expressions, Metrics Insights, anomaly detection bands; dashboards are global (multi-Region, multi-account), 3 free.
• Cross-account observability (OAM): a monitoring account with a sink, source accounts with links; metrics and logs shared at no extra cost, per Region.
• Metric Streams: push, near real time, through Amazon Data Firehose to S3, partners (Datadog, Dynatrace, Elastic, New Relic, Splunk, Sumo Logic) or any HTTP endpoint; JSON or OpenTelemetry; include or exclude filters. Better than polling GetMetricData.

Self-check: CloudWatch Metrics and Metric Streams30 questions