CloudWatch Logs, Agents and Subscriptions

By the end of this lesson you will be able to

• Explain how CloudWatch Logs stores logs: log events, log streams and log groups, and which settings live on the group (retention, KMS key, log class, access).
• Name the sources that send logs to CloudWatch Logs, and explain why nothing leaves an EC2 instance until an agent sends it.
• Turn log lines into a metric with a metric filter, and build the exam’s chain logs → metric filter → metric → alarm → SNS.
• Query stored logs with Logs Insights (and know its three query languages and its price), and watch new logs as they arrive with Live Tail.
• Choose between an export task to S3 (batch, up to 12 hours behind) and subscription filters (real time) to Kinesis Data Streams, Data Firehose, Lambda or OpenSearch.
• Collect the logs of many accounts and Regions in one log-archive account with destinations, access policies and IAM roles.
• Choose the unified CloudWatch agent over the old CloudWatch Logs agent, and give it credentials on EC2 and on premises.

#Why CloudWatch Logs: every log in one place

Lesson 19.1 was about numbers: CPU, latency, request counts. Numbers tell you that something is wrong. To learn why, you read the logs: the stack trace, the failed SQL statement, the IP that hammered the login page. This lesson is about where those logs live on AWS.

#The problem

A Dhaka news portal runs twelve web servers on EC2, a few Lambda functions that resize photos, an API Gateway for its mobile app, and an old printing system on a server in a Tejgaon data centre. Every one of them writes logs, each to its own disk. On election night the home page slows down. To find the cause, an engineer would have to log into twelve servers, read twelve files, and the servers that Auto Scaling already terminated took their logs with them.

Amazon CloudWatch Logs solves this: it collects the logs of applications, operating systems and AWS services (VPC Flow Logs, Route 53, CloudTrail and many more) in one regional service, keeps them as long as you choose, and lets you search them, turn them into metrics, alert on them and send them elsewhere.

#Events, streams and groups

CloudWatch Logs has a three-level structure:

A group can hold any number of streams, and the streams are usually created automatically by whatever writes to them (the agent, Lambda, ECS).

#Settings on the group

Retention. By default a log group keeps its events forever (Never Expire). You can set a retention period of 1 day up to 10 years, in fixed steps: 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1,096, 1,827, 2,192, 2,557, 2,922, 3,288 or 3,653 days. Events older than the retention are marked for deletion (they stop being billed at once; the delete itself can take up to 72 hours) and cannot be recovered. A common design: short retention in CloudWatch Logs for searching, long-term copies in S3 with lifecycle rules (lesson 10.5).

Encryption. Log data is always encrypted at rest, by default with 256-bit AES under keys the service manages. For more control you can associate a customer managed KMS key (symmetric only) with a log group, when you create it or later with associate-kms-key. The key policy must allow the CloudWatch Logs service principal of that Region. New events are then encrypted with your key; if the key is disabled the logs cannot be read, and if it is deleted they are lost for good. KMS itself is lesson 21.2.

Access. IAM policies control who can read or write a group (by its ARN, or with tags). Streams have no permissions of their own; access is decided at the group level. Deletion protection can stop a group from being deleted by mistake.

Log class. Chosen when the group is created and cannot be changed later:

#Where logs come from

• Your code: the AWS SDK and CLI (PutLogEvents), in batches of up to 1 MB. Events older than 14 days or more than 2 hours in the future are rejected.
• Agents on servers: the unified CloudWatch agent (and the deprecated CloudWatch Logs agent), on EC2 or on premises. More below.
• Elastic Beanstalk: streams instance logs of the application when log streaming is turned on.
• ECS and EKS: containers write to stdout and the awslogs log driver (or Fluent Bit) ships each container’s output as a stream (lesson 14.2).
• Lambda: every function writes to /aws/lambda/<function-name> automatically, if its execution role allows it (lesson 15.1).
• VPC Flow Logs: accepted and rejected connections of a VPC, subnet or network interface (lesson 22.3).
• API Gateway: execution logs and access logs per stage.
• CloudTrail: a trail can also deliver its API events to a log group, so metric filters can alarm on them (lesson 19.6).
• Route 53: DNS query logs. Public hosted zone query logs go to a log group in us-east-1; Resolver query logs record the DNS queries made from inside your VPCs.
• And many more: RDS and Aurora database logs, AWS WAF, Step Functions, Systems Manager Run Command output…

#Metric filters: from log lines to numbers

Alarms watch metrics, not text. To be told when a certain line appears in the logs, you first turn matching lines into a number. That is a metric filter:

• It belongs to a log group and has a filter pattern: a word or phrase (ERROR, "payment failed"), several terms (?ERROR ?Timeout for either one), a space-delimited pattern for fields of a line ([ip, user, ts, request, status=5*, size]), or a JSON pattern ({ $.level = "error" }, { $.latency > 2000 }).
• For each new event that matches, it publishes a value (usually 1, or a field such as the latency) to a custom metric in the namespace and name you choose, optionally with dimensions taken from fields of the event (up to 3).
• A default value (usually 0) is published for periods with no match; without it, quiet periods have no data at all.
• It applies only to events ingested after it is created: it never scans old data. Up to 100 metric filters per log group, Standard class only.

Then the usual chain completes the story: an alarm on that metric (lesson 19.3), with an action that publishes to an SNS topic (email, SMS, a Lambda function, a chat channel; lesson 13.4).

#Logs Insights: ask questions of stored logs

CloudWatch Logs Insights is an interactive query engine over the events already stored in your log groups. You pick a time range and one or more log groups (also in other accounts, through CloudWatch cross-account observability) and write a query. Typical questions: count the ERROR lines per 5 minutes, find every request of one IP address, list the ten slowest endpoints, see the top error messages after a deployment.

• Three query languages. The purpose-built Logs Insights query language (commands joined by pipes: fields, filter, stats, sort, limit, parse, dedup, pattern…), and since late 2024 also OpenSearch PPL and OpenSearch SQL (SQL even allows joins across log groups).
• Discovered fields. It finds fields automatically in logs of AWS services (Lambda, VPC Flow Logs, CloudTrail, Route 53) and in any JSON log, so you filter on status or duration by name; parse extracts fields from plain text.
• Not real time. It queries what is already stored, from seconds ago to years ago. A query runs at most 60 minutes, and results are kept 7 days. Up to 100 queries can run at the same time per account.
• Price. You pay per GB of data scanned (about 0.005 USD per GB in us-east-1), however few rows match. Narrow time ranges and fewer log groups cost less; field indexes can skip events.
• Reuse. Save queries (with parameters), add them to CloudWatch dashboards, run them on a schedule, or generate them from a question in plain English.

#Live Tail: watch logs as they arrive

Live Tail is the opposite of Logs Insights: instead of searching the past, it streams new events from up to 10 log groups to your console or CLI within about a second of ingestion, with filter terms and highlighting. Developers open it during a deployment or an incident to see, right now, what the new version prints.

Sessions are interactive: up to 15 at a time per account, and an API session lasts up to 3 hours.

#Getting logs out: export or subscription

CloudWatch Logs is a good place to search recent logs, but not always the place to keep them for years, nor the tool the security team already uses. There are two ways out.

#Export to S3: batch

An export task (CreateExportTask) copies the events of one log group and one time range to an S3 bucket as gzip files, under a prefix you choose.

• It is not real time: log data can take up to 12 hours to become available for export.
• Only one export task can be active at a time per account and Region; tasks that run longer than 24 hours time out.
• The bucket policy must allow the CloudWatch Logs service to write. Buckets encrypted with SSE-S3 or SSE-KMS and buckets with Object Lock are supported (DSSE-KMS is not). The bucket may be in another account.
• AWS recommends not to use regular exports for continuous archiving; use subscriptions.

Good for: a one-off copy of last month for an auditor, or a migration.

#Subscription filters: real time

A subscription filter sends the events of a log group that match a pattern (an empty pattern means all), as they are ingested, to:

• Kinesis Data Streams: for custom real-time consumers and replay (lesson 13.5).
• Amazon Data Firehose (formerly Kinesis Data Firehose): near real-time delivery to S3, Redshift, OpenSearch or a partner such as Splunk or Datadog, with no code (lesson 13.6).
• Lambda: custom processing of each batch of events.
• OpenSearch Service: the console wires this through a Lambda function it creates, for full-text search and dashboards (lesson 17.3).

Details that matter:

• Events arrive base64-encoded and gzip-compressed, usually well within three minutes, at least once (a rare duplicate is possible).
• Up to 5 subscription filters per log group, each with its own pattern and destination.
• CloudWatch Logs needs permission to write to the destination: an IAM role for Kinesis and Firehose, a resource-based permission on the Lambda function.
• If the destination throttles or fails with a retryable error, delivery is retried for up to 24 hours; after that the events are dropped from the subscription. Watch the delivery throttling metric.
• Account-level subscription filter (since 2024): one per account and Region that covers all log groups (or all except those you exclude) and sends to Kinesis Data Streams, Firehose or Lambda. Exclude the log groups of the delivery pipeline itself, or the subscription can feed itself in an endless, expensive loop.
• Standard class only.

#Collecting logs from many accounts and Regions

Large organisations put every application in its own AWS account (lesson 20.1). The security team then wants all their logs in a separate log-archive account: one place to search, and a place application teams cannot tamper with.

The classic way is a cross-account subscription:

1. Build the receiver in the archive account

   Create a Kinesis data stream (or a Firehose stream), and an IAM role that CloudWatch Logs can assume, with permission to put records into it.

2. Create a destination

   A CloudWatch Logs destination (PutDestination) is a named endpoint that wraps the stream and the role. Attach a destination access policy that names the source accounts allowed to subscribe, or the whole organization with a condition on the organization ID.

3. Subscribe from each source account

   In each source account, create a subscription filter on the log group with the destination’s ARN as its target. The source account never gets access to the stream itself.

4. Deliver and store

   Kinesis Data Streams → Firehose → S3 in the archive account (and any real-time consumers on the stream). Protect the bucket with Object Lock and a strict bucket policy (lesson 10.9).

Two rules: the log group and the destination must be in the same Region, so you create one destination per source Region; the resource behind a destination (the stream) may be in another Region. Destinations can also wrap a Firehose stream directly, and Lambda can be a cross-account target too.

#Agents: the unified CloudWatch agent

EC2 instances and on-premises servers need software inside the operating system to send log files, and to report what only the operating system knows. There have been two agents:

The metrics the unified agent adds (in the CWAgent namespace by default):

• CPU: time spent active, idle, guest, system, user, steal…
• Disk: free, used, used percent, total; and disk I/O: reads, writes, bytes, IOPS.
• Memory (RAM): free, used, used percent, cached, inactive, total.
• Netstat: number of TCP connections by state and UDP sockets; network packets and bytes.
• Processes: total, running, sleeping, blocked, idle, dead (and per-process CPU and memory with procstat).
• Swap: free, used, used percent.

Compare with the built-in EC2 metrics, which come from the hypervisor, outside the guest operating system: CPU utilisation, network in and out, disk operations for instance store, and status checks. Memory and disk space used are not among them (lesson 19.1).

Permissions.

• On EC2: attach an instance role with the AWS managed policy CloudWatchAgentServerPolicy (it allows writing metrics and logs and reading AmazonCloudWatch-* parameters). No keys on disk. Lesson 2.4 covers roles.
• On premises there is no instance role. The agent reads credentials from a named profile (AmazonCloudWatchAgent by default) in a shared credentials file: the access keys of an IAM user with the same policy, or better, short-lived credentials refreshed by IAM Roles Anywhere. The server also needs a network path to AWS (internet, VPN or Direct Connect).

Central configuration. Create the config once (the agent’s wizard writes it), store it in Parameter Store as AmazonCloudWatch-linux, and every server runs amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:AmazonCloudWatch-linux -s (-m onPremise on premises). Systems Manager can install and configure the agent on the whole fleet (lesson 25.3). The agent creates the log groups and streams it needs.

#More features worth knowing

#Which feature for which need?

#What goes wrong

• No logs from an instance: no agent, agent stopped, no instance role or wrong policy, wrong Region in the config, wrong file path. The agent’s own log on the instance says which.
• Logs gone: retention deleted them; or a log group was deleted. Keep an archive in S3 for anything you might need later, and use deletion protection.
• Huge bill: retention left at Never Expire, verbose debug logs ingested at full price, or Logs Insights queries over months of data. Set retention, log less, use the cheaper class for rarely read logs, narrow queries.
• Metric shows nothing for past errors: metric filters count only new events. Query the past with Logs Insights.
• Alarm in insufficient data at night: the metric filter has no default value; publish 0 when nothing matches.
• Unreadable logs: the customer managed KMS key was disabled, its policy changed, or it was scheduled for deletion.
• Subscription stops delivering: the destination throttles (Lambda concurrency, too few Kinesis shards) and events are dropped after 24 hours of retries; or the role lost its permission. Watch the delivery metrics.
• Endless loop and bill: an account-level subscription that also subscribes the log group of its own Lambda or Firehose. Exclude those log groups.
• Cross-account subscription denied: the source account is not in the destination access policy, or the destination is in another Region.
• Export missing the last hours: data can take up to 12 hours to become exportable; and a second export waits behind the first.
• Cannot add a subscription or metric filter: the group is in the Infrequent Access (Essentials) class, which cannot be changed.
• Secrets in logs: passwords or card numbers printed by debug code. Fix the code, add a data protection policy, and restrict who can read the group.

#CloudWatch Logs in system design

• Observability baseline: every service writes structured JSON logs to a log group per component, with retention set by infrastructure code, metric filters for the errors that matter, and alarms to SNS. Dashboards combine the resulting metrics with Logs Insights widgets.
• Security and compliance archive: subscription filters (or centralization rules) send every account’s logs to a log-archive account; Firehose writes to an Object Lock bucket with lifecycle rules; Athena queries the archive when an investigation needs years of data (lesson 17.1).
• Real-time detection: subscription to Kinesis Data Streams feeding a SIEM or a Lambda detector; VPC Flow Logs and CloudTrail in CloudWatch Logs with metric filters for rejected traffic or root logins (lessons 19.6 and 21.7).
• Hybrid estate: the unified agent on EC2 and on-premises servers, one config in Parameter Store, the same dashboards for both.
• Cost control: the cheaper class for debug logs, short retention plus S3 archive, log sampling for very chatty services.

Where else to look: performance graphs are metrics (19.1), deeper container and Lambda views are the Insights family (lesson 19.4), who called which API is CloudTrail (19.6), and the full comparison of CloudWatch, CloudTrail and Config is in lesson 19.7.

#Try it yourself

These commands use the AWS CLI v2 in ap-southeast-1. Ingestion, storage, queries and Live Tail cost a little; delete everything at the end.

# 1. A log group with 30-day retention, and a test event
aws logs create-log-group --log-group-name /wallet/api
aws logs put-retention-policy --log-group-name /wallet/api --retention-in-days 30
aws logs create-log-stream --log-group-name /wallet/api --log-stream-name test-1
aws logs put-log-events --log-group-name /wallet/api --log-stream-name test-1 \
  --log-events "timestamp=$(date +%s000),message={\"level\":\"error\",\"path\":\"/send-money\",\"status\":502}"
# { "nextSequenceToken": "49654...", "rejectedLogEventsInfo": null }

# 2. A metric filter that counts error lines, with 0 when nothing matches
aws logs put-metric-filter --log-group-name /wallet/api --filter-name payment-errors \
  --filter-pattern '{ $.level = "error" }' \
  --metric-transformations metricName=PaymentErrors,metricNamespace=Wallet,metricValue=1,defaultValue=0
aws logs test-metric-filter --filter-pattern '{ $.level = "error" }' \
  --log-event-messages '{"level":"error","status":502}' '{"level":"info","status":200}'
# { "matches": [ { "eventNumber": 1, "eventMessage": "{\"level\":\"error\",\"status\":502}", ... } ] }

# 3. A Logs Insights query over the last hour
QID=$(aws logs start-query --log-group-name /wallet/api \
  --start-time $(date -d '-1 hour' +%s) --end-time $(date +%s) \
  --query-string 'filter level = "error" | stats count(*) as errors by bin(5m)' \
  --query queryId --output text)
aws logs get-query-results --query-id "$QID"
# { "status": "Complete", "statistics": { "recordsMatched": 1.0, "recordsScanned": 1.0, "bytesScanned": 78.0 },
#   "results": [ [ { "field": "bin(5m)", "value": "2026-11-01 09:15:00.000" }, { "field": "errors", "value": "1" } ] ] }

# 4. Live Tail for 5 minutes, highlighting errors (Ctrl+C to stop)
aws logs start-live-tail \
  --log-group-identifiers arn:aws:logs:ap-southeast-1:111122223333:log-group:/wallet/api \
  --log-event-filter-pattern '{ $.level = "error" }'

# 5. Export yesterday to S3 (bucket policy must allow logs.ap-southeast-1.amazonaws.com)
aws logs create-export-task --task-name wallet-2026-10-31 --log-group-name /wallet/api \
  --from 1761868800000 --to 1761955200000 \
  --destination wallet-logs-archive-example --destination-prefix exports/2026-10-31
# { "taskId": "c2a1f6d8-5b9e-4e1a-9a4f-0e7d2b3c4f51" }
aws logs describe-export-tasks --task-id c2a1f6d8-5b9e-4e1a-9a4f-0e7d2b3c4f51 --query 'exportTasks[0].status'
# { "code": "COMPLETED", "message": "Completed successfully" }

# 6. Real-time subscription to an existing Firehose stream (role lets logs put records)
aws logs put-subscription-filter --log-group-name /wallet/api --filter-name to-firehose \
  --filter-pattern '' \
  --destination-arn arn:aws:firehose:ap-southeast-1:111122223333:deliverystream/wallet-logs \
  --role-arn arn:aws:iam::111122223333:role/CWLtoFirehoseRole

# 7. Cross-account: in the archive account 444455556666, a destination and its access policy
aws logs put-destination --destination-name org-logs \
  --target-arn arn:aws:kinesis:ap-southeast-1:444455556666:stream/org-logs \
  --role-arn arn:aws:iam::444455556666:role/CWLtoKinesisRole
aws logs put-destination-policy --destination-name org-logs --access-policy '{
  "Version": "2012-10-17",
  "Statement": [{ "Effect": "Allow", "Principal": { "AWS": ["111122223333", "777788889999"] },
    "Action": "logs:PutSubscriptionFilter",
    "Resource": "arn:aws:logs:ap-southeast-1:444455556666:destination:org-logs" }] }'
#    ... and in each source account:
aws logs put-subscription-filter --log-group-name /wallet/api --filter-name to-archive --filter-pattern '' \
  --destination-arn arn:aws:logs:ap-southeast-1:444455556666:destination:org-logs

# 8. Unified agent on EC2 (instance role with CloudWatchAgentServerPolicy), config from Parameter Store
sudo dnf install -y amazon-cloudwatch-agent
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config -m ec2 -c ssm:AmazonCloudWatch-linux -s
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status
# { "status": "running", "starttime": "2026-11-01T09:20:11+00:00", "configstatus": "configured", "version": "1.300..." }

# 9. Clean up
aws logs delete-subscription-filter --log-group-name /wallet/api --filter-name to-firehose
aws logs delete-log-group --log-group-name /wallet/api

Also try in the console: open Logs Insights, choose a Lambda log group and run the sample query for the most expensive invocations; then open Live Tail on the same group and invoke the function.

#Summary

Key takeaways

• Log event (message + timestamp) → log stream (one source, in order) → log group (settings: retention, KMS key, access, log class). Retention 1 day to 10 years, default Never Expire.
• Logs are always encrypted at rest; optionally associate a customer managed KMS key with the group (lose the key, lose the logs).
• Sources: SDK, agents, Beanstalk, ECS, Lambda, VPC Flow Logs, API Gateway, CloudTrail, Route 53 and more. Nothing leaves an EC2 instance without an agent and IAM permissions.
• Metric filter turns matching lines into a metric, only for new events; exam chain: logs → metric filter → metric → alarm → SNS.
• Logs Insights: query stored logs (own language, OpenSearch PPL, OpenSearch SQL), pay per GB scanned, not real time. Live Tail: watch new logs right now.
• Export task (CreateExportTask) → S3, batch, up to 12 hours behind, one active task. Subscription filter → Kinesis Data Streams, Data Firehose, Lambda, OpenSearch, real time, 5 per log group, plus account-level filters.
• Multi-account / multi-Region: subscription filter → destination with access policy (same Region) → Kinesis Data Streams → Firehose → S3 in a log-archive account. Newer: centralization rules, cross-account observability.
• Unified CloudWatch agent (logs + memory, disk, swap, netstat, processes; Linux, Windows, macOS; EC2 and on premises; config in SSM Parameter Store) replaces the deprecated CloudWatch Logs agent. EC2 needs a role with CloudWatchAgentServerPolicy; on premises needs credentials.

Self-check: CloudWatch Logs31 questions