AWS CloudTrail

By the end of this lesson you will be able to

• Explain what AWS CloudTrail records (API calls and sign-ins: who did what, when, from where, and whether it worked) and what it does not record.
• Use Event history: on by default, the last 90 days of management events per Region, free, read-only.
• Create a trail: multi-Region or single-Region, an organization trail from the management account or a delegated administrator, delivery to S3 (about 5 minutes on average) and optionally to CloudWatch Logs, SNS notifications, SSE-KMS.
• Tell the event types apart: management (read and write, on by default, first copy free), data (S3 objects, Lambda Invoke, DynamoDB items…, off by default, paid), Insights (unusual API call rate and error rate) and network activity events (VPC endpoints).
• Keep audit logs for years and prove they were not changed: S3 Lifecycle to Glacier, Athena, log file integrity validation, bucket protection and a separate log archive account.
• React to any recorded API call within seconds with EventBridge, including root sign-ins, and know CloudTrail Lake’s place (and its 2026 status).

#Why CloudTrail: the account’s CCTV camera

Chapter 19 is about four services that watch an AWS account, and the exam picks one of them from the words in a scenario. Performance numbers and application logs are CloudWatch (lessons 19.1 to 19.4); reacting to something that happened is EventBridge (19.5); what a resource looked like and whether it is compliant is Config (19.7). This lesson is the fourth: who called which API.

#The problem

Friday night, 11 pm, at a Dhaka online shop. The orders table in DynamoDB is gone. The application is down, the backups will bring the data back, but the managers ask the obvious question: who deleted it, from where, and was it a mistake or an attack? Nobody remembers doing it. The table cannot tell you; it no longer exists.

Every action on AWS, whether a click in the console, a CLI command, an SDK call from an application or a call that one AWS service makes for you, ends up as a request to an AWS API. If every one of those requests is written down, the question has an answer. That written record is AWS CloudTrail.

#What CloudTrail is

AWS CloudTrail records the API calls and sign-in activity in your AWS account, for governance, compliance, operational and risk auditing:

• It is enabled by default in every account. You do not have to switch anything on to get the last 90 days of management activity.
• It records calls made through the console, the CLI, the SDKs and other AWS services (when CloudFormation creates an instance for you, the record shows the call was invoked by CloudFormation).
• Each record says who made the call (the IAM user, role session, root user or AWS service), what was called (the service and the action, with parameters), when (UTC time), from where (source IP address, Region, user agent) and whether it worked (an error code such as AccessDenied if it failed).
• Records can be kept long term in S3, searched in CloudWatch Logs, queried with Athena, and turned into automatic reactions with EventBridge.

#What an event record looks like

Every call becomes one JSON event. Here is the one from the Friday night incident, shortened:

{
  "eventVersion": "1.10",
  "userIdentity": {
    "type": "AssumedRole",
    "arn": "arn:aws:sts::111122223333:assumed-role/ops-admin/rafi",
    "accountId": "111122223333",
    "sessionContext": { "attributes": { "mfaAuthenticated": "true" } }
  },
  "eventTime": "2026-10-30T17:02:44Z",
  "eventSource": "dynamodb.amazonaws.com",
  "eventName": "DeleteTable",
  "awsRegion": "ap-southeast-1",
  "sourceIPAddress": "198.51.100.7",
  "userAgent": "aws-cli/2.17.0 Python/3.12 Linux",
  "requestParameters": { "tableName": "orders" },
  "responseElements": { "tableDescription": { "tableStatus": "DELETING" } },
  "readOnly": false,
  "eventType": "AwsApiCall",
  "managementEvent": true,
  "eventCategory": "Management",
  "recipientAccountId": "111122223333"
}

The answer to Friday’s question is in the first lines: Rafi, through the ops-admin role with MFA, from the CLI at 198.51.100.7, at 11:02 pm Dhaka time. It was a script pointed at the wrong environment. CloudTrail did not prevent it; it told the team exactly what happened within minutes, and the fix (an SCP and a deletion-protection flag on production tables) followed.

#Event history: on by default

You do not need to create anything to start using CloudTrail. Event history is there from the day the account is created:

• It holds the last 90 days of management events (read and write) in each Region. Look in the Region where the call happened; global services such as IAM appear in us-east-1.
• It is free, searchable in the console (filter by one attribute at a time: event name, user name, resource type or name, event source, access key, read-only, event ID) and with aws cloudtrail lookup-events, and downloadable as CSV or JSON.
• It is immutable: nobody can edit or delete events in it, and it is not affected by changes to your trails.
• It does not contain data events, network activity events or anything older than 90 days, and you cannot make it longer.

Event history is the first place to look for “what happened in the last few weeks?”. It is not an audit archive: the window keeps moving, and an auditor’s question about last year will find nothing.

#Trails: keeping events beyond 90 days

A trail is a configuration that delivers CloudTrail events to an S3 bucket you choose, and optionally to CloudWatch Logs and (through the default event bus) EventBridge. With a trail, you keep events as long as you like, and you can add data, Insights and network activity events.

#Multi-Region or single-Region

• A multi-Region trail records events in all enabled Regions of the account and delivers them to one bucket. It also records global service events (IAM, STS’s global endpoint, CloudFront, Route 53…), which are logged as us-east-1 events. It is the best practice, and every trail created in the console is multi-Region.
• A single-Region trail records only the Region where it was created. It can only be created with the CLI or the API, where it is the default unless you pass --is-multi-region-trail. A single-Region trail outside us-east-1 misses the IAM changes, which is a classic audit gap.
• An account can have up to 5 trails per Region. The bucket can be in any Region.

(The slide says a trail applies to all Regions “by default”: true for the console, the opposite for the CLI and API.)

#Delivery and the details

• CloudTrail writes gzip-compressed JSON log files to the bucket about every 5 minutes, and an event typically arrives within about 5 minutes on average of the API call. This is not real time and not guaranteed.
• Files go under a predictable prefix: AWSLogs/<account-id>/CloudTrail/<region>/<yyyy>/<mm>/<dd>/. An organization trail adds the organization ID.
• The bucket policy must allow the CloudTrail service to check the bucket ACL and to write objects, ideally limited to your trail with an aws:SourceArn condition. The console writes this policy for you.
• Encryption: log files are always encrypted at rest. The console proposes SSE-KMS with a customer managed key for new trails; otherwise SSE-S3 is used. With SSE-KMS, readers need kms:Decrypt on the key as well as S3 access (lesson 21.2).
• SNS notifications can announce every new log file delivered, so a log-processing system knows when to read.
• CloudWatch Logs delivery sends the same events to a log group (CloudTrail needs an IAM role to write there). That is how you get a metric filter and an alarm on events, for example on failed sign-ins (lessons 19.2 and 19.3).
• If the bucket becomes unreachable (deleted, policy broken), CloudTrail keeps retrying for 30 days, and those events are still charged. Delete or fix a broken trail.

#Organization trails

In AWS Organizations, an organization trail logs every account in the organization into one place:

• It is created in the management account or in a delegated administrator account for CloudTrail (the notes say “master account”, the old name for the management account).
• It applies automatically to all member accounts, including accounts that join later.
• Member accounts can see the trail but cannot change or delete it, and by default cannot read its log files.
• Best practice is to deliver it to a bucket in a separate log archive account that only the security team controls. Organizations, SCPs and delegated administrators are lesson 20.1.

#Event types: what gets recorded

CloudTrail sorts activity into four kinds of events. Only the first is on by default.

#Management events (control plane)

Management events are operations on the resources of your account: creating, configuring and deleting them, and reading their configuration. Examples from the notes and the slides: RunInstances (EC2), CreateBucket (S3), CreateUser and AttachRolePolicy (IAM, configuring security), CreateSubnet (EC2, routing), CreateTrail (setting up logging). Console sign-ins are recorded as management events too.

• On by default, in Event history and in every new trail.
• Split into write events (may change something: DeleteTable) and read events (do not change anything: DescribeInstances, ListBuckets, GetSecretValue). A trail can log all, read-only or write-only.
• A trail can exclude AWS KMS events (Encrypt, Decrypt, GenerateDataKey are read management events and can be very many) and RDS Data API events, to cut volume.
• Cost: the first copy of management events in each Region is free. A second trail that logs the same management events again costs 2 USD per 100,000 events.

#Data events (data plane)

Data events are operations on or inside a resource, usually at very high volume:

• S3 object-level activity: GetObject, PutObject, DeleteObject (read and write can be logged separately).
• Lambda function execution: the Invoke API.
• DynamoDB item-level activity: GetItem, PutItem, DeleteItem, queries.
• And many more resource types: S3 Express One Zone, SNS publish, EBS direct APIs, and others.

They are off by default, because a busy bucket or function produces millions of events an hour. You turn them on per trail with advanced event selectors, which can narrow logging to one bucket, a prefix, one action (eventName), or exclude a role. Cost: 0.10 USD per 100,000 data events delivered. Data events never appear in Event history.

#Insights events

CloudTrail Insights detects unusual activity: it learns the normal rate of API calls and API errors in your account and Region (the baseline) and writes an Insights event when the current rate moves far away from it. There are two kinds:

• API call rate: the number of write management events per minute.
• API error rate: the number of management calls that fail (read and write) per minute.

Since November 2025 a trail can also run both kinds of Insights on data events. Insights is off by default and charged per events analysed (0.35 USD per 100,000 management events per Insights type). The next section has the details.

#Network activity events (side note, 2024–25)

Network activity events record AWS API calls that pass through your VPC endpoints, for the endpoint owner, including calls the endpoint policy denies (VpceAccessDenied). They show, for example, credentials from outside your organization trying to use your KMS or S3 endpoint. They are off by default, chosen per service with advanced event selectors, and cost 0.10 USD per 100,000 events. They are new and rarely tested; VPC endpoints are lesson 22.2.

Sort some real activities yourself. Watch for the traps: a KMS decrypt, a secret read and a sign-in.

#CloudTrail Insights: spotting unusual activity

Insights is for the question “is something strange happening right now?”, without anyone writing a rule for each possible attack. The slides list the kinds of problems it surfaces: inaccurate resource provisioning (a script launching far too many instances), hitting service limits, bursts of IAM actions, and gaps in periodic maintenance activity (a nightly job that suddenly stops).

How it works:

• Insights analyses the management events (and, if enabled, the data events) of a trail, Region by Region, and builds a baseline of normal call and error rates, which it keeps adjusting.
• When the rate breaks away from the baseline, it writes an Insights event marked Start; when the rate returns to normal, a matching End event follows, with the duration and the top users, user agents and error codes involved.
• Insights events are delivered to a separate folder of the trail’s S3 bucket, shown on the console’s Insights page for 90 days, and sent to EventBridge (so a rule can page someone).
• After you first enable Insights on a trail, it can take up to 36 hours before the first Insights events appear.

#Keeping logs for years: S3, lifecycle, Athena and CloudWatch Logs

Event history keeps 90 days. Regulations often ask for much more: a bank may need years of audit records, a hospital seven. The pattern from the notes and the slides:

1. A multi-Region trail (or organization trail) delivers log files to an S3 bucket. S3 keeps them as long as you want.
2. An S3 Lifecycle rule moves older files to cheaper storage, for example Glacier Flexible Retrieval after 90 days or Glacier Deep Archive for files that will almost never be read, and expires them when the retention period ends (lesson 10.5).
3. To ask questions, Athena runs SQL directly on the log files in S3 (the slide’s “log to S3 and use Athena”), with a ready-made table definition for the CloudTrail format, partitioned by Region and date (lesson 17.1). Files in Glacier Flexible Retrieval or Deep Archive must be restored first.
4. For alerts on patterns of events, the trail also delivers to CloudWatch Logs, where a metric filter counts matching events and an alarm notifies SNS.

#Protecting the audit trail

An audit log that the people being audited can switch off or edit proves nothing. Attackers know this: one of the first things a careful intruder tries is to stop logging or delete the logs. Protection has four layers.

#1. Nobody can switch the camera off

• Use an organization trail: member accounts, even their administrators, cannot stop, change or delete it.
• Add an SCP that denies cloudtrail:StopLogging, cloudtrail:DeleteTrail and cloudtrail:UpdateTrail (and deleting the log bucket) to everyone except a break-glass role (lesson 20.1; the explicit-deny idea is in lesson 2.2).
• Watch for those calls anyway: a StopLogging attempt is itself recorded, and an EventBridge rule can alert on it.

#2. Nobody can delete the footage

• Deliver logs to a bucket in a separate log archive account. Its bucket policy lets CloudTrail write and the security team read; nobody in the workload accounts has any access.
• Turn on S3 Object Lock in compliance mode (no one, not even the root user of the archive account, can delete or overwrite an object before its retention date), or at least versioning with MFA Delete (lesson 10.9).
• Encrypt with SSE-KMS and a key whose policy allows only CloudTrail to encrypt and the auditors to decrypt.

#3. Anyone can prove the footage is untouched: log file integrity validation

With log file integrity validation turned on (the console turns it on for new trails; the CLI needs --enable-log-file-validation), CloudTrail:

• writes a digest file every hour into the same bucket (under a separate digest prefix), listing every log file delivered in that hour with its SHA-256 hash;
• signs each digest file with a private key, using SHA-256 with RSA, and puts the hash and signature of the previous digest into each new one, so digests form an unbroken chain.

aws cloudtrail validate-logs then checks the signatures with CloudTrail’s public key and recomputes the hash of each log file. It reports log files that were modified, deleted or moved, and missing or altered digest files. Validation detects tampering; it does not prevent it, which is why it goes together with layers 1 and 2. The exam phrase: “prove that log files have not been tampered with” or “integrity of CloudTrail logs” → log file integrity validation.

#4. Alert when someone tries

An EventBridge rule (next section) on StopLogging, DeleteTrail, PutEventSelectors that drops data events, or a bucket policy change on the log bucket, sends an alert within seconds.

#Reacting to API calls: CloudTrail and EventBridge

Logs are for looking back. Often you want to act immediately when a certain call happens. CloudTrail sends its events to the default event bus of Amazon EventBridge in the same Region, where a rule with an event pattern can match any of them and invoke targets such as Lambda, SNS, SQS or Step Functions (rules, patterns and targets are lesson 19.5).

The slides’ examples, plus the notes’ root user story:

• A user calls DeleteTable on DynamoDB → CloudTrail logs it → EventBridge rule → SNS alert.
• A user calls AssumeRole on a sensitive IAM role → CloudTrail → EventBridge → SNS.
• A user edits security group inbound rules (AuthorizeSecurityGroupIngress) → CloudTrail → EventBridge → SNS, or a Lambda function that removes a rule open to 0.0.0.0/0 on port 22.
• Someone signs in as the root user → CloudTrail → EventBridge → SNS email to the security team and a Lambda function that opens an incident.

What to know to make this work:

• The event types on the bus: AWS API Call via CloudTrail (API calls), AWS Console Sign In via CloudTrail (sign-ins), AWS Insight via CloudTrail (Insights), plus console actions, service events and network activity events.
• A trail with logging enabled must cover the account (an organization trail counts). Event history alone does not feed EventBridge.
• Rules in the normal enabled state receive write management events, sign-ins, and the data events the trail logs. Read-only management events (Get, List, Describe) reach a rule only if its state is set to accept all CloudTrail management events.
• Events arrive on the bus of the Region where the call was recorded. Root user console sign-ins are recorded in us-east-1, us-east-2 or us-west-2, and IAM and other global services in us-east-1. A rule built only in ap-southeast-1 never sees them. Create the rule in those Regions, or in every Region with forwarding to one central bus.
• This is near real time (seconds to a few minutes), much faster than waiting for log files in S3.

#What goes wrong

• “The event is not there.” Wrong Region in Event history (the call was made elsewhere; IAM is in us-east-1), older than 90 days with no trail, or it was a data event that no trail was logging.
• A single-Region trail created with the CLI misses every other Region and the IAM changes. Use multi-Region trails.
• The root sign-in rule never fires: it was created in the workload Region instead of us-east-1, us-east-2 and us-west-2, or no trail is logging.
• No logs arrive in S3: the bucket policy does not allow CloudTrail, the KMS key policy does not let CloudTrail encrypt, or the bucket was deleted. CloudTrail retries for 30 days and the trail status shows the delivery error.
• A huge data-event bill: data events turned on for all buckets, including a busy static-content bucket. Scope advanced event selectors to the buckets and actions that matter.
• Paying twice for management events: a second trail that logs the same management events (only the first copy is free). An organization trail plus old per-account trails is a common cause.
• Expecting real time from S3 files: delivery averages about 5 minutes and is not guaranteed. For immediate reactions use EventBridge.
• Logs can be deleted: trail bucket in the same account with no Object Lock, no MFA Delete, and broad S3 permissions. Validation will show the damage but cannot undo it.
• Insights says nothing for a day: the first Insights events can take up to 36 hours after enabling; and successful read floods never trigger a call-rate Insight.
• Athena cannot read old logs: they were moved to Glacier Flexible Retrieval or Deep Archive and need a restore first.

#CloudTrail in system design

• Multi-account landing zone: an organization trail (often set up by Control Tower) into a log archive account, Object Lock, validation, SSE-KMS, an SCP that protects CloudTrail, and Athena or a SIEM-style tool reading the central bucket.
• Security automation: CloudTrail events on EventBridge drive automatic responses: revoke public security group rules, re-enable logging, tag or stop resources created outside approved Regions, page on root sign-in.
• Incident investigation: Event history for the last weeks, Athena over the trail bucket for older events, and GuardDuty, which reads CloudTrail management and S3 data events to find threats (lesson 21.7).
• Compliance: CloudTrail answers “who did it”, AWS Config answers “what did the resource look like and was it compliant” (lesson 19.7). Auditors usually want both.
• Least privilege: IAM Access Analyzer generates policies from the actions a role really used, read from CloudTrail (lesson 2.4).

#Try it yourself

These commands use the AWS CLI v2. The trail and data events cost money (small amounts for a test account); delete them at the end.

# 1. Event history: who deleted a DynamoDB table in the last 7 days (free, management events, this Region)
aws cloudtrail lookup-events --region ap-southeast-1 \
  --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteTable \
  --start-time 2026-10-24T00:00:00Z \
  --query 'Events[].[EventTime,Username,EventName]' --output table
# -------------------------------------------------------
# |                    LookupEvents                     |
# +----------------------------+-------+----------------+
# |  2026-10-30T17:02:44+00:00 |  rafi |  DeleteTable   |
# +----------------------------+-------+----------------+

# 2. A multi-Region trail with log file validation and SSE-KMS
#    (the bucket and its CloudTrail bucket policy, and the KMS key policy, must exist first)
aws cloudtrail create-trail --name org-audit \
  --s3-bucket-name audit-logs-example \
  --is-multi-region-trail --enable-log-file-validation \
  --kms-key-id arn:aws:kms:ap-southeast-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab
aws cloudtrail start-logging --name org-audit
aws cloudtrail get-trail-status --name org-audit --query '[IsLogging,LatestDeliveryTime]'
# [ true, "2026-10-31T06:15:09.112000+06:00" ]

# 3. Data events for one bucket only, GetObject only (advanced event selectors)
aws cloudtrail put-event-selectors --trail-name org-audit --advanced-event-selectors '[
  { "Name": "Management events", "FieldSelectors": [ { "Field": "eventCategory", "Equals": ["Management"] } ] },
  { "Name": "Invoice downloads", "FieldSelectors": [
      { "Field": "eventCategory", "Equals": ["Data"] },
      { "Field": "resources.type", "Equals": ["AWS::S3::Object"] },
      { "Field": "eventName", "Equals": ["GetObject"] },
      { "Field": "resources.ARN", "StartsWith": ["arn:aws:s3:::invoices-example/"] } ] } ]'

# 4. Insights on call rate and error rate
aws cloudtrail put-insight-selectors --trail-name org-audit \
  --insight-selectors '[{"InsightType":"ApiCallRateInsight"},{"InsightType":"ApiErrorRateInsight"}]'

# 5. Also deliver to CloudWatch Logs (log group and role created beforehand)
aws cloudtrail update-trail --name org-audit \
  --cloud-watch-logs-log-group-arn arn:aws:logs:ap-southeast-1:111122223333:log-group:cloudtrail-audit:* \
  --cloud-watch-logs-role-arn arn:aws:iam::111122223333:role/CloudTrailToCloudWatchLogs

# 6. Keep 7 years: Deep Archive after 90 days, delete after 2,557 days
aws s3api put-bucket-lifecycle-configuration --bucket audit-logs-example --lifecycle-configuration '{
  "Rules": [ { "ID": "audit-7y", "Status": "Enabled", "Filter": { "Prefix": "AWSLogs/" },
    "Transitions": [ { "Days": 90, "StorageClass": "DEEP_ARCHIVE" } ],
    "Expiration": { "Days": 2557 } } ] }'

# 7. Root sign-in alert: the rule lives in us-east-1 (repeat in us-east-2 and us-west-2)
aws events put-rule --region us-east-1 --name root-signin-alert --event-pattern '{
  "detail-type": ["AWS Console Sign In via CloudTrail"],
  "detail": { "userIdentity": { "type": ["Root"] } } }'
aws events put-targets --region us-east-1 --rule root-signin-alert \
  --targets Id=sns,Arn=arn:aws:sns:us-east-1:111122223333:security-alerts

# 8. Prove the logs were not changed (after a few hours of digests)
aws cloudtrail validate-logs \
  --trail-arn arn:aws:cloudtrail:ap-southeast-1:111122223333:trail/org-audit \
  --start-time 2026-10-31T00:00:00Z
# Results requested for 2026-10-31T00:00:00Z to 2026-10-31T09:12:45Z
# Results found for 2026-10-31T00:12:01Z to 2026-10-31T09:12:01Z:
#
# 9/9 digest files valid
# 107/107 log files valid

# 9. Clean up
aws cloudtrail delete-trail --name org-audit
aws events remove-targets --region us-east-1 --rule root-signin-alert --ids sns
aws events delete-rule --region us-east-1 --name root-signin-alert

Also try in the console: open CloudTrail → Event history, filter by Event name = ConsoleLogin, and open one event to read its JSON. Then edit one downloaded log file in a test bucket, upload it back, and run step 8 again: the edited file is reported as invalid.

#Summary

Key takeaways

• CloudTrail = the account’s CCTV: records API calls and sign-ins from the console, CLI, SDKs and AWS services: who, what, when, from where, did it work. It does not prevent actions, measure performance, or see inside instances and databases.
• Event history: on by default, 90 days of management events, per Region, free, searchable, immutable. Nothing older, no data events.
• Trail: delivers to S3 (about 5 minutes on average), optionally CloudWatch Logs, SNS on delivery, SSE-KMS. Console trails are multi-Region; CLI trails are single-Region unless you ask. Organization trail from the management account or delegated administrator; members cannot change it.
• Management events: control plane, read and write, on by default, first copy free. Data events: S3 objects, Lambda Invoke, DynamoDB items…, off by default, paid, scope with advanced event selectors. Insights: unusual API call rate (write management events) and error rate, to S3, console and EventBridge. Network activity events: calls through your VPC endpoints.
• Long retention: trail to S3, Lifecycle to Glacier / Deep Archive, Athena to query. CloudTrail Lake = SQL over event data stores (up to 10 years), closed to new customers since May 2026.
• Protect it: organization trail, SCP against StopLogging and DeleteTrail, separate log archive account, Object Lock or MFA Delete, SSE-KMS, and log file integrity validation (hourly digests, SHA-256 hashes, SHA-256 with RSA signatures, validate-logs).
• React: CloudTrail → EventBridge rule → Lambda / SNS for any recorded API call (DeleteTable, AuthorizeSecurityGroupIngress, AssumeRole, root sign-in). Needs a logging trail; root sign-ins arrive in us-east-1, us-east-2 or us-west-2.

Self-check: AWS CloudTrail31 questions