Container, Lambda, Contributor and Application Insights

By the end of this lesson you will be able to

• Explain why the basic CloudWatch metrics of a service are not enough to troubleshoot containers, Lambda functions, noisy clients or enterprise applications, and what the four Insights features add.
• Turn on Container Insights for ECS, EKS and Kubernetes on EC2 (on EC2 and Fargate), and read pod, task and container metrics, including enhanced observability.
• Add Lambda Insights to a function as a layer, and use it to measure cold starts, memory, CPU time and network.
• Write a Contributor Insights rule that ranks the top-N contributors in a log group, alarm on it, and use the built-in rules for DynamoDB.
• Onboard an application into Application Insights and follow a detected problem to EventBridge and Systems Manager OpsCenter.
• Pick the right tool from a scenario’s words, and tell the four Insights apart from Logs Insights, CloudTrail Insights, Application Signals and X-Ray.

#The Insights family: a closer look at one part of the system

#Why the basic metrics are not enough

Lesson 19.1 showed the metrics every AWS service publishes on its own, and 19.2 the logs you can collect. They answer “is something wrong?”. They often cannot answer “where exactly?”:

• An EKS node is at 58% memory, but one pod inside it is being killed every few minutes because it hits its own memory limit. The node metric hides it.
• A Lambda function’s standard metrics are invocations, duration, errors and throttles. They do not show memory used, CPU time or how long the cold start took.
• The web servers are at 95% CPU. CPU graphs say they are busy, not who is making them busy.
• A .NET application on IIS returns errors while SQL Server logs deadlocks. Two teams get two unrelated alarms and nobody sees that it is one problem.

The CloudWatch Insights family is four features that go deeper into one part of the system each. They do not replace metrics, logs and alarms: they are built on them. Each collects extra data (or reads data you already send), builds the metrics and dashboards for its question, and lets you alarm on the result.

#Four features, four questions

#Container Insights

#What it collects

Container Insights collects, aggregates and summarises metrics and logs from containers: CPU, memory, disk and network, plus diagnostic information such as container restart failures. It is available for:

• Amazon ECS, on EC2 and on Fargate;
• Amazon EKS, on EC2 nodes and on Fargate;
• Kubernetes platforms on EC2 that you run yourself.

The data is sent as performance log events in the embedded metric format (structured JSON, lesson 19.1) into a log group that Container Insights creates for you, for example /aws/ecs/containerinsights/<cluster>/performance or /aws/containerinsights/<cluster>/performance. From those events CloudWatch creates aggregated metrics at cluster, node, service, task and pod level (container level with enhanced observability) in the ContainerInsights namespace (ECS/ContainerInsights for ECS), and fills the automatic dashboards. CloudWatch does not turn every possible field into a metric, to keep costs down; the rest stays in the performance events, where you can query it with Logs Insights.

Typical metrics: pod_cpu_utilization, pod_memory_utilization, pod_number_of_container_restarts, node_cpu_utilization, cluster_failed_node_count on Kubernetes; CpuUtilized, MemoryUtilized, RunningTaskCount on ECS. They behave like any other metric: graph them, alarm on them (lesson 19.3), scale on them. Lesson 14.3 showed one dependency: an ECS service that scales on queue backlog per task needs RunningTaskCount, which exists only when Container Insights is on.

For logs, on EKS and Kubernetes the add-on runs Fluent Bit, which ships what containers write to stdout and stderr into log groups such as /aws/containerinsights/<cluster>/application, plus host and data-plane logs. Container Insights can encrypt what it collects with a KMS key you set on the log group (symmetric keys only).

#Turning it on

ECS: a cluster setting, no agent. The ECS agent (on EC2) and Fargate already report task data. You turn Container Insights on per cluster, or as an account default for new clusters:

• enabled: the original Container Insights (cluster, service and task metrics);
• enhanced: Container Insights with enhanced observability (added for ECS in late 2024): more dimensions, container-level metrics and curated dashboards to drill down from cluster to container. AWS recommends it over the original.

EKS and Kubernetes on EC2: a containerised CloudWatch agent. On Kubernetes, Container Insights uses a containerised version of the CloudWatch agent to discover every running container. On EKS you install the Amazon CloudWatch Observability add-on (amazon-cloudwatch-observability), which runs the CloudWatch agent and Fluent Bit as DaemonSets, one copy on each node. The agent needs permission to send data: attach the CloudWatchAgentServerPolicy managed policy to the node role, or give the agent its own role through EKS Pod Identity (or IAM roles for service accounts). On a self-managed Kubernetes cluster on EC2 you deploy the same agent yourself.

EKS on Fargate: no DaemonSet. Fargate gives each pod its own isolated environment and has no node you manage, so a DaemonSet cannot run there. The supported path is an AWS Distro for OpenTelemetry (ADOT) collector, which scrapes the kubelet and sends a smaller set of pod metrics (CPU, memory, network). Enhanced observability is not available on Fargate.

#Enhanced observability and the OpenTelemetry pipeline

Enhanced observability for EKS (2023) adds detailed metrics down to the container, control-plane metrics, curated health dashboards and accelerated hardware metrics (GPUs and Neuron chips), and it is charged per observation instead of per custom metric stored and per GB of logs. The original Container Insights is billed as custom metrics plus log ingestion.

#Lambda Insights

#What the standard metrics miss

Every Lambda function publishes Invocations, Duration, Errors, Throttles and concurrency metrics for free (lesson 15.1). The REPORT line in its log also prints the memory used and, on a cold start, the init duration, but only line by line, never as a metric you can graph across thousands of invocations.

Lambda Insights is a monitoring and troubleshooting solution for serverless applications on Lambda. It collects, aggregates and summarises:

• system-level metrics: CPU time (cpu_total_time), memory (memory_utilization as a percentage of the configured memory, used_memory_max, total_memory), network (rx_bytes, tx_bytes, total_network) and disk (tmp_used, tmp_free for /tmp);
• diagnostic information: cold starts (init_duration, the time spent in the init phase) and Lambda worker shutdowns.

The aggregated metrics live in the LambdaInsights namespace with the dimensions function_name (and version); a few more values (threads, file descriptors, the agent’s own memory) stay in the log events for Logs Insights queries. CloudWatch builds a multi-function overview and a single-function view with these graphs.

#How it works

Lambda Insights is delivered as a Lambda extension, packaged as a Lambda layer. When you add the layer (LambdaInsightsExtension, published by AWS per Region), the extension starts with the execution environment, measures each invocation, and emits one performance log event per invocation in the embedded metric format into the log group /aws/lambda-insights. CloudWatch extracts the metrics from those events.

To turn it on:

1. Give the function’s execution role the CloudWatchLambdaInsightsExecutionRolePolicy managed policy (permission to write to the log group).
2. Add the layer: one click in the Lambda console (“Enhanced monitoring”), aws lambda update-function-configuration --layers …, or a property in SAM / CloudFormation.
3. For a function in a private subnet with no NAT gateway, add an interface VPC endpoint for CloudWatch Logs, or the extension has nowhere to send its events (lesson 15.3).

Functions packaged as container images cannot use layers: copy the extension into the image instead. Lambda Insights supports runtimes on Amazon Linux 2 and Amazon Linux 2023. You pay for the metrics and logs it creates, and for the extension’s run time in 1 ms increments (it runs inside your function’s billed duration).

#Contributor Insights

#Top-N from your logs

Contributor Insights analyses log data and creates time series of contributor data: the top-N contributors, the total number of unique contributors, and how much each one contributes. It finds the top talkers and shows who or what is affecting performance: bad hosts, the heaviest network users, the URLs that generate the most errors, the customers making the most expensive calls.

It works on any log group in CloudWatch Logs whose events are JSON or Common Log Format (CLF), which covers most AWS-generated logs: VPC Flow Logs, Route 53 resolver query logs, API Gateway access logs, CloudTrail, ALB-style access logs you ship, and your own application logs. You can build a rule from scratch or start from AWS’s sample rules in the console.

#Rules

A rule is a small JSON document:

• Log groups: names (a trailing * matches a prefix) or, in a cross-account monitoring account, log group ARNs in source accounts (lesson 19.1 explains cross-account observability).
• Log format: JSON or CLF. For CLF and other space-separated logs, Fields names positions (field 4 = srcAddr).
• Contribution keys: up to four fields that define a contributor. Two keys (source and destination address) make each pair one contributor.
• Filters: up to four, combined with AND (In, NotIn, StartsWith, GreaterThan, LessThan, EqualTo, NotEqualTo, IsPresent), for example only ACCEPT flow records, or only status codes of 500 and above.
• Aggregate on: Count (number of log events) or Sum of a numeric field named in ValueOf (bytes, duration).

Two behaviours matter in practice and in the exam:

• Rules analyse incoming data in real time. A rule sees the log events that arrive after it is created; it never goes back over older events. For “who did this last Thursday?”, query the stored logs with Logs Insights.
• It reports; it does not act. Contributor Insights does not block anyone. To stop an abusive address, add a deny rule to the network ACL or a WAF rule (lessons 21.6 and 24.3).

You can alarm on a rule with the INSIGHT_RULE_METRIC metric math function, for example on UniqueContributors (a sudden jump suggests a botnet) or MaxContributorValue (one contributor is too big). Pricing has two parts: a monthly charge per rule and a charge per million log events that match it, so do not point a * wildcard at every log group by accident.

#Built-in rules: DynamoDB and PrivateLink

Some services come with built-in rules, where you only switch the feature on:

• Contributor Insights for DynamoDB: enable it on a table or global secondary index, and DynamoDB creates rules for the most accessed keys (by consumed throughput) and the most throttled keys, for the partition key and, if there is one, for partition plus sort key. A hot partition (lesson 15.4) shows up immediately. There are now two modes: accessed and throttled keys (every read and write is an event) and throttled keys only, which produces events only when a request is throttled and is cheap enough to leave on permanently. It adds no load to the table.
• AWS PrivateLink: built-in rules for VPC endpoint services show which endpoints and consumers send the most traffic.

#Application Insights

#What it does

CloudWatch Application Insights sets up monitoring for an application and its underlying AWS resources, and then watches it for problems. It is aimed at enterprise applications on EC2 built from well-known technologies:

• Front end: Microsoft IIS web server.
• Worker and application tiers: .NET Framework and .NET Core, Java (it installs a Prometheus exporter through Systems Manager to read Java metrics).
• Directory and collaboration: Active Directory, SharePoint.
• Databases: SQL Server (on EC2 or RDS, including high-availability setups), MySQL, PostgreSQL, Oracle, DynamoDB, plus several large ERP suites.
• Related AWS resources: EBS, RDS, Elastic Load Balancing, Auto Scaling groups, Lambda, SQS, SNS, S3 buckets, Step Functions, API Gateway REST stages, ECS and EKS.

How it works:

1. You add an application by choosing a resource group (from tags or a CloudFormation stack). One resource group = one application.
2. It scans the resources and recognises components (a SQL Server database, an IIS web tier, an Auto Scaling group of workers).
3. It recommends and configures the right metrics and logs per technology (Windows event logs, IIS logs, the SQL Server error log, counters such as deadlocks per second), installing and configuring the CloudWatch agent through Systems Manager (the agent configuration is stored in Parameter Store).
4. It creates dynamic alarms whose thresholds follow anomaly detection on the previous two weeks of data.
5. It keeps analysing metrics and logs. Each anomaly or log error is an observation; related observations are correlated into a problem, with an automatic dashboard and, for common .NET and SQL Server issues, an insight that points to a likely root cause and steps to fix it.
6. Problems are sent as events to EventBridge (rules can notify SNS or start a Lambda function, lesson 19.5), can notify SNS directly, and create OpsItems in Systems Manager OpsCenter, where you can run an Automation runbook to fix them (lesson 25.3).

AWS describes it as powered by SageMaker and other AWS technologies. The feature itself costs nothing; you pay for the CloudWatch metrics, logs and alarms it sets up (alarms are usually most of the bill) and for the OpsItems.

For a technology it does not know, a component can be set to Custom: you choose the metrics and log files, but you do not get technology-specific insights.

#Beyond the four: traces and APM

The four Insights look at resources and logs. Following one request across many services needs tracing. Three related services, one line of exam knowledge each:

• AWS X-Ray records distributed traces: a request is followed through API Gateway, Lambda, containers and DynamoDB, each hop a segment with its timing, and X-Ray draws a service map with latency and errors per edge. “Which microservice is slow?”, “trace requests end to end” → X-Ray. Instrumentation is moving to OpenTelemetry: the X-Ray SDKs and daemon entered maintenance mode on 25 February 2026 (end of support 25 February 2027); AWS recommends the AWS Distro for OpenTelemetry (ADOT) or the CloudWatch agent to send traces. X-Ray itself, its console and its traces are unaffected.
• CloudWatch Application Signals (2024) is AWS’s APM: it auto-instruments Java, Python, Node.js and .NET applications on EKS, ECS, EC2 and Lambda, shows call volume, availability, latency, faults and errors per service and operation, draws an application map, and lets you define service level objectives (SLOs). It builds on X-Ray traces and works with Synthetics canaries (lesson 19.3) and CloudWatch real user monitoring of web pages.
• Amazon Managed Service for Prometheus stores Prometheus metrics (common in Kubernetes) without running Prometheus servers, and Amazon Managed Grafana gives managed Grafana dashboards over CloudWatch, Prometheus, X-Ray and other sources. “We already use Prometheus and Grafana, keep them but stop running the servers” → these two.

#Choosing the right tool

#What goes wrong

• Container Insights on EKS shows no data: the CloudWatch agent has no permission. Attach CloudWatchAgentServerPolicy to the node role or give the agent a role with EKS Pod Identity; the agent’s own log shows AccessDenied.
• ECS scaling on backlog per task never scales: Container Insights is off for the cluster, so RunningTaskCount does not exist (lesson 14.3).
• Fargate pods on EKS have no Container Insights metrics: the DaemonSet cannot run on Fargate. Use the ADOT collector, and accept fewer metrics and no enhanced observability.
• Node graphs are fine but a service keeps failing: you are looking one level too high. Check pod and container metrics and the restart count; containers are killed at their own limit, not the node’s.
• The Container Insights bill jumped: the original mode bills every metric as a custom metric and every log GB; a big cluster with many pods creates many metrics. Compare with enhanced observability’s per-observation pricing, and set retention on the performance log groups.
• Lambda Insights page stays empty: the layer is missing, the execution role lacks CloudWatchLambdaInsightsExecutionRolePolicy, or the function runs in a private subnet without a CloudWatch Logs endpoint or NAT.
• Lambda Insights does not work on a container-image function: layers are not supported there; add the extension to the image.
• A Contributor Insights rule shows nothing for last week: rules only see events that arrive after they are created. Use Logs Insights for the past.
• A Contributor Insights rule shows nothing at all: wrong log format (JSON vs CLF), wrong field name (matching is case-sensitive), or a filter that matches no event.
• “Contributor Insights found the attacker, why is it still attacking?”: it only reports. Block at the network ACL or WAF.
• DynamoDB Contributor Insights costs too much: accessed-and-throttled mode counts every item read and written (twice with a sort key). Switch to throttled-keys-only mode for permanent use.
• Application Insights found no tiers: the resources are not in the resource group (missing tag), or the technology is not supported; use a Custom component, and Application Signals for request-level views.
• Application Insights raised a problem nobody saw: nothing listens. Add an EventBridge rule or SNS notification, and someone who owns OpsCenter.

#Insights in system design

• Layer your observability. The baseline is service metrics, logs with retention, and alarms (lessons 19.1–19.3). Add the Insights feature that matches each platform: Container Insights on every cluster, Lambda Insights on the functions that matter, Application Insights on the enterprise stack, and a few Contributor Insights rules on the logs where “who?” is a common question.
• Close the loop. Every Insights feature produces metrics or events, so it can feed an alarm (19.3) or an EventBridge rule (19.5) that pages a person, scales a service or starts a runbook.
• Control cost. Insights features create metrics, log events and alarms. Turn them on per cluster or per function, set log retention, use DynamoDB’s throttled-keys mode, and do not point a Contributor Insights rule at *.
• Multi-account. With CloudWatch cross-account observability, a central monitoring account can see source accounts’ Container Insights and Lambda Insights data, run Contributor Insights rules over source log groups, and Application Insights supports it too.
• Security. The data may include keys and IPs: DynamoDB Contributor Insights publishes key values to CloudWatch, so do not enable it on a table whose keys are sensitive data that fine-grained access control is meant to hide.

#Try it yourself

These commands use the AWS CLI v2 in ap-south-1 (Mumbai). Cluster, function, table and account names (111122223333) are placeholders. Every feature here is billed through CloudWatch metrics, logs or alarms, so turn things off when you finish.

# 1. ECS: turn on Container Insights with enhanced observability for one cluster
aws ecs update-cluster-settings --cluster shop-prod \
  --settings name=containerInsights,value=enhanced \
  --query 'cluster.settings'
# [ { "name": "containerInsights", "value": "enhanced" } ]

# 2. EKS: give the node role permission, then install the observability add-on
aws iam attach-role-policy --role-name shop-eks-node-role \
  --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
aws eks create-addon --cluster-name shop-eks \
  --addon-name amazon-cloudwatch-observability \
  --query 'addon.[addonName,status]' --output text
# amazon-cloudwatch-observability  CREATING

# 3. Which pods restart? (metrics appear a few minutes after the agent starts)
aws cloudwatch list-metrics --namespace ContainerInsights \
  --metric-name pod_number_of_container_restarts \
  --dimensions Name=ClusterName,Value=shop-eks \
  --query 'Metrics[].Dimensions[?Name==`PodName`].Value' --output text
# cart  search  checkout

# 4. Lambda Insights: permission + layer (check the current layer version for your Region)
aws iam attach-role-policy --role-name payout-fn-role \
  --policy-arn arn:aws:iam::aws:policy/CloudWatchLambdaInsightsExecutionRolePolicy
aws lambda update-function-configuration --function-name payout \
  --layers arn:aws:lambda:ap-south-1:580247275435:layer:LambdaInsightsExtension:56 \
  --query 'Layers[].Arn'

# 5. Cold starts in the last hour, from the Lambda Insights performance events
aws logs start-query --log-group-name /aws/lambda-insights \
  --start-time $(date -d '-1 hour' +%s) --end-time $(date +%s) \
  --query-string 'filter function_name = "payout" and ispresent(init_duration)
    | stats count(*) as coldStarts, avg(init_duration) as avgInitMs, max(memory_utilization) as maxMemPct'
# { "queryId": "a1b2c3d4-5678-90ab-cdef-111122223333" }
aws logs get-query-results --query-id a1b2c3d4-5678-90ab-cdef-111122223333
# coldStarts 214   avgInitMs 1847.6   maxMemPct 61.3

# 6. Contributor Insights: top source addresses in the VPC Flow Logs (accepted traffic only)
cat > top-talkers.json <<'EOF'
{
  "Schema": { "Name": "CloudWatchLogRule", "Version": 1 },
  "LogGroupNames": [ "/vpc/news-portal-flow-logs" ],
  "LogFormat": "CLF",
  "Fields": { "4": "srcAddr", "13": "action" },
  "Contribution": {
    "Keys": [ "srcAddr" ],
    "Filters": [ { "Match": "action", "In": [ "ACCEPT" ] } ]
  },
  "AggregateOn": "Count"
}
EOF
aws cloudwatch put-insight-rule --rule-name news-top-talkers \
  --rule-state ENABLED --rule-definition file://top-talkers.json

# 7. Read the top 3 contributors of the last 5 minutes
aws cloudwatch get-insight-rule-report --rule-name news-top-talkers \
  --start-time $(date -u -d '-5 min' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --period 300 --max-contributor-count 3 \
  --query 'Contributors[].[Keys[0],ApproximateAggregateValue]' --output text
# 203.0.113.66    92400.0
# 198.51.100.23   4350.0
# 203.0.113.9     4020.0

# 8. Alarm when one address sends more than 20,000 accepted records in 5 minutes
aws cloudwatch put-metric-alarm --alarm-name news-one-ip-dominates \
  --metrics '[{"Id":"m1","Expression":"INSIGHT_RULE_METRIC(\"news-top-talkers\", \"MaxContributorValue\")","Period":300,"ReturnData":true}]' \
  --comparison-operator GreaterThanThreshold --threshold 20000 --evaluation-periods 1 \
  --alarm-actions arn:aws:sns:ap-south-1:111122223333:oncall

# 9. DynamoDB built-in Contributor Insights, throttled keys only (cheap to leave on)
aws dynamodb update-contributor-insights --table-name Orders \
  --contributor-insights-action ENABLE --contributor-insights-mode THROTTLED_KEYS
aws dynamodb describe-contributor-insights --table-name Orders \
  --query '[ContributorInsightsStatus,ContributorInsightsMode]'
# [ "ENABLED", "THROTTLED_KEYS" ]

# 10. Application Insights: onboard a resource group and list detected problems
aws application-insights create-application --resource-group-name loans-app \
  --auto-config-enabled --ops-center-enabled \
  --ops-item-sns-topic-arn arn:aws:sns:ap-south-1:111122223333:oncall
aws application-insights list-problems --resource-group-name loans-app \
  --query 'ProblemList[].[Title,SeverityLevel,Status]' --output text
# SQL Server deadlocks causing web tier errors   High   PENDING

# 11. Clean up
aws cloudwatch delete-insight-rules --rule-names news-top-talkers
aws cloudwatch delete-alarms --alarm-names news-one-ip-dominates
aws dynamodb update-contributor-insights --table-name Orders --contributor-insights-action DISABLE
aws ecs update-cluster-settings --cluster shop-prod --settings name=containerInsights,value=disabled

In the console, open CloudWatch → Insights: Container Insights shows a map of clusters with the busiest pods; Lambda Insights a multi-function table sorted by cold starts; Contributor Insights the rules and their top-N graphs; Application Insights the applications and their problems.

#Summary

Key takeaways

• The CloudWatch Insights family goes deeper than the basic metrics, one part of the system each, and is built on CloudWatch metrics, logs and alarms.
• Container Insights: metrics and logs from containers on ECS, EKS, Kubernetes on EC2 and Fargate (ECS and EKS), at cluster, node, service, task, pod and container level, including restarts. ECS: a cluster setting (enabled or enhanced), no agent. EKS / Kubernetes: a containerised CloudWatch agent (EKS add-on, DaemonSet) plus Fluent Bit; needs IAM permission. EKS on Fargate: ADOT, fewer metrics, no enhanced observability. Enhanced observability is billed per observation on EKS.
• Lambda Insights: an extension delivered as a Lambda layer; one performance event per invocation into /aws/lambda-insights; CPU time, memory, network, /tmp, cold starts (init_duration) and worker shutdowns. Needs the managed execution-role policy, and a Logs endpoint or NAT from private subnets.
• Contributor Insights: a rule over CloudWatch Logs (JSON or CLF) ranks the top-N contributors and counts unique ones, in real time, from creation onwards (use Logs Insights for the past). Up to four keys and four filters, Count or Sum. Alarm with INSIGHT_RULE_METRIC. Built-in for DynamoDB (most accessed / most throttled keys, hot partitions) and PrivateLink. It reports; NACLs and WAF block.
• Application Insights: point it at a resource group; it recognises .NET, IIS, SQL Server, Java, Active Directory and more on EC2, configures metrics, logs and dynamic alarms through Systems Manager, correlates observations into problems with a likely root cause, and sends them to EventBridge and OpsCenter.
• Look-alikes: Logs Insights (query stored logs), CloudTrail Insights (unusual API activity), X-Ray (traces, service map; SDKs in maintenance since February 2026, use OpenTelemetry), Application Signals (APM, SLOs), Managed Prometheus / Grafana, and the unified agent for EC2 memory and disk.

Self-check: the CloudWatch Insights family30 questions