22.3
VPC Flow Logs and Troubleshooting
Reading a flow log record, telling a NACL from a security group by the ACCEPT/REJECT pattern, and analysis pipelines.
20 min Reading + animations
Tip: tap or hover any dotted-underlined term to see what it means.
By the end of this lesson you will be able to
- Explain what VPC Flow Logs capture (metadata of IP flows, never the payload), at which levels you create them (VPC, subnet, network interface, plus Transit Gateway flow logs), and which traffic they never see.
- Read a default-format record field by field, tell inbound from outbound, and know the extra fields of a custom format (
pkt-srcaddr,tcp-flags,flow-direction,traffic-path…). - Decide whether a security group or a network ACL dropped traffic from the ACCEPT/REJECT pattern, for connections that come in and connections the instance starts.
- Choose a destination: CloudWatch Logs (metric filters, alarms, Logs Insights, Contributor Insights), S3 with Athena (months of history, cheapest), or Amazon Data Firehose (Splunk, OpenSearch), and set the right permissions: an IAM role for CloudWatch Logs, a bucket policy for S3.
- Debug “my instance cannot reach the internet” with the IPv4 checklist, using flow logs as evidence, and know when Reachability Analyzer, Traffic Mirroring or GuardDuty is the better tool.
#The VPC’s call records
When the police investigate a phone, they rarely need the conversation. The operator’s call detail records are enough: which number called which, when, for how long, and whether the call connected. VPC Flow Logs are exactly that for your VPC. They record, for every network interface (ENI), which address talked to which, on which ports and protocol, how many packets and bytes, when, and whether the traffic was accepted or rejected. They never record what was said.
#Metadata only, never the payload
A flow log record holds the source and destination address, the ports, the protocol number, packet and byte counts, the start and end time and the action: ACCEPT or REJECT. It never holds the HTTP request, the SQL query, the file or the password inside the packet. That is the notes’ main exam trick: if a question needs the payload, flow logs are the wrong answer.
The action is the decision of the security groups and network ACLs on that network interface. Flow logs do not say why an application failed above the network, and they do not report routing dead ends as REJECT (the checklist section below shows what a missing route looks like instead).
#Three levels, and the ENIs of managed services
You create a flow log for one of three resources (slide p740):
| Level | What it covers | Typical use |
|---|---|---|
| VPC | Every network interface in the VPC, including ones created later | The default for a production VPC |
| Subnet | Every network interface in that subnet | A sensitive subnet, such as the database tier |
| Network interface (ENI) | One interface, for example one EC2 instance’s | Debugging one server cheaply |
A flow log on a VPC or subnet monitors every ENI inside it, and an ENI launched later gets its own log stream as soon as it sends traffic. That includes the interfaces that AWS services create in your subnets: Elastic Load Balancing, RDS, ElastiCache, Redshift, WorkSpaces, NAT gateways, Transit Gateway attachments and interface VPC endpoints. You still create the flow log from the EC2/VPC side, whatever service owns the ENI.
Three more choices when you create one:
- Traffic type:
ALL,ACCEPTonly, orREJECTonly.REJECTonly is a cheap way to watch blocked traffic;ALLis what you want for investigations. - Destination: CloudWatch Logs, S3 or Amazon Data Firehose (section below).
- Record format and maximum aggregation interval (next section).
Transit Gateway Flow Logs are a separate resource type with their own fields (tgw-id, tgw-attachment-id, packets-lost-no-route, packets-lost-blackhole…), created on the transit gateway or one attachment (lesson 22.6). You can create a flow log for a peered VPC only if it is in your own account.
Flow logs are collected outside the path of the traffic, so turning them on or off never affects throughput or latency.
#Reading a flow log record
#The default format
The default format is version 2: fourteen fields, separated by spaces, always in this order (slide p742):
| # | Field | Meaning |
|---|---|---|
| 1 | version | Format version: 2 for the default format |
| 2 | account-id | The account that owns the network interface |
| 3 | interface-id | The ENI that saw the traffic |
| 4 | srcaddr | Source address (for outgoing traffic, the ENI’s private IPv4 or IPv6 address) |
| 5 | dstaddr | Destination address (for incoming traffic, the ENI’s private address) |
| 6 | srcport | Source port |
| 7 | dstport | Destination port |
| 8 | protocol | IANA protocol number: 6 = TCP, 17 = UDP, 1 = ICMP |
| 9 | packets | Packets in this flow during the interval |
| 10 | bytes | Bytes in this flow during the interval |
| 11 | start | Unix time of the first packet in the interval |
| 12 | end | Unix time of the last packet in the interval |
| 13 | action | ACCEPT or REJECT |
| 14 | log-status | OK; NODATA (no traffic in the interval); SKIPDATA (records were skipped because of an internal capacity limit or error) |
A field that does not apply shows -. A NODATA record has dashes in place of addresses and counts.
#A worked record
2 111122223333 eni-0a1b2c3d4e5f67890 203.0.113.50 10.0.1.25 51544 443 6 12 4310 1730448000 1730448060 ACCEPT OK
Read it left to right: version 2; account 111122223333; seen by ENI eni-0a1b2c3d4e5f67890; from 203.0.113.50 port 51544 (a browser’s ephemeral port) to 10.0.1.25 port 443; protocol 6, TCP; 12 packets, 4,310 bytes; between 08:00:00 and 08:01:00 UTC on 1 November 2024; accepted; logged normally.
The direction is not a field in the default format. You read it from where the ENI’s own private address sits: 10.0.1.25 is in dstaddr, so this is inbound to the ENI. The reply appears as a second record with the addresses and ports swapped. One TCP connection, two records, each from the point of view of the same interface.
Slide p742 lists the uses: srcaddr/dstaddr find a problem address, srcport/dstport a problem port, action tells success or failure caused by security groups or network ACLs, and the whole set feeds analytics on usage patterns or malicious behaviour, with Athena on S3 or CloudWatch Logs Insights.
#Aggregation interval and delivery delay
Flow logs are not real time. Packets of the same flow (same five-tuple on the same ENI) are added up during a capture window:
- The maximum aggregation interval is 10 minutes by default; you can choose 1 minute for finer records (more records, so more cost).
- On Nitro-based instances the interval is always 1 minute or less, whatever you chose.
- After the window closes, delivery usually takes about 5 minutes to CloudWatch Logs and about 10 minutes to S3, best effort.
So the record of a problem appears several minutes after the problem. For seconds, use CloudWatch metrics or Traffic Mirroring, not flow logs.
#Custom format: extra fields
A custom format chooses which fields appear and in what order. The version of the record becomes the highest version among the fields you pick. The fields worth knowing:
| Field (version) | What it adds |
|---|---|
vpc-id, subnet-id, instance-id (3) | Where the ENI lives, without joining against an inventory |
tcp-flags (3) | Bitmask of FIN 1, SYN 2, RST 4, SYN-ACK 18 (OR-ed within the interval: 19 = SYN-ACK + FIN). Many SYN-only flows with no SYN-ACK suggest a port scan |
type (3) | IPv4, IPv6 or EFA |
pkt-srcaddr, pkt-dstaddr (3) | The original source and destination in the packet. On a NAT gateway’s ENI, srcaddr/dstaddr show the NAT’s own address; these show the instance and the real destination. Also EKS pod addresses |
region, az-id, sublocation-type, sublocation-id (4) | Region, AZ ID, Local Zone, Wavelength or Outpost |
pkt-src-aws-service, pkt-dst-aws-service (5) | Which AWS service an address belongs to (S3, DYNAMODB, EC2, ROUTE53…) |
flow-direction (5) | ingress or egress, so you no longer infer direction |
traffic-path (5) | For egress: 1 inside the VPC, 2 internet gateway or gateway endpoint, 3 virtual private gateway, 4 intra-Region peering, 5 inter-Region peering, 6 Local or Wavelength Zone, 7 gateway endpoint, 8 internet gateway |
ecs-… fields (7) | Cluster, service, task and container of ECS tasks in awsvpc mode |
reject-reason (8) | BPA when VPC Block Public Access rejected the flow, EC for encryption controls |
Newer versions (9 to 11, 2025–2026) add regional NAT gateway IDs, the encryption status of a flow, EC2 and Auto Scaling group tag values, and next-hop fields. They are useful in production; the exam knows the default fields, pkt-srcaddr/pkt-dstaddr and the idea of a custom format.
#What flow logs never see
Some traffic is not logged at all:
- Traffic to the Amazon DNS server (the Amazon-provided resolver, the VPC base address + 2, also reachable at 169.254.169.253). If you run your own DNS server, its traffic is ordinary traffic and is logged.
- Traffic to and from 169.254.169.254, the instance metadata service.
- Traffic to and from 169.254.169.123, the Amazon Time Sync Service.
- DHCP traffic and ARP traffic.
- Windows license activation traffic to Amazon’s activation servers.
- Traffic to the reserved address of the default VPC router.
- The source side of mirrored traffic (you see only the copies arriving at the mirror target).
- Traffic between an endpoint network interface and a Network Load Balancer’s network interface.
Two address quirks complete the picture: traffic sent to a secondary private IPv4 address shows the primary address in dstaddr (use pkt-dstaddr), and with a NAT gateway or an EKS pod, the packet’s real addresses also live only in pkt-srcaddr/pkt-dstaddr.
Blind spots, NAT and a miner
A private instance talks to the Amazon DNS server, the metadata service and, through a NAT gateway, the internet. Some of this traffic never appears in flow logs; some appears twice.
Scenario
DNS to the Amazon resolver and metadata calls leave no record at all.
Flow log records
| ENI | srcaddr | dstaddr | pkt-srcaddr | pkt-dstaddr | action |
|---|---|---|---|---|---|
| — | — | — | — | — | nothing |
The instance resolves a name with the Amazon-provided DNS server (the VPC base address plus two, here 10.0.0.2). The query and answer happen, yet the flow log writes nothing: traffic to the Amazon DNS server is not logged.
#Security group or network ACL?
This is the notes’ “ultimate trick” and the exam’s favourite flow log question. Everything rests on one difference from lesson 3.5: a security group is stateful (a reply to allowed traffic is allowed automatically, in either direction), a network ACL is stateless (the reply needs its own rule, usually for the ephemeral ports 1024–65535).
Look at the action field of the request and of its reply (slide p743):
| The connection… | Request record | Reply record | Culprit |
|---|---|---|---|
| comes in (a client calls the instance) | inbound REJECT | none | Security group or network ACL: check both |
| comes in | inbound ACCEPT | outbound REJECT | Network ACL outbound rules |
| goes out (the instance is the client) | outbound REJECT | none | Security group or network ACL (outbound rules) |
| goes out | outbound ACCEPT | inbound REJECT | Network ACL inbound rules |
Why the “ACCEPT then REJECT” rows can only be the network ACL: the security group that accepted the request has already remembered the connection, and it has no power to block the reply. A network ACL remembers nothing; it judges the reply as a brand-new packet to port 51544, and if no rule allows 1024–65535, its final * rule denies it.
When the request itself is rejected, the pattern alone cannot separate the two. Look at the other layer: a default network ACL allows everything, so the security group is to blame; a security group that clearly allows the source points at a network ACL deny rule. Reachability Analyzer (below) names the exact rule.
Reading ACCEPT and REJECT
An e-commerce web server in a public subnet, with a flow log on its network interface. Each packet that crosses the network ACL and the security group leaves a line in the table below.
Scenario
One HTTPS connection produces two records on the ENI: the request in, the reply out.
A customer opens the shop. Her browser picks a temporary (ephemeral) port, 51544, and connects to port 443. The network ACL is checked first, at the subnet edge: its inbound rule allows TCP 443.
Two details that sometimes confuse real investigations:
REJECTis also written for packets that arrive after the connection was closed (late retransmissions), so a fewREJECTlines on an otherwise healthy server are normal.- The record is on the ENI, after the network ACL and security group have judged it. A packet the network ACL drops on the way in is still logged on the destination ENI as
REJECT; you do not need a flow log “on the NACL”.
Practise on raw lines. Pick a case, click a line to label its fields, and name the culprit:
Flow log detective
Pick a case, read the raw flow log lines (default format), click a line to label its fields, then name the culprit.
Web server 10.0.1.25 in a public subnet. Customers say the shop “keeps loading” and then times out. The security group allows 443 from anywhere.
Raw records (click one to parse it)
Click a line above to see its fields.
Who is the culprit?
#Where flow logs go
Slide p744 shows the architectures. A flow log publishes to one destination; create two flow logs on the same resource if you need two (up to 250 per resource).
#CloudWatch Logs: watch and alert
The flow log writes into a log group, one log stream per ENI (lesson 19.2). From there:
- Metric filters turn matching lines into a metric, for example rejected SSH (
dstport22,actionREJECT) or any RDP attempt; a CloudWatch alarm on that metric notifies SNS (lesson 19.3). - Logs Insights queries the stored records; it discovers the flow log fields by name (
srcAddr,dstPort,action,bytes…). - Contributor Insights keeps a live top-N: the “top 10 IP addresses” of slide p744, the busiest source–destination pairs, the most rejected sources (lesson 19.4).
- Subscription filters can stream the group on to Lambda, Kinesis or Firehose.
Records arrive within minutes, which is the closest flow logs get to “live”. Storing many months in CloudWatch Logs costs more than in S3.
#S3: keep it long, query it with Athena
The flow log writes compressed log files to a bucket, under AWSLogs/<account-id>/vpcflowlogs/<region>/<year>/<month>/<day>/. Options at creation:
- File format: plain text (default) or Apache Parquet, columnar, smaller and much cheaper to query.
- Hive-compatible prefixes (
year=2026/month=10/…) so Athena and Glue understand the partitions. - Hourly partitions instead of daily, so a query for one afternoon reads only those hours.
Athena then runs standard SQL straight on the files, charging by data scanned (lesson 17.1); QuickSight can chart the results. The VPC console even generates the Athena table and saved queries for you. S3 is the cheapest place for months or years of records, and lifecycle rules can move old files to colder classes.
#Amazon Data Firehose: to another tool
The flow log publishes to a Firehose stream (in the same or another account), which buffers, can transform, and delivers to Splunk, OpenSearch, an HTTP endpoint or S3, with a backup bucket for failures (lesson 13.6). Choose it when a third-party or central security tool must receive the records continuously. Amazon Data Firehose was called Kinesis Data Firehose until February 2024; the notes and the slides use the old name, and exam questions may use either.
| You need | Destination |
|---|---|
| Alarms on rejected traffic, top talkers, quick searches over recent days | CloudWatch Logs (metric filters, alarms, Logs Insights, Contributor Insights) |
| Months of history, SQL, the lowest cost | S3 + Athena |
| Splunk, OpenSearch, a partner SIEM, another account’s pipeline | Amazon Data Firehose |
#Permissions: a role for CloudWatch, a bucket policy for S3
Flow logs cannot write anywhere without permission (slide p745):
- CloudWatch Logs needs an IAM role that you pass to the flow log. Its trust policy must allow the service principal
vpc-flow-logs.amazonaws.comto assume it, and its permissions policy must allowlogs:CreateLogGroup,logs:CreateLogStream,logs:PutLogEvents,logs:DescribeLogGroupsandlogs:DescribeLogStreams. The person creating the flow log needsiam:PassRolefor that role. Addingaws:SourceAccountandaws:SourceArnconditions to the trust policy protects against the confused deputy problem. - S3 needs no IAM role. The bucket policy must allow the log delivery service
delivery.logs.amazonaws.comtos3:PutObject(ands3:GetBucketAcl). If you own the bucket and may change its policy, AWS adds this policy automatically when you create the flow log; for a bucket in another account, its owner adds it. - Firehose delivery uses a service-linked role in your account (and, across accounts, a role the destination account trusts).
Three destinations, two permission models
A flow log on a VPC can publish to CloudWatch Logs, to S3 or to Amazon Data Firehose. Each destination answers a different question, and CloudWatch Logs is the only one that needs an IAM role.
Scenario
Records arrive in minutes; metric filters and Insights turn them into alarms and top-10 lists.
The flow log assumes its IAM role, which trusts vpc-flow-logs.amazonaws.com and allows logs:CreateLogGroup, CreateLogStream, PutLogEvents and the Describe calls. It writes into the log group, one log stream per network interface.
#Debugging a VPC: the checklist and the evidence
Lesson 3.3 gave the checklist for “my instance cannot reach the internet”. Here is the same list with the evidence a flow log gives for each fault, because the evidence is what tells you which item to fix.
Public subnet:
- Public IPv4 address or Elastic IP on the instance.
- Internet gateway attached to the VPC.
- Route table of the subnet sends
0.0.0.0/0to the IGW. - Security group allows the traffic (inbound for the service port; outbound is open by default).
- Network ACL allows the traffic both ways, especially the ephemeral ports.
Private subnet:
- A NAT gateway exists in a public subnet, and that public subnet’s route table sends
0.0.0.0/0to the IGW. - The private subnet’s route table sends
0.0.0.0/0to the NAT gateway. - Then the same security group and network ACL checks.
| Fault | What the flow log shows | Fix |
|---|---|---|
| No public IP (public subnet) | Outbound ACCEPT with a few packets (SYN retries), no inbound line, no REJECT | Elastic IP, or auto-assign public IPv4 |
No route 0.0.0.0/0 → igw-… | ACCEPT lines only, nothing back; a route is not a firewall | Add the route to the subnet’s own route table |
| No IGW attached | Same as above | Attach the IGW, then add the route |
| Security group blocks the port | Inbound REJECT, no reply line | Add the inbound rule |
| Network ACL without ephemeral ports | Inbound ACCEPT, outbound REJECT (or outbound ACCEPT, inbound REJECT for calls the instance makes) | Allow 1024–65535 in the right direction |
| Private route to NAT missing | Outbound ACCEPT on the instance, nothing on the NAT gateway’s ENI | 0.0.0.0/0 → nat-… in the private route table |
| NAT gateway’s subnet has no IGW route | Records on the NAT ENI going out, none coming back | Put the NAT gateway in a real public subnet |
The pattern to remember: REJECT means a firewall; ACCEPT with silence means routing or addressing.
Debug the VPC
A web server in a public subnet, an app server in a private subnet behind a NAT gateway. Each scenario breaks one item of the checklist and shows what the flow log says about it.
Scenario
Public IP, IGW attached, route to the IGW, security group, network ACL both ways.
Checks 1 and 2: the web server has a public IPv4 address (or an Elastic IP), and an internet gateway is attached to the VPC. Without both, no internet traffic can reach it.
#Other tools for the same question
Flow logs record what did happen. Three neighbours answer different questions:
- VPC Reachability Analyzer analyses the configuration between a source and a destination (instance, ENI, IGW, VPN gateway, peering connection, transit gateway…) and reports either a reachable path or the exact component that blocks it: “security group sg-… has no rule for TCP 8080”. It sends no packets, so it works before any traffic exists and even when flow logs would show nothing. It is the checklist above, automated, at about 0.10 USD per analysis.
- Network Access Analyzer checks intent across a whole network: “list every path from the internet to the database subnet” or “nothing in production may reach development”, and reports the paths that violate it. It is for audits, not for one broken connection.
- Traffic Mirroring copies full packets, payload included, to an IDS or packet analyser (lesson 22.7). Use it when metadata is not enough.
GuardDuty analyses VPC flow log data too, but from its own independent feed: you do not have to enable flow logs for it, and enabling them changes nothing for GuardDuty (lesson 21.7). Your own flow logs are for your own investigations and audits.
#What goes wrong
- No data in CloudWatch Logs: the IAM role lacks
logs:PutLogEvents/CreateLogStream, or its trust policy does not namevpc-flow-logs.amazonaws.com.describe-flow-logsshowsDeliverLogsStatus: FAILEDwith “Access error”. - “Flow log created but empty” after two minutes: nothing is wrong yet. The first records appear only after traffic, one aggregation interval and the delivery delay: 10 to 20 minutes is normal.
- Creation to S3 fails: the bucket is in another account and its owner has not added the
delivery.logs.amazonaws.combucket policy, or you lackPutBucketPolicyon your own bucket. Note that the automatic policy replaces an existing bucket policy. - Needed a field you did not include: flow logs cannot be edited. Delete and recreate with a custom format; the past stays without that field.
- NAT gateway logs show only the NAT’s own address: add
pkt-srcaddrandpkt-dstaddrto see the instance and the real destination. - Assuming no line means no traffic: DNS to the Amazon resolver, metadata, Time Sync, DHCP and ARP are never logged.
- Blaming a security group for a dropped reply: impossible; a dropped reply after an
ACCEPTis the network ACL. - A routing fault hunted in REJECT lines: routes, missing public IPs and detached gateways leave
ACCEPTand silence, notREJECT. Use Reachability Analyzer. - Huge bill from a VPC-wide
ALLflow log at 1-minute interval into CloudWatch Logs: send bulk history to S3 in Parquet, keep CloudWatch Logs for the few things you alarm on, or logREJECTonly for watching. - Many
REJECTlines on a healthy server: late packets after a connection closed are rejected and logged. Look for patterns, not single lines. - Expecting payload, URLs or SQL: flow logs never have them. WAF logs, application logs or Traffic Mirroring do.
#Flow logs in system design
Patterns to remember:
- Every production VPC has a flow log, created by the account baseline (CloudFormation stack sets or Control Tower customisations), with the AWS Config managed rule
vpc-flow-logs-enabledreporting any VPC without one (lesson 19.7). - Central log archive: every account’s flow logs go to one S3 bucket in a log-archive account (Parquet, hourly partitions), queried with Athena by the security team. Bucket policies allow the log delivery service from the member accounts only, and S3 Object Lock or a deny-delete policy keeps the evidence safe.
- Two flow logs, two jobs: a
REJECT-only flow log into CloudWatch Logs for metric filters and alarms, and anALLflow log into S3 for history. - SIEM feed: flow logs through Firehose into Splunk or OpenSearch, next to WAF logs and CloudTrail (lesson 13.6).
- Troubleshooting toolkit: Reachability Analyzer before traffic exists, flow logs after, Traffic Mirroring when the payload matters, GuardDuty for threats.
- Blocking what you find: a top talker found in flow logs is blocked with a network ACL deny rule, a WAF rule, or Network Firewall, depending on where the traffic enters (lesson 24.3 compares them).
- Cost awareness: flow logs themselves are charged as vended logs (in us-east-1 about 0.50 USD per GB into CloudWatch Logs and less into S3 or Firehose, with tiered discounts at high volume; check the CloudWatch pricing page, prices checked November 2026), plus storage and Athena scans. The traffic they reveal is often the bigger saving: flow logs are how teams discover that NAT gateway or cross-AZ traffic dominates the bill (lesson 22.8).
#Try it yourself
These commands use the AWS CLI v2 in ap-southeast-1 with made-up IDs. Flow logs cost a little per GB; delete what you create.
# 1. The IAM role for CloudWatch Logs delivery: trust the flow logs service, allow the logs actions
cat > trust.json <<'EOF'
{ "Version": "2012-10-17",
"Statement": [ { "Effect": "Allow",
"Principal": { "Service": "vpc-flow-logs.amazonaws.com" },
"Action": "sts:AssumeRole",
"Condition": { "StringEquals": { "aws:SourceAccount": "111122223333" } } } ] }
EOF
cat > perms.json <<'EOF'
{ "Version": "2012-10-17",
"Statement": [ { "Effect": "Allow",
"Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents",
"logs:DescribeLogGroups", "logs:DescribeLogStreams" ],
"Resource": "*" } ] }
EOF
aws iam create-role --role-name flow-logs-to-cwl --assume-role-policy-document file://trust.json
aws iam put-role-policy --role-name flow-logs-to-cwl --policy-name publish --policy-document file://perms.json
# 2. A REJECT-only flow log on the VPC into CloudWatch Logs, 1-minute interval
aws ec2 create-flow-logs --resource-type VPC --resource-ids vpc-0a1b2c3d4e5f67890 \
--traffic-type REJECT --log-destination-type cloud-watch-logs \
--log-group-name /vpc/shop-rejects \
--deliver-logs-permission-arn arn:aws:iam::111122223333:role/flow-logs-to-cwl \
--max-aggregation-interval 60
# { "ClientToken": "...", "FlowLogIds": [ "fl-0123456789abcdef0" ], "Unsuccessful": [] }
# 3. Is it delivering?
aws ec2 describe-flow-logs --flow-log-ids fl-0123456789abcdef0 \
--query 'FlowLogs[].[FlowLogStatus,DeliverLogsStatus,DeliverLogsErrorMessage]'
# [ [ "ACTIVE", "SUCCESS", null ] ]
# with a role missing PutLogEvents: [ [ "ACTIVE", "FAILED", "Access error" ] ]
# 4. ALL traffic into S3: custom format, Parquet, Hive-compatible hourly partitions (no role needed)
aws ec2 create-flow-logs --resource-type VPC --resource-ids vpc-0a1b2c3d4e5f67890 \
--traffic-type ALL --log-destination-type s3 \
--log-destination arn:aws:s3:::shop-flow-logs-example/vpc/ \
--log-format '${version} ${vpc-id} ${subnet-id} ${instance-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${tcp-flags} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${pkt-srcaddr} ${pkt-dstaddr} ${flow-direction} ${traffic-path}' \
--destination-options FileFormat=parquet,HiveCompatiblePartitions=true,PerHourPartition=true
aws s3api get-bucket-policy --bucket shop-flow-logs-example --query Policy --output text
# ... "Principal":{"Service":"delivery.logs.amazonaws.com"},"Action":"s3:PutObject" ... (added automatically)
Watch a metric filter and an Insights query work on the CloudWatch copy:
# 5. Count rejected SSH attempts (space-delimited pattern over the 14 default fields) and alarm on them
aws logs put-metric-filter --log-group-name /vpc/shop-rejects --filter-name ssh-rejects \
--filter-pattern '[version, account, eni, src, dst, srcport, dstport="22", protocol="6", packets, bytes, start, end, action="REJECT", status]' \
--metric-transformations metricName=SshRejects,metricNamespace=Shop/Network,metricValue=1
aws cloudwatch put-metric-alarm --alarm-name ssh-scan --namespace Shop/Network --metric-name SshRejects \
--statistic Sum --period 300 --evaluation-periods 1 --threshold 100 \
--comparison-operator GreaterThanThreshold \
--alarm-actions arn:aws:sns:ap-southeast-1:111122223333:netops-oncall
# 6. Top rejected sources in the last hour with Logs Insights
aws logs start-query --log-group-name /vpc/shop-rejects \
--start-time $(date -d '-1 hour' +%s) --end-time $(date +%s) \
--query-string 'filter action = "REJECT" | stats count(*) as rejects by srcAddr, dstPort | sort rejects desc | limit 10'
# { "queryId": "a1b2c3d4-5678-90ab-cdef-EXAMPLE33333" }
aws logs get-query-results --query-id a1b2c3d4-5678-90ab-cdef-EXAMPLE33333
# srcAddr 203.0.113.66 dstPort 22 rejects 4812
# srcAddr 198.51.100.9 dstPort 3389 rejects 1290
Query months of history in S3 with Athena (run in the Athena console or with aws athena start-query-execution):
-- 7. A table over the Parquet files (field names use underscores in Parquet)
CREATE EXTERNAL TABLE vpc_flow_logs (
version int, vpc_id string, subnet_id string, instance_id string, interface_id string,
srcaddr string, dstaddr string, srcport int, dstport int, protocol bigint, tcp_flags int,
packets bigint, bytes bigint, start bigint, `end` bigint, action string, log_status string,
pkt_srcaddr string, pkt_dstaddr string, flow_direction string, traffic_path int)
PARTITIONED BY (`aws-account-id` string, `aws-service` string, `aws-region` string,
`year` string, `month` string, `day` string, `hour` string)
STORED AS PARQUET
LOCATION 's3://shop-flow-logs-example/vpc/AWSLogs/';
MSCK REPAIR TABLE vpc_flow_logs;
-- 8. The auditor's question: who reached MySQL last quarter, and how much data moved?
SELECT srcaddr, dstaddr, count(*) AS records, sum(bytes) AS total_bytes
FROM vpc_flow_logs
WHERE year = '2026' AND month IN ('07', '08', '09')
AND dstport = 3306 AND action = 'ACCEPT'
GROUP BY srcaddr, dstaddr
ORDER BY total_bytes DESC;
-- srcaddr dstaddr records total_bytes
-- 10.0.2.40 10.0.3.15 12960 88412339201
-- 10.0.2.41 10.0.3.15 12941 87990211554
-- 10.0.9.7 10.0.3.15 3 1840 <- the analytics subnet? ask why
Let Reachability Analyzer name the blocking rule for the “two subnets cannot talk” case:
# 9. Analyse the path web → app on TCP 8080 without sending a packet
aws ec2 create-network-insights-path --source i-0aaa1111bbbb22223 --destination i-0ccc3333dddd44445 \
--protocol tcp --destination-port 8080
# { "NetworkInsightsPath": { "NetworkInsightsPathId": "nip-0a1b2c3d4e5f67890", ... } }
aws ec2 start-network-insights-analysis --network-insights-path-id nip-0a1b2c3d4e5f67890
aws ec2 describe-network-insights-analyses --network-insights-path-id nip-0a1b2c3d4e5f67890 \
--query 'NetworkInsightsAnalyses[0].[NetworkPathFound,Explanations[0].ExplanationCode,Explanations[0].SecurityGroup.Id]'
# [ false, "ENI_SG_RULES_MISMATCH", "sg-0app1234567890abc" ]
# 10. Clean up
aws ec2 delete-flow-logs --flow-log-ids fl-0123456789abcdef0 fl-0fedcba9876543210
From an instance, curl -m 5 https://example.com that times out, followed a few minutes later by the matching records in Logs Insights, is the quickest way to feel the delay between an event and its flow log.
#Summary
Key takeaways
- VPC Flow Logs record metadata of IP flows on every ENI (addresses, ports, protocol, packets, bytes, time, ACCEPT/REJECT), never the payload. Levels: VPC, subnet, ENI; managed-service ENIs (ELB, RDS, ElastiCache, Redshift, WorkSpaces, NAT gateway, Transit Gateway) included; Transit Gateway Flow Logs are their own type. No effect on performance.
- Default format v2: version, account-id, interface-id, srcaddr, dstaddr, srcport, dstport, protocol, packets, bytes, start, end, action, log-status. Direction = where the ENI’s own address sits. Custom format adds
vpc-id,subnet-id,instance-id,tcp-flags,pkt-srcaddr/pkt-dstaddr(NAT gateway),flow-direction,traffic-path…. - Not real time: aggregation 10 minutes default or 1 minute (always ≤ 1 minute on Nitro), then ~5 minutes to CloudWatch Logs, ~10 to S3. A flow log cannot be edited: delete and recreate. Nothing is back-filled.
- Not logged: Amazon DNS server, 169.254.169.254 metadata, Time Sync, DHCP, ARP, Windows activation, default router reserved IP, mirrored source traffic.
- SG vs NACL: inbound REJECT → SG or NACL; inbound ACCEPT + outbound REJECT → NACL; outbound ACCEPT + inbound REJECT → NACL. A security group never drops a reply.
ACCEPTand silence → routing or addressing. - Destinations: CloudWatch Logs (metric filters, alarms, Logs Insights, Contributor Insights top-10; needs an IAM role trusted by
vpc-flow-logs.amazonaws.comwithlogs:CreateLogGroup/CreateLogStream/PutLogEvents); S3 (no role, bucket policy fordelivery.logs.amazonaws.com, Parquet + hourly partitions, Athena = months of SQL, cheapest); Amazon Data Firehose (formerly Kinesis Data Firehose) → Splunk, OpenSearch. - Checklist: public IP, IGW, route to IGW, SG, NACL both ways; private: NAT in a public subnet that routes to the IGW, route to NAT. Reachability Analyzer names the blocking configuration without packets; Traffic Mirroring for payload; GuardDuty reads flow data from its own feed.
Answer on your own first, then read the explanation, even when you got it right.
Self-checkQuestion
What does a VPC flow log record contain?
Self-checkQuestion
For which resources can you create a VPC flow log?
Self-checkQuestion
What is the default maximum aggregation interval of a flow log, and what other value can you choose?
Self-checkQuestion
Which two kinds of traffic never appear in VPC flow logs?
Choose 2
Self-checkQuestion
A record reads: 2 111122223333 eni-0a1b2c3d4e5f67890 203.0.113.50 10.0.1.25 51544 443 6 12 4310 1730448000 1730448060 ACCEPT OK. The ENI’s private address is 10.0.1.25. What does it show?
Self-checkQuestion
Customers of a web shop time out. Flow logs show TCP 443 requests as inbound ACCEPT and the server’s replies to the customers’ ports as outbound REJECT. What is the cause?
Self-checkQuestion
A flow log shows only an inbound REJECT for TCP port 3389 from an office address, and no reply record. What can you conclude from this record alone?
Self-checkQuestion
An instance downloads updates from the internet. Its flow log shows the outbound request to port 443 as ACCEPT and the response from port 443 to the instance’s port 40312 as REJECT. What should you fix?
Self-checkQuestion
The security team must analyse twelve months of VPC flow logs with standard SQL queries, as cheaply as possible. What should they do?
Self-checkQuestion
An engineer created a VPC flow log with CloudWatch Logs as the destination. After an hour, the log group has no data. What should be checked first?
Self-checkQuestion
What permission does a flow log need to publish to an S3 bucket in the same account?
Self-checkQuestion
Which two are required for a flow log that publishes to CloudWatch Logs?
Choose 2
Self-checkQuestion
A team needs the pkt-srcaddr field to investigate NAT gateway traffic, but the existing VPC flow log uses the default format. What must they do?
Self-checkQuestion
A security operations team uses Splunk and wants VPC flow logs delivered to it continuously without managing servers. Which destination fits?
Self-checkQuestion
During an incident a manager asks for the 10 source IP addresses sending the most traffic to the VPC, updated continuously. The flow logs already go to CloudWatch Logs. What is the simplest tool?
Self-checkQuestion
The team wants to be paged when more than 100 SSH connection attempts are rejected in 5 minutes anywhere in the VPC. Flow logs go to CloudWatch Logs. What should they build?
Self-checkQuestion
Flow logs on a NAT gateway’s network interface show only the NAT gateway’s own private address as the source of traffic to the internet. How can you find which instance started each flow?
Self-checkQuestion
Which two statements about VPC flow logs are true?
Choose 2
Self-checkQuestion
An EC2 instance in a public subnet cannot download updates. Its flow log shows outbound ACCEPT records with three packets each and no inbound records at all. The security group allows all outbound traffic and the subnet uses the default network ACL. What is the most likely cause?
Self-checkQuestion
Instances in two subnets of the same VPC cannot communicate on port 8080. Where should you look first?
Self-checkQuestion
An instance in a private subnet cannot reach the internet. A NAT gateway exists and the private route table sends 0.0.0.0/0 to it. The flow log on the NAT gateway’s ENI shows the traffic leaving but nothing returning. What is missing?
Self-checkQuestion
A company enables Amazon GuardDuty. An engineer says they must now also enable VPC Flow Logs in every VPC so GuardDuty has data. Is that right?
Self-checkQuestion
Flow logs from a test instance show the same outbound flow to an unknown address on port 3333 in every 10-minute window, about 48 MB each, all ACCEPT. What does this most likely indicate?
Self-checkQuestion
A public web server is unreachable on HTTPS. Its security group allows inbound 443 from anywhere. Which two configuration problems could cause this?
Choose 2
Self-checkQuestion
Before any traffic exists, a team wants to know whether a new instance in subnet A can reach a database in subnet B on port 5432, and if not, which component blocks it. Which tool answers this directly?
Self-checkQuestion
An intrusion detection system must inspect the content of traffic to a payment server for malware signatures. What should feed it?
Self-checkQuestion
A developer opens a port at 10:00 and expects the flow log records in CloudWatch Logs within seconds. When should they expect them?
Self-checkQuestion
An exam answer says to send flow logs to Kinesis Data Firehose, but the current console only lists Amazon Data Firehose. What is the relationship?
Self-checkQuestion
Which two values can appear in the log-status field of a VPC flow log record?
Choose 2
Self-checkQuestion
To detect port scans, a team wants to see flows that sent SYN packets but never got a SYN-ACK, together with the subnet of each ENI. What do they need?
Self-checkQuestion
A network team must see traffic between VPC attachments of a transit gateway, including packets dropped because no route matched. What should they enable?