Amazon Data Firehose

By the end of this lesson you will be able to

• Explain what Amazon Data Firehose (formerly Kinesis Data Firehose) does: load streaming data into destinations with no consumer code, no servers and no shards.
• Name its sources (Direct PUT, Kinesis Data Streams, Amazon MSK, and AWS services such as CloudWatch Logs, IoT, EventBridge, SNS and WAF) and its destinations (S3, Redshift through S3, OpenSearch, Iceberg tables, Splunk, Snowflake, partners and any HTTP endpoint).
• Predict when Firehose delivers a batch from the buffer size and buffer interval hints, and calculate file sizes, file counts and delays.
• Use Lambda transformation, format conversion to Parquet or ORC, and dynamic partitioning, and know the limits of each.
• Describe what happens when a destination fails: retries, the S3 backup, the 24-hour limit, and duplicates.
• Choose between Kinesis Data Streams and Firehose for an exam scenario, and know when to use both.

#From stream to storage

#The last-mile problem

Lesson 13.5 showed how Kinesis Data Streams collects a fast, endless flow of records. Collecting is only half the job. Sooner or later most of that data must end up somewhere it can be kept and queried: files in S3, tables in Redshift, an index in OpenSearch, a security tool such as Splunk.

Writing that last step yourself is more work than it looks. A consumer that loads a stream into S3 must:

• read every shard and keep track of its position;
• group thousands of small records into files of a sensible size, because millions of tiny files make S3 queries slow and expensive;
• compress, encrypt and name the files so they can be found later;
• retry when the destination is slow or down, without losing or doubling data;
• scale up when traffic grows on match day and back down at night;
• run somewhere, be patched and be monitored.

None of that is your business logic. It is plumbing, and every team would write the same plumbing.

#What Firehose is

Amazon Data Firehose is a fully managed service that takes streaming data and loads it into a destination. You create a Firehose stream, choose a source and a destination, and Firehose does the plumbing: buffering, batching, compression, encryption, optional transformation, delivery and retries. As the notes put it, you do not write any consumer code; it takes the data and stores it directly.

• Serverless and fully managed. There are no servers, no shards and no capacity to plan. Firehose scales up and down on its own.
• Pay for what you use. You pay per GB that goes in (plus optional features), not for idle capacity.
• Near real-time. It collects records in a buffer and delivers them in batches, so data arrives seconds to minutes after it was sent, not milliseconds.
• No storage of its own. It keeps data only until it is delivered (at most 24 hours if delivery keeps failing). There is no replay.

The service was called Amazon Kinesis Data Firehose until February 2024, when AWS renamed it Amazon Data Firehose. Only the name changed: the API, the CLI command (aws firehose), IAM actions (firehose:*) and CloudWatch metrics are the same. Older exam questions and blog posts still say “Kinesis Data Firehose”; it is the same service.

#Sources: how data gets in

A Firehose stream has exactly one source, chosen when you create it:

Many AWS services deliver through Direct PUT on your behalf, which is why Firehose turns up everywhere in architecture diagrams:

• CloudWatch Logs subscription filters (optionally decompressed by Firehose, since December 2023), and CloudWatch metric streams;
• AWS IoT Core rules (sensor data);
• EventBridge rules, with Firehose as a target;
• SNS subscriptions (lesson 13.4: SNS to S3 through Firehose);
• AWS WAF logs, VPC Flow Logs, Route 53 Resolver query logs, API Gateway access logs, CloudFront standard logs (lesson 11.1).

Some details worth knowing:

• A record is any blob of bytes up to 1,000 KiB (before base64 encoding). The slides say “up to 1 MB”, which is the same idea. Firehose does not care whether it is JSON, CSV, Avro, plain text or binary; only format conversion needs JSON.
• Direct PUT quotas per stream: 500,000 records/s, 2,000 requests/s and 5 MiB/s in N. Virginia, Oregon and Ireland; 100,000 records/s, 1,000 requests/s and 1 MiB/s elsewhere. Firehose raises them by itself when it sees throttling, and you can ask for more. With a Kinesis data stream as the source, this quota does not apply.
• A stream whose source is a Kinesis data stream or MSK does not accept PutRecord calls; producers write to the data stream instead.
• Side note: in November 2024 AWS previewed databases as a source (change data capture from MySQL and PostgreSQL into Iceberg tables). The preview was removed in September 2025, so it is not a Firehose feature today, and it is not on the exam.

#Destinations: where data goes

Each Firehose stream has one destination (plus an optional S3 bucket for backup, see below):

Things to remember:

• Redshift goes through S3. Firehose never inserts rows into Redshift directly. It stages the files in an S3 bucket you choose, then issues COPY. The cluster (or Serverless workgroup) must be publicly accessible with Firehose’s IP addresses allowed, because Firehose connects to it from outside your VPC.
• What is not a destination: DynamoDB, RDS, an SQS queue, an EC2 instance, an EFS file system. To reach those, use an HTTP endpoint you build, or a Lambda consumer on a Kinesis data stream.
• S3 object names. By default the key starts with the UTC date and hour, YYYY/MM/dd/HH/, and you can add your own prefix in front, or build one with expressions such as !{timestamp:yyyy}. Records are concatenated with no delimiter, so turn on new line delimiter if Athena will read them.
• Cross-account and cross-Region delivery are possible (for example a central logging account’s bucket), with the right bucket policy and IAM role.

#Buffering: why it is near real-time

#Size and interval hints

Firehose keeps incoming records in a buffer and delivers the buffer when the first of two conditions is met:

• the buffer size hint (how many MB to collect), or
• the buffer interval hint (how many seconds to wait).

If you set one, you must set the other. They are called hints because Firehose may deliver a little differently when that is more efficient; for example, if delivery falls behind, it raises the buffer size by itself to catch up.

So with the defaults, a record sent to an S3 stream can wait up to five minutes before it lands. That is the meaning of near real-time.

#Zero buffering

The notes (and the classic exam answer) say Firehose buffers for at least 1 minute or 1 MB. That was true until December 2023. Today the interval can be 0 seconds (“zero buffering”): Firehose then does not wait and delivers within a few seconds. Limits:

• not available for the S3 backup copy, nor with dynamic partitioning;
• below 60 seconds Firehose uses multipart uploads to S3, so you pay for more PUT requests, and the files are small;
• check the partner’s recommended hints before lowering them.

Even at 0 seconds Firehose is still the near real-time choice in exam terms. For “real-time, sub-second, custom processing”, the answer stays Kinesis Data Streams with a consumer.

#Working out a flush

The rule is simple: time to fill = buffer size ÷ incoming rate. If that is shorter than the interval, size wins; otherwise the interval wins.

• 50 KB/s into a 5 MB buffer: 5,120 KB ÷ 50 KB/s ≈ 102 s, shorter than 300 s, so a 5 MB file about every 100 seconds; 840 files a day.
• 5 KB/s into the same buffer: 1,024 s to fill, so the 300-second interval wins: a file of 1.5 MB every 5 minutes; 288 files a day.
• 5 KB/s into a 128 MB buffer with a 900 s interval: still the interval, a 4.4 MB file every 15 minutes; 96 files a day, records up to 15 minutes late.

Larger buffers mean fewer, bigger files (cheaper and faster to query) and more delay. Small buffers mean fresher data and the small files problem: S3 charges per PUT, and Athena or Glue spend more time opening tiny files than reading them.

Try the numbers yourself:

#Transforming data on the way

Firehose can change the data between the source and the destination. The processing order is: de-aggregation (if enabled), Lambda transformation, dynamic partitioning, format conversion, delivery.

#Lambda transformation

Turn on data transformation and give Firehose a Lambda function. Typical jobs: CSV to JSON (or JSON to CSV, as in the notes), masking phone numbers or card numbers, adding fields (a city name from a postcode), dropping debug events, fixing timestamps.

• Firehose keeps a separate buffer for Lambda: 0.2 to 3 MB (default 1 MB, 256 KB for Splunk and Snowflake) or 0 to 900 seconds (default 60).
• It invokes the function synchronously; the request and response are limited to 6 MB, and the function may run at most 5 minutes.
• The function returns each record with the same recordId, a result and the new data (base64). result is Ok (deliver it), Dropped (throw it away on purpose) or ProcessingFailed.
• If the invocation fails (timeout, throttling), Firehose retries three times by default, then treats the whole batch as failed.
• Failed records go to the processing-failed/ prefix in S3, with the error and the function’s ARN. Nothing disappears silently.
• You can keep an untouched copy with source record backup to S3, useful when the transformation has a bug and you need to run it again.
• AWS provides blueprints (sample functions) for common jobs such as Apache log to JSON.

#Format conversion: JSON to Parquet or ORC

For a data lake, columnar formats are far better than JSON: Athena reads only the columns a query needs, and the files compress well. Firehose can convert JSON to Apache Parquet or Apache ORC by itself, without Lambda:

• It needs a schema from a table in the AWS Glue Data Catalog (lesson 17.6).
• It reads the JSON with a deserializer (OpenX JSON SerDe, the usual choice, or Hive JSON SerDe) and writes Parquet or ORC.
• The input must be JSON. For CSV or other text, first turn it into JSON with a Lambda transformation.
• Records that do not match the schema go to the error prefix as format-conversion-failed.
• Compression for S3: gzip, Snappy, Zip or Hadoop-compatible Snappy (Redshift: gzip only).

The slides list the formats Firehose “supports” (CSV, JSON, Parquet, Avro, raw text, binary) and the conversions (to Parquet or ORC, compression with gzip or Snappy). Read it as: any bytes can pass through, but the built-in conversion only goes from JSON to Parquet or ORC.

#Dynamic partitioning

By default every file goes under the time prefix. Dynamic partitioning (since 2021) groups records by keys inside the data, such as customer_id or city, and writes each group to its own prefix, for example city=sylhet/2026/10/26/. A query for one city then scans only that folder.

• Keys come from inline parsing with jq (for JSON) or from your Lambda function; up to 50 keys.
• The prefix uses expressions like !{partitionKeyFromQuery:city}.
• S3 destination only. It can be enabled only when you create the stream, and cannot be turned off later.
• Each partition value has its own buffer, so files are smaller and the delay can reach 1.5 times the interval. No zero buffering.
• Quota: 500 active partitions per stream at a time (up to 2,500 on request). Records beyond that go to the error prefix (activePartitionExceeded). Partition by keys with few values, never by a transaction ID.
• It costs extra per GB and per object delivered.

#When delivery fails

Firehose retries, and when it cannot deliver it backs the data up to S3 so you can load it later. Each destination behaves a little differently:

The S3 backup has two modes for non-S3 destinations: failed data only (the default) or all data (a full copy, handy as a cheap archive). The slides show it as “all or failed data” going to an S3 backup bucket. If both the destination and the backup bucket fail, Firehose pauses delivery until one of them works again.

Delivery is at least once. When an acknowledgement is lost, Firehose sends the batch again, so Splunk, HTTP endpoints, OpenSearch and Redshift can receive duplicates. S3, Iceberg tables and Snowflake are the exceptions. Put an event ID in each record so the destination (or your queries) can remove duplicates.

#Security, monitoring and cost

Access. Firehose works with an IAM role that you give it: the role allows it to read the source (for example kinesis:GetRecords), invoke the Lambda function, read the Glue table and write to the bucket. Producers need firehose:PutRecord / PutRecordBatch permission. A broken role is the most common reason delivery stops.

Encryption.

• In transit: all API calls and deliveries use TLS (HTTPS).
• At rest inside Firehose: for Direct PUT, turn on server-side encryption with KMS (an AWS owned key or your own customer managed key). With a Kinesis data stream as the source, Firehose relies on the stream’s encryption instead.
• At the destination: for S3, the bucket’s default encryption or an SSE-KMS key you choose (lesson 10.8).

Networking. Producers in a private subnet can reach Firehose through an interface VPC endpoint without the internet (chapter 22). Firehose can deliver to an OpenSearch domain inside a VPC (it places network interfaces in your subnets, which has its own small charge).

Monitoring. CloudWatch metrics such as IncomingRecords, DeliveryToS3.Success and, the most useful one, DeliveryToS3.DataFreshness: the age of the oldest record not yet delivered. Alarm on freshness, and turn on error logging to CloudWatch Logs.

Cost. You pay per GB ingested (about $0.029 per GB for Direct PUT and Kinesis sources in the first tier), plus extra per GB for format conversion and dynamic partitioning, plus what the destination charges. One trap: for Direct PUT and Kinesis sources, each record is rounded up to the next 5 KB. 1,000 records of 1 KB per second are billed as 5 KB each, five times the real volume; packing them into fewer, larger records (5 KB or more) cuts the ingestion bill.

#Kinesis Data Streams vs Firehose

These two are the classic exam confusion, because both carry streaming data and both were once “Kinesis”.

The notes list the speed of Firehose as “minimum 1 minute or 1 MB buffer” and the scaling of Data Streams as “manual shard management”. Both are outdated: Firehose can buffer for 0 seconds since 2023, and Data Streams has had an on-demand mode that scales for you since 2021 (lesson 13.5). The exam still describes Data Streams as the one you size and Firehose as the one that scales itself.

They are not rivals. The most common design uses both: producers write to a Kinesis data stream, a real-time consumer (Lambda or Managed Service for Apache Flink) reads it for alerts and dashboards, and a Firehose stream with that data stream as its source archives everything to S3, often as Parquet, for Athena.

#What goes wrong

#Firehose in system design

• Central logging. CloudWatch Logs subscription filters from many accounts → Firehose → S3 in a logging account (archive, Glacier after 90 days) and OpenSearch (search). Lesson 19.2 has the details.
• Security data to a security analytics tool. WAF logs, VPC Flow Logs and CloudTrail (through CloudWatch Logs) → Firehose → Splunk or another partner, with failed data backed up to S3.
• A data lake from clickstreams. App events → Firehose (Lambda clean-up, Parquet, dynamic partitioning) → S3 → Glue Data Catalog → Athena and dashboards. Chapter 17 builds this pipeline end to end.
• Real-time plus archive. Kinesis data stream → Lambda for live alerts, and Firehose from the same stream to S3.
• IoT. Sensors → AWS IoT Core rule → Firehose → S3 for history, with a second rule to a real-time path if needed.
• Fan-out archive. An SNS topic with a Firehose subscription keeps a copy of every published message in S3 (lesson 13.4).
• Streaming into tables. Firehose into Iceberg tables or S3 Tables gives SQL tables that update as data arrives, without a separate ETL job (side note; the exam rarely asks about it yet).

#Try it yourself

Create a Firehose stream to S3 in Singapore, send a few records, and watch them arrive. The role and bucket are examples; the role must trust firehose.amazonaws.com and allow s3:PutObject on the bucket.

# 1. Create a Direct PUT stream to S3: 1 MB or 60 seconds, gzip, custom prefix
aws firehose create-delivery-stream --region ap-southeast-1 \
  --delivery-stream-name clicks-example \
  --delivery-stream-type DirectPut \
  --extended-s3-destination-configuration '{
    "RoleARN": "arn:aws:iam::111122223333:role/firehose-clicks-example",
    "BucketARN": "arn:aws:s3:::clicks-lake-example",
    "Prefix": "clicks/",
    "ErrorOutputPrefix": "errors/",
    "BufferingHints": { "SizeInMBs": 1, "IntervalInSeconds": 60 },
    "CompressionFormat": "GZIP"
  }'
# {
#     "DeliveryStreamARN": "arn:aws:firehose:ap-southeast-1:111122223333:deliverystream/clicks-example"
# }

# 2. Wait until it is ACTIVE (about a minute)
aws firehose describe-delivery-stream --region ap-southeast-1 \
  --delivery-stream-name clicks-example \
  --query 'DeliveryStreamDescription.DeliveryStreamStatus'
# "ACTIVE"

# 3. Send one record (CLI v2 needs raw-in-base64-out to send plain text); add \n yourself
aws firehose put-record --region ap-southeast-1 --delivery-stream-name clicks-example \
  --cli-binary-format raw-in-base64-out \
  --record '{"Data":"{\"match\":\"BAN-IND\",\"event\":\"open_scorecard\"}\n"}'
# {
#     "RecordId": "aBcD1234...example",
#     "Encrypted": false
# }

# 4. Send a batch (up to 500 records or 4 MiB per call); always check FailedPutCount
aws firehose put-record-batch --region ap-southeast-1 --delivery-stream-name clicks-example \
  --cli-binary-format raw-in-base64-out \
  --records '[{"Data":"{\"event\":\"share\"}\n"},{"Data":"{\"event\":\"replay\"}\n"}]'
# {
#     "FailedPutCount": 0,
#     "Encrypted": false,
#     "RequestResponses": [ { "RecordId": "..." }, { "RecordId": "..." } ]
# }

# 5. About a minute later (the interval wins: far less than 1 MB was sent), one object appears
aws s3 ls s3://clicks-lake-example/clicks/ --recursive
# 2026-10-26 09:14:03        212 clicks/2026/10/26/09/clicks-example-1-2026-10-26-09-13-02-1a2b3c4d.gz

# 6. Check freshness: the age in seconds of the oldest undelivered record
aws cloudwatch get-metric-statistics --region ap-southeast-1 --namespace AWS/Firehose \
  --metric-name DeliveryToS3.DataFreshness \
  --dimensions Name=DeliveryStreamName,Value=clicks-example \
  --start-time 2026-10-26T09:00:00Z --end-time 2026-10-26T10:00:00Z \
  --period 300 --statistics Maximum

# 7. Clean up (Firehose has no fixed fee, but the bucket keeps costing)
aws firehose delete-delivery-stream --region ap-southeast-1 --delivery-stream-name clicks-example

The API still calls a Firehose stream a delivery stream, a leftover from the old name. Try changing the hints with aws firehose update-destination (it needs the current version ID from describe-delivery-stream) and watch the file sizes change.

#Summary

Key takeaways

• Amazon Data Firehose (called Kinesis Data Firehose until February 2024) loads streaming data into destinations: fully managed, serverless, automatic scaling, pay per GB, no consumer code, no storage and no replay.
• Sources (one per stream): Direct PUT (SDK PutRecord/PutRecordBatch, Kinesis Agent, CloudWatch Logs, IoT, EventBridge, SNS, WAF, VPC Flow Logs…), Kinesis Data Streams, Amazon MSK. Records up to 1,000 KiB.
• Destinations (one per stream): S3, Redshift (S3 first, then COPY), OpenSearch, Iceberg tables / S3 Tables, Splunk, Snowflake, partners (Datadog, New Relic, MongoDB…), custom HTTP endpoint.
• Buffering: deliver when the size (S3: 1–128 MB, default 5) or the interval (0–900 s, default 300) is reached first. Near real-time. Zero buffering exists since December 2023; the old “minimum 60 s” is outdated.
• Lambda transformation (CSV to JSON, masking; Ok/Dropped/ProcessingFailed, 5-minute and 6 MB limits), format conversion JSON to Parquet/ORC with a Glue Data Catalog schema (no Lambda needed), dynamic partitioning by keys in the data (S3 only, set at creation, 500 active partitions).
• Failures: retries, then S3 backup (failed data or all data); S3 retries up to 24 h for Direct PUT, then data is lost; Redshift manifests in errors/. At-least-once: duplicates possible except for S3, Iceberg and Snowflake.
• Data Streams vs Firehose: real-time, replay, many consumers, your code, shards or on-demand → Data Streams. Load into storage, no code, near real-time, auto scaling → Firehose. Often both: Data Streams → Firehose → S3.

Self-check: Amazon Data Firehose31 questions