A Serverless Big Data Ingestion Pipeline

By the end of this lesson you will be able to

• Draw the classic serverless big data ingestion pipeline from memory: IoT devices → IoT Core → Kinesis Data Streams → Data Firehose (with a Lambda transformation) → S3 ingestion bucket → SQS → Lambda → Athena → S3 reporting bucket → QuickSight and Redshift.
• Say what job each hop does, why it is there, and which earlier lesson teaches it in depth.
• Tell which parts are real time and which are near real time, and estimate the delay from a device to a report.
• Name the cost driver of each hop and show, with numbers, why file format and partitions decide the Athena bill.
• Diagnose the usual failures: malformed records, schema drift, late data, small files, unpartitioned scans, throttled Lambda and Athena, poison messages, duplicate events and recursive triggers.
• Adapt the pattern: drop or swap a hop, add a hot path, and pick the exam answer when several pipelines look plausible.

#Why a pipeline, not a service

#The six requirements

Chapters 13, 15 and 17 taught the pieces one by one. This lesson puts them together into the pattern that closes the Data & Analytics section of the exam. The requirement usually reads like this:

1. The ingestion pipeline must be fully serverless: nothing to size, patch or keep running.
2. Data must be collected in real time.
3. The data must be transformed on the way.
4. The transformed data must be queryable with SQL.
5. The reports produced by those queries must be stored in S3.
6. The reports must be loaded into a data warehouse and shown on dashboards.

Each line points to one service. Real time collection is a stream; delivery to storage without code is Firehose; SQL on files is Athena; dashboards are QuickSight; the warehouse is Redshift. The skill is in wiring them so that each hop does one job, hands over cleanly, and fails in a way you can see.

#Why no single service does it all

Could one service do everything? Not well:

• Kinesis Data Streams collects in real time and keeps records for replay, but it does not write files or answer SQL (lesson 13.5).
• Firehose writes well-sized, compressed, partitioned files without code, but it buffers, so it is near real time, and it cannot be replayed (lesson 13.6).
• S3 stores anything cheaply and durably, but it does not compute.
• Athena answers SQL over S3 with no servers, but it needs someone to start the query and pays by the data it scans (lesson 17.1).
• Redshift is a warehouse for fast joins across business data, and QuickSight draws the charts (lesson 17.2, lesson 17.5).

Each one is excellent at one step and poor at the others. A pipeline is the chain that lets every service do only what it is best at.

#The pipeline at a glance

Read it as three stages: ingest (hops 1–4, data arrives and lands as files), process (hops 5–8, files become reports) and serve (hop 9, people use the reports).

#Hop by hop

Each part below says what the hop does in this pipeline and what to configure. The depth lives in the linked lessons; here the focus is on the hand-over between hops.

#Collect: devices to IoT Core

AWS IoT Core is the managed front door for devices. It was not covered in an earlier lesson, so here is what the exam expects:

• Devices connect with MQTT (also MQTT over WebSockets and HTTPS) and authenticate with an X.509 certificate per device. No AWS access keys sit on a meter that anyone could open with a screwdriver.
• A message is published to a topic, such as factory/gazipur-1/floor-3/meter-2217. The maximum payload is 128 KB.
• The rules engine runs a SQL statement on topics (SELECT * FROM 'factory/+/+/+') and sends matching messages to actions: Kinesis Data Streams, Firehose, S3, Lambda, SQS, SNS, DynamoDB, Timestream and more, up to 10 actions per rule. Each rule can have an error action, for example a backup S3 bucket, so a failed hand-over is not silently lost.
• It is serverless: you pay per million messages and connection minutes.

In the slide’s words, IoT Core lets you harvest data from IoT devices. For the exam, “thousands of sensors”, “MQTT” and “device certificates” point here; the machine learning and device management side of IoT is out of scope.

#Buffer: Kinesis Data Streams

The IoT rule writes each reading to a Kinesis data stream with the meter ID as the partition key, so each meter’s readings stay in order inside one shard. Why a stream here, and not straight to Firehose?

• Real time. A consumer can read a record about 200 ms after it was written (about 70 ms with enhanced fan-out). This is the “collect in real time” requirement.
• Several readers. Firehose is one consumer; an alert function or a Flink application (lesson 17.7) can read the same records at the same time, each at its own pace.
• Replay and a safety buffer. Records stay for the retention period (24 hours by default, up to 365 days). If everything downstream stops, the data waits here.

Sizing for the garments group: 4,000 meters every 10 seconds is 400 records/s of about 0.5 KB, so 200 KB/s. One shard takes 1 MB/s or 1,000 records/s in and gives 2 MB/s out, shared by the standard consumers. One shard would do on paper, but after a factory’s internet outage hundreds of meters send hours of stored readings at once. The team chooses on-demand mode, which scales the shards for it, and keeps the pipeline free of capacity planning.

#Deliver and transform: Data Firehose with Lambda

Amazon Data Firehose (formerly Kinesis Data Firehose) reads the stream and loads it into the ingestion bucket. It does what a hand-written consumer would otherwise have to do: batch, compress, convert, partition, retry and scale.

• Buffering. For S3 the buffer is a size (1–128 MB, default 5 MB) and an interval (0–900 seconds, default 300); whichever is reached first triggers a file. The team sets 128 MB and 300 seconds. At 200 KB/s the interval wins, so a file arrives about every 5 minutes, about 290 files a day.
• Lambda transformation. Firehose sends batches (about 1 MB or 60 seconds by default) to a Lambda function synchronously and waits up to 5 minutes. For every record the function returns the recordId, a result of Ok, Dropped or ProcessingFailed, and the new data. Here it converts watt-hours to kWh, adds the floor name from the meter ID and rejects broken JSON. The slide’s line “Lambda can help Firehose with data transformations” is this feature.
• Format conversion. Firehose converts JSON to Parquet (or ORC) by itself, using a table schema from the Glue Data Catalog (lesson 17.6). Columnar files are what make Athena cheap later.
• Dynamic partitioning. Firehose reads a key from each record (here the reading’s day) and writes to data/day=2026-10-29/. Partition by keys with few values; each value gets its own buffer.
• Errors are kept, not lost. Records that fail the Lambda function go to an error prefix as processing-failed; records that do not fit the schema go there as format-conversion-failed.

#Land: the ingestion bucket

The ingestion bucket (the raw zone of the data lake) holds every reading for as long as the business needs it. Design choices that matter later:

• Prefix layout with Hive-style partitions: data/day=2026-10-29/. Athena can skip whole folders when a query filters on day (lesson 17.1).
• Fewer, bigger files. Tens to hundreds of MB per file is ideal; thousands of tiny files slow every query and raise S3 request costs (lesson 10.6).
• Encryption with SSE-S3 or SSE-KMS; with KMS, the Firehose role needs permission on the key (lesson 21.2).
• Lifecycle rules move old raw data to cheaper storage classes (lesson 10.5); reports, which are small, stay in Standard.
• Two prefixes for two kinds of output: data/ for good files and errors/ for Firehose’s failed records. The event rule in the next hop listens on data/ only.

#Trigger: S3 event, SQS, Lambda

Something must notice a new file and start the query. The classic design uses:

1. An S3 event notification for s3:ObjectCreated:* with the prefix filter data/, sent to an SQS queue (lesson 10.7).
2. A Lambda function with an event source mapping on the queue: Lambda’s pollers read messages in batches and invoke the function (lesson 15.1).
3. The function calls Athena’s StartQueryExecution for the day the file belongs to.

The slide says “Lambda can subscribe to SQS”. Strictly, nothing subscribes to a queue: SQS is pull-based, and the event source mapping polls it for the function. The effect is the same, but the word matters when a question contrasts SNS (push, subscriptions) with SQS (pull).

The slide marks SQS as optional, and it is: S3 can invoke Lambda directly. The queue earns its place because:

• it absorbs bursts: a backfill of 3,000 files becomes 3,000 waiting messages instead of 3,000 simultaneous queries;
• the event source mapping’s maximum concurrency (2–1,000) caps how many copies run, which keeps Athena under its quota of active queries (lesson 15.2);
• failed messages come back after the visibility timeout and, after maxReceiveCount, move to a dead-letter queue you can inspect and redrive (lesson 13.2);
• with partial batch responses (ReportBatchItemFailures) one bad message does not make the whole batch retry.

Set the queue’s visibility timeout to at least six times the function timeout, as lesson 13.2 explains.

#Query: Athena

Amazon Athena runs SQL directly on the files in S3, reading table definitions from the Glue Data Catalog. In this pipeline:

• The function runs a query such as UNLOAD (SELECT floor, date_trunc('hour', ts) AS hour, sum(kwh) … WHERE day = '2026-10-29' GROUP BY 1, 2) TO 's3://meter-reports-example/hourly/day=2026-10-29/' WITH (format = 'PARQUET'). UNLOAD writes the result as Parquet into the reporting bucket; CTAS (CREATE TABLE AS SELECT) does the same and also registers a table.
• Every query also writes its result to the workgroup’s query result location in S3. That is why the slide says Athena results are stored in S3.
• Athena is asynchronous: StartQueryExecution returns a query ID immediately, and GetQueryExecution reports QUEUED, RUNNING, SUCCEEDED or FAILED. A Lambda function should start the query and return, not sit polling for minutes. For long or multi-step reports, Step Functions runs Athena with the .sync integration and waits at no compute cost (lesson 15.7).
• Price: 5 US dollars per TB scanned in most Regions, rounded up to the MB with a 10 MB minimum per query (lesson 17.1 has the details). Parquet, compression and partitions are the levers. A workgroup can cap the data scanned per query as a safety net.
• Quotas: a limited number of active queries per account (running plus queued, varying by Region); beyond it, StartQueryExecution fails with TooManyRequestsException. A query times out after 30 minutes by default (adjustable up to 4 hours).

#Report: the reporting bucket, QuickSight and Redshift

The reporting bucket holds analysed data: small, clean, ready for people. From there:

• QuickSight builds the dashboards. It reads the reports through Athena (direct query, fresh but each view runs a query) or imports them into SPICE, its in-memory engine, on a refresh schedule (fast and cheap per view, as fresh as the last refresh). QuickSight can also read Redshift. Since October 2025 QuickSight is part of Amazon Quick Suite (shortened to Amazon Quick in 2026), where the BI part is called Quick Sight; the API and exam questions still say QuickSight (lesson 17.5).
• Redshift loads the reports with COPY from S3 for joins with orders, salaries and cost tables, or queries them in place with Redshift Spectrum (lesson 17.2).

The slide labels the warehouse Redshift Serverless. That is what keeps requirement 1: a provisioned Redshift cluster has nodes you choose and pay for by the hour, so a pipeline ending in one is not fully serverless. If a question insists on “fully serverless” and offers both, pick Redshift Serverless.

#Real time or near real time: the latency budget

“Real time” in a question usually means milliseconds to about a second; “near real time” means seconds to minutes. Add up the hops of the garments pipeline:

From a meter to the hourly dashboard is therefore about 5–10 minutes: near real time. Only the hot path, a consumer reading the stream, is real time. If a question demands sub-second reaction, the answer is a consumer on Kinesis Data Streams (Lambda, Flink or KCL), never “lower the Firehose buffer” or “query with Athena faster”.

#The cost of each hop

Serverless means you pay for use, so every hop has a cost driver you can push down:

#A worked example of the scan bill

The meters produce 400 readings/s × 0.5 KB = 200 KB/s, about 17 GB of JSON a day and 520 GB a month.

• Raw JSON, no partitions. “kWh per floor yesterday” must read the whole month: 0.52 TB × 5 dollars ≈ 2.60 dollars per query. A dashboard refreshing every 15 minutes runs it 96 times a day: about 250 dollars a day for one chart.
• Parquet, partitioned by day. The query reads one day (about 1.7 GB after conversion and compression) and only three of the twelve columns: about 0.4 GB, a fifth of a cent. Even 96 runs a day cost under 20 cents.
• Reports instead of raw data. QuickSight reads the hourly report (a few MB), not the readings. Viewers cost almost nothing to serve.

The SQL never changed. The layout of the data made it more than 1,000 times cheaper, and that layout is decided upstream, in Firehose.

#What goes wrong

#Designing with the pattern

#Variations you will meet

The classic chain is a starting point. Real systems and exam answers swap hops when the requirement changes:

• No real-time need: IoT Core → Firehose directly → S3. One hop fewer, cheaper, but no hot path and no replay.
• Existing Kafka producers: Amazon MSK instead of Kinesis Data Streams; Firehose can read from MSK too (17.7). MSK provisioned is not serverless; MSK Serverless is.
• Aggregates every minute on the stream: Managed Service for Apache Flink reading the stream, writing results to S3 or another stream.
• Search on the data: a Firehose stream or a second branch into OpenSearch (17.3).
• Warehouse without the S3 hop: Firehose can deliver to Redshift (it stages files in S3 and runs COPY for you). Firehose can also write to Apache Iceberg tables, including S3 Tables, a newer option for transactional data lakes; the exam answer is still the S3 bucket.
• Heavier transformation: an S3 event or schedule starts a Glue job instead of an Athena query; Glue crawlers keep the catalog up to date.
• Reports on a clock, not per file: EventBridge Scheduler starts the Lambda function (or a Step Functions workflow) every hour (19.5); no queue needed.
• Governed access: Lake Formation decides which analysts see which columns and rows of the lake (17.6).

#Zones of a data lake

The two buckets are the smallest version of a common layout: a raw zone (what arrived, never changed), a cleaned zone (validated, Parquet, partitioned) and a curated or reporting zone (aggregates for people). Keeping raw data unchanged means you can always rebuild the rest after a bug.

#Try it yourself

A worked exercise: build a small version of the garments pipeline with the AWS CLI in ap-southeast-1. Names end in -example; the account is 111122223333. The Firehose, IoT and Lambda roles (firehose-meters-role, iot-meters-role, report-starter-role) are assumed to exist with the permissions each step needs.

# 1. The stream, in on-demand mode (no shard planning)
aws kinesis create-stream --stream-name meters-example \
  --stream-mode-details StreamMode=ON_DEMAND

# 2. An IoT rule: every meter topic → the stream, meter ID as partition key,
#    with an error action into a backup bucket
aws iot create-topic-rule --rule-name meters_to_kinesis --topic-rule-payload '{
  "sql": "SELECT *, topic(4) AS meter_id FROM '\''factory/+/+/+'\''",
  "actions": [{"kinesis": {"streamName": "meters-example",
    "partitionKey": "${topic(4)}",
    "roleArn": "arn:aws:iam::111122223333:role/iot-meters-role"}}],
  "errorAction": {"s3": {"bucketName": "meter-iot-errors-example",
    "key": "${timestamp()}", "roleArn": "arn:aws:iam::111122223333:role/iot-meters-role"}}
}'

# 3. Firehose: read the stream, Lambda transform, Parquet, partition by day,
#    errors to errors/ (abridged: the Glue table meters.readings already exists)
aws firehose create-delivery-stream --delivery-stream-name meters-to-s3-example \
  --delivery-stream-type KinesisStreamAsSource \
  --kinesis-stream-source-configuration \
    KinesisStreamARN=arn:aws:kinesis:ap-southeast-1:111122223333:stream/meters-example,RoleARN=arn:aws:iam::111122223333:role/firehose-meters-role \
  --extended-s3-destination-configuration file://firehose-s3.json
# firehose-s3.json sets: BucketARN meter-raw-example, BufferingHints 128 MB / 300 s,
# Prefix data/day=!{partitionKeyFromQuery:day}/, ErrorOutputPrefix errors/!{firehose:error-output-type}/,
# ProcessingConfiguration (Lambda meter-clean-example), DataFormatConversionConfiguration (Parquet),
# DynamicPartitioningConfiguration Enabled=true

# 4. New files under data/ → SQS (the queue policy must allow s3.amazonaws.com)
aws s3api put-bucket-notification-configuration --bucket meter-raw-example \
  --notification-configuration '{"QueueConfigurations": [{
    "QueueArn": "arn:aws:sqs:ap-southeast-1:111122223333:new-files-example",
    "Events": ["s3:ObjectCreated:*"],
    "Filter": {"Key": {"FilterRules": [{"Name": "prefix", "Value": "data/"}]}}}]}'

# 5. Lambda polls the queue: at most 5 copies, partial batch failures
aws lambda create-event-source-mapping --function-name start-report-example \
  --event-source-arn arn:aws:sqs:ap-southeast-1:111122223333:new-files-example \
  --batch-size 10 --scaling-config MaximumConcurrency=5 \
  --function-response-types ReportBatchItemFailures

The starter function only builds the SQL for the file’s day and starts the query:

import boto3, hashlib, json, urllib.parse
athena = boto3.client("athena")

SQL = """UNLOAD (SELECT floor, date_trunc('hour', ts) AS hour, sum(kwh) AS kwh
                 FROM meters.readings WHERE day = '{day}' GROUP BY 1, 2)
TO 's3://meter-reports-example/hourly/day={day}/run={run}/' WITH (format = 'PARQUET')"""

def handler(event, context):
    failures = []
    for msg in event["Records"]:
        try:
            for rec in json.loads(msg["body"]).get("Records", []):
                key = urllib.parse.unquote_plus(rec["s3"]["object"]["key"])
                day = key.split("day=")[1].split("/")[0]          # data/day=2026-10-29/...
                token = hashlib.sha256((key + rec["s3"]["object"]["eTag"]).encode()).hexdigest()[:64]
                athena.start_query_execution(
                    QueryString=SQL.format(day=day, run=token[:12]),
                    ClientRequestToken=token,                       # same file twice → one query
                    WorkGroup="reports-example")
        except Exception:
            failures.append({"itemIdentifier": msg["messageId"]})  # only this one is retried
    return {"batchItemFailures": failures}

Check what happened:

# A query started by the function (ID from its log)
aws athena get-query-execution --query-execution-id 6f1c2b8e-1a2b-4c3d-9e8f-0a1b2c3d4e5f \
  --query 'QueryExecution.[Status.State,Statistics.DataScannedInBytes,Statistics.TotalExecutionTimeInMillis]'
# [
#     "SUCCEEDED",
#     412316860,
#     6420
# ]
# 412,316,860 bytes ≈ 0.38 GB scanned: one partition, three columns

aws s3 ls s3://meter-reports-example/hourly/day=2026-10-29/ --recursive
# 2026-10-30 00:07:12      18734 hourly/day=2026-10-29/run=9c1e5a7b2d40/20261030_000706_00042_abcde_0a1b2c3d.parquet

# Messages that failed three times
aws sqs get-queue-attributes --queue-url https://sqs.ap-southeast-1.amazonaws.com/111122223333/new-files-dlq-example \
  --attribute-names ApproximateNumberOfMessages
# { "Attributes": { "ApproximateNumberOfMessages": "0" } }

Load the reports into Redshift Serverless for finance:

COPY energy.hourly_floor
FROM 's3://meter-reports-example/hourly/day=2026-10-29/'
IAM_ROLE 'arn:aws:iam::111122223333:role/redshift-reports-role'
FORMAT AS PARQUET;

Exercise: a new requirement says “alert the floor engineer within one second”. Which hop do you add, and where? (A Lambda function or Flink application reading the Kinesis stream, not anything after Firehose.) And if the owners later say “we do not need anything faster than an hour”? (Drop the stream: IoT Core → Firehose; replace the SQS trigger with an hourly schedule.)

Clean up: delete the event source mapping, the bucket notification, the Firehose stream, the IoT rule and the Kinesis stream, then empty and delete the buckets.

#Summary

Key takeaways

• The classic serverless pipeline: IoT Core → Kinesis Data Streams → Firehose (+ Lambda) → S3 ingestion bucket → (SQS) → Lambda → Athena → S3 reporting bucket → QuickSight / Redshift. Each hop does one job.
• Real time lives on the stream (about 200 ms to a consumer). From Firehose on, everything is near real time: the S3 buffer is 0–900 s, 300 s by default; “every 1 minute” is the pre-2023 minimum.
• Firehose does the heavy lifting without code: Lambda transformation, Parquet conversion with a Glue schema, dynamic partitioning, and an error prefix for failed records.
• SQS between S3 and Lambda is optional but buys burst absorption, maximum concurrency, retries and a DLQ. Lambda polls the queue; it does not subscribe.
• Athena is asynchronous and priced per TB scanned. Parquet + partitions + reports instead of raw data cut the bill by orders of magnitude. Long work → Step Functions.
• Keep reports in a separate bucket to avoid recursive triggers, and make reports idempotent because events can repeat.
• Fully serverless means Redshift Serverless, not a provisioned cluster; MSK provisioned and EMR on EC2 also break the rule.
• Watch the quiet failures: data freshness, error prefixes, DLQ depth, late data and small files.

Self-check: the serverless big data ingestion pipeline31 questions