Kinesis Data Streams

By the end of this lesson you will be able to

• Explain what streaming data is and how a stream differs from a queue: records stay for a retention period (24 hours to 365 days), many consumers read the same records at their own pace, and any of them can replay.
• Name the producers (SDK, KPL, Kinesis Agent, AWS services) and consumers (custom apps, KCL, Lambda, Data Firehose, Managed Service for Apache Flink) of Kinesis Data Streams.
• Size a provisioned stream from MB/s, records/s and the number of consumers, using the shard limits: 1 MB/s or 1,000 records/s in, 2 MB/s out.
• Predict which shard a record goes to from its partition key, and what Kinesis does and does not promise about ordering and duplicates.
• Diagnose ProvisionedThroughputExceededException and hot shards, and pick the right fix: a better key, retries with backoff, splitting, or enhanced fan-out.
• Reshard with splits, merges and UpdateShardCount, choose between provisioned and on-demand capacity, and secure a stream with IAM, KMS and VPC endpoints.

#Streaming data and Kinesis

#What streaming data is

Some data does not arrive as a few big files; it arrives as a never-ending flow of small events, thousands per second, each one a few hundred bytes to a few kilobytes:

• Clickstreams: every page view, search and tap in a website, app or game.
• IoT telemetry: temperature and soil-moisture readings from sensors in a tea garden in Sylhet, or location pings from delivery bikes.
• Metrics and logs from thousands of servers.
• Business events: every payment in a mobile wallet, every order in a food-delivery app, every ball in a live cricket match.

The value of this data is highest in the first seconds: a fraud check must run before the payment settles, a live score must reach phones before the television broadcast does. Waiting for an hourly batch file is too late. That is real-time big data streaming, and on AWS the core service for it is Amazon Kinesis Data Streams (KDS): a fully managed service that collects and stores streaming data in real time and lets many applications process it.

The Kinesis family once had four members. Today they are: Kinesis Data Streams (this lesson), Amazon Data Firehose (formerly Kinesis Data Firehose, lesson 13.6), Amazon Managed Service for Apache Flink (formerly Kinesis Data Analytics, chapter 17) and Kinesis Video Streams (video, not covered here).

#A queue forgets, a stream remembers

You met SQS in lesson 13.1. A queue is a to-do list: each message is a job, one consumer takes it, does it and deletes it. A stream is different in three ways that the exam tests again and again:

• Records are not deleted when they are read. They stay in the stream for the retention period: 24 hours by default, up to 365 days. You cannot delete an individual record yourself; Kinesis removes it only when it expires. (Data is immutable once written.)
• Many consumers read the same records, each at its own pace. A live-score app, an analytics job and an archiver all read every ball; nobody "takes" a record away from the others.
• Consumers can replay. Each consumer keeps its own position (a sequence number). After a bug fix, a consumer can go back to any point inside the retention period and read again.

#Producers, records and consumers

#What a record is

Everything written to a stream is a record with three parts:

Record size. For years the maximum data blob was 1 MB, and that is still the number in exam questions and the default for every stream. Since October 2025 you can raise a stream’s maximum record size up to 10 MiB (with UpdateMaxRecordSize), for occasional large records such as a big IoT snapshot; the shard limits do not change, so it is not meant for a steady flow of large records. The typical use case is still a lot of small records.

#Producers

• AWS SDK: PutRecord writes one record; PutRecords writes up to 500 records per call (up to 10 MiB per request). PutRecords is not all-or-nothing: some records can fail while others succeed, so check FailedRecordCount and retry only the failed ones.
• Kinesis Producer Library (KPL): a library for high-throughput producers. It batches records into PutRecords calls (collection), packs many small user records into one Kinesis record (aggregation, which saves the 1,000 records/s limit) and retries automatically. The trade-off is a small extra delay while it batches, and consumers must de-aggregate (the KCL and Lambda do it for you).
• Kinesis Agent: a standalone Java program on Linux servers that watches log files and sends new lines to a stream (or to Firehose).
• AWS services that write to a stream for you: CloudWatch Logs subscription filters, AWS IoT Core rules, DMS (database change data capture, CDC), DynamoDB (change data capture into a stream), API Gateway as a front door, and more.

#Consumers

• Custom applications with the SDK (GetShardIterator, then GetRecords in a loop).
• Kinesis Client Library (KCL) applications on EC2, containers or on-premises servers: the standard way to write a scalable consumer (details below).
• AWS Lambda through an event source mapping: no servers, scales per shard.
• Amazon Data Firehose with the stream as its source: delivers the records to S3, Redshift, OpenSearch, Splunk and HTTP endpoints with no consumer code (lesson 13.6).
• Amazon Managed Service for Apache Flink: SQL or Java/Python stream processing (windows, joins, anomaly detection) on the stream (chapter 17).

#Shards: the unit of capacity

A stream is made of one or more shards. A shard is an ordered log of records with a fixed capacity, and the stream’s capacity is the sum of its shards.

#Per-shard limits

Two details catch people out. First, the write limit has two parts: a flood of tiny records hits 1,000 records/s long before 1 MB/s. Second, the 2 MB/s shared read limit is shared by all classic consumers of the shard: three apps that each read everything share it.

#How many shards: the sizing formula

For a provisioned stream, compute the shards each limit needs and take the largest:

write MB/s      = records/s × average record size
shards (MB)     = ceil(write MB/s ÷ 1)
shards (count)  = ceil(records/s ÷ 1,000)
shards (reads)  = ceil(write MB/s × number of shared consumers ÷ 2)
shards needed   = max of the three

Worked examples:

• The notes’ example: an app produces 3 MB/s. Each shard takes 1 MB/s in, so you need at least 3 shards.
• Small records: 5,000 sensor readings per second of 200 bytes each is only about 1 MB/s, but 5,000 records/s needs 5 shards. Aggregating ten readings into one record with the KPL brings it down to 500 records/s and 1 shard.
• Many readers: 2 MB/s written and read by three shared consumers is 6 MB/s of reads: 6 ÷ 2 = 3 shards, although writes alone need 2. Or keep 2 shards and give the consumers enhanced fan-out.

Plan for the peak, not the average, plus some headroom, and remember that a single partition key can never use more than one shard.

#Partition keys and ordering

#How a key picks a shard

The producer never chooses a shard. It sends a partition key, and Kinesis runs the key through an MD5 hash to get a 128-bit number, the hash key. Each shard owns one contiguous range of hash keys, and the record goes to the shard whose range contains its hash. (A producer can override the hash with an explicit hash key, but that is rare.)

The rule to remember: records with the same partition key always go to the same shard (until you reshard). Many different keys spread across all shards, and that spread is what lets a stream use all of its capacity.

#What order Kinesis promises

Inside one shard, records are stored and read in the order they were written (by sequence number). Across shards there is no order: two shards are two separate logs, read in parallel.

So Kinesis guarantees order per shard, which in practice means per partition key, never for the whole stream. Choose the key so that exactly the records that must stay in order share it: all transactions of one wallet, all readings of one sensor, all events of one order. If you really need one global order, you need one key, one shard and therefore the limit of one shard (1 MB/s, 1,000 records/s).

#Retries, reordering and duplicates

Kinesis delivers at least once, and both ends can create surprises:

• Producer retries reorder. If a PutRecords batch has records A then B for the same key and only A fails, retrying A puts it after B. When order matters, send a key’s records in sequence and retry a failure before sending the next one (PutRecord also accepts SequenceNumberForOrdering for strict order from one producer).
• Producer retries duplicate. If a write succeeds but the response is lost (a mobile network drop), the producer retries and the record is stored twice.
• Consumer retries duplicate. A consumer that crashes after processing but before checkpointing reads those records again.

So put a unique ID (transaction ID, event ID) in each record and make consumers idempotent: processing the same record twice must have the same effect as once.

#Hot shards and ProvisionedThroughputExceededException

#What the error means

When a producer writes more than a shard’s 1 MB/s or 1,000 records/s, or consumers read more than its 2 MB/s or 5 GetRecords calls per second, Kinesis rejects the extra requests with ProvisionedThroughputExceededException. The shard is throttled, not broken: it keeps accepting traffic up to its limit, and the rejected requests must be retried. In CloudWatch you see it as WriteProvisionedThroughputExceeded and ReadProvisionedThroughputExceeded.

If the whole stream is simply too small, the fix is more shards (split, or UpdateShardCount). But very often the stream as a whole has spare capacity and only one shard is overloaded.

#The hot shard

Fixes, from best to last resort:

1. Choose a better partition key: many distinct values, none much busier than the others.
2. Add a random suffix to a key you must keep (Bangladesh-0 … Bangladesh-9): the load spreads over ten hash values, but you lose ordering for that key.
3. Retry with exponential backoff and jitter (the SDKs and the KPL do it), so short spikes are absorbed instead of lost.
4. Batch and aggregate with PutRecords and the KPL when the records/s limit is the problem.
5. Split the hot shard, which helps only when the shard holds many busy keys. A single giant key is one point in the hash range: splitting puts it in one child, and that child is just as hot.
6. For read throttling, use enhanced fan-out, or fewer consumers polling the same shard.

How to spot it: throttling while IncomingBytes for the whole stream is well below capacity. Turn on enhanced (shard-level) monitoring to see the metrics per shard and find the hot one.

#Resharding: split and merge

Traffic is not constant. Changing the number of shards of a provisioned stream is called resharding:

• Shard split (scale out). One parent shard is split into two children at a new starting hash key; each child gets part of the parent’s hash range. Capacity for those keys doubles, and so does the hourly cost for that part of the stream.
• Shard merge (scale in). Two adjacent shards (whose hash ranges touch) are merged into one child. Capacity and cost go down. You cannot merge two shards whose ranges are not next to each other.

What happens to the parents: they move from open to closed (no new writes; new records go to the children) and finally to expired once the retention period has passed. Until then their old records stay readable. That is why ordering needs care: after a split, a key’s older records are in the parent and its newer ones in a child. A consumer must finish the parent before reading the children; the KCL does this for you.

Instead of single splits and merges, you can call UpdateShardCount with a target and let Kinesis do the work (uniform scaling). Its default limits per stream: at most double or half the current count in one call, at most 10 scaling operations in a rolling 24 hours, and up to 10,000 shards per stream. Producers and consumers keep working while the stream is UPDATING.

Resharding in provisioned mode is your job: manually in the console, from a script, or with your own automation (a CloudWatch alarm that calls a Lambda function which calls UpdateShardCount). On-demand mode does it for you.

#Capacity modes: provisioned and on-demand

The notes describe resharding as manual. That is true for provisioned mode only; since 2021 there is also on-demand mode.

You can switch a stream between the two modes twice in 24 hours, without downtime. Even in on-demand mode, one partition key is still limited to one shard’s 1 MB/s and 1,000 records/s, so a hot key still throttles. On-demand mode splits shards evenly when traffic grows but does not isolate one busy hash range, so for highly uneven keys AWS recommends provisioned mode, where you can split exactly the shard that is hot. By default an account can have up to 50 on-demand streams (adjustable); provisioned streams have no such count limit.

Retention costs extra beyond the free 24 hours: extended retention up to 7 days is billed per shard-hour (provisioned) or per GB, and long-term retention beyond 7 days, up to 365 days, per GB-month, plus a per-GB charge when long-term data is read with GetRecords.

#Consumers in depth

#Shared throughput and enhanced fan-out

Rule of thumb: one or two consumers that can tolerate a little delay → shared; three or more consumers, or a consumer that needs the lowest latency → enhanced fan-out.

#The Kinesis Client Library (KCL)

Reading a stream with raw GetRecords calls means you must track shards, positions, failures and resharding yourself. The KCL does all of that:

• It gives each shard to exactly one worker at a time through a lease; one worker can hold several leases.
• Leases and checkpoints (the sequence number each shard was processed up to) live in a DynamoDB lease table, one per KCL application. That is why a KCL app needs DynamoDB permissions and adds a small DynamoDB cost.
• When a worker dies, another worker takes its leases and continues from the last checkpoint. Records processed after that checkpoint are processed again, so processing must be idempotent.
• It reads parent shards before child shards after resharding, keeping per-key order.
• It uses enhanced fan-out by default from version 2 on, and KCL 3 balances leases by worker CPU use.
• It is a Java library; other languages (Python, Node.js, .NET, Ruby) use it through the MultiLangDaemon.

Because one shard is read by one worker, more workers than shards is useless: with 4 shards, a fifth worker holds no lease. To process faster, add shards (or, with Lambda, raise the parallelization factor).

#Lambda as a consumer

A Lambda event source mapping polls each shard (or subscribes with enhanced fan-out) and invokes your function with a batch of records: by default up to 100 records, configurable up to 10,000, with an optional batching window of up to 5 minutes.

• Batches from one shard are processed one after another, so Lambda keeps per-shard order. Concurrency is one invocation per shard by default; a parallelization factor of up to 10 allows up to 10 concurrent batches per shard while Lambda still keeps the order of each partition key.
• If the function throws, Lambda by default retries the whole batch until it succeeds or the records expire, and that shard is blocked meanwhile. A single bad record (a poison pill) can stop a shard for a day. Fix it with bisect batch on error, a maximum retry attempts value, a maximum record age, and an on-failure destination (SQS, SNS or S3) that receives details of the failed batch. Lambda event source mappings are covered in depth in chapter 15.

#Firehose, Flink and MSK

• Data Firehose reading from the stream is the no-code way to land the same records in S3, Redshift or OpenSearch, in batches (near-real-time). A common design is a stream for real-time consumers plus Firehose as one more consumer for the archive. Lesson 13.6 compares Firehose and Data Streams in depth.
• Managed Service for Apache Flink runs continuous queries and stateful processing on the stream.
• Amazon MSK (managed Apache Kafka) is the alternative streaming service when a team already uses Kafka’s APIs and tools; it is a different service, not a Kinesis consumer. Both are covered in chapter 17.

#Security

• Access control with IAM. Producers need kinesis:PutRecord/PutRecords on the stream ARN; consumers need GetRecords, GetShardIterator, DescribeStream/ListShards (and SubscribeToShard for enhanced fan-out). Give EC2, Lambda and containers an IAM role, never access keys in code (chapter 2).
• Resource-based policies (since 2023) can be attached to a stream or a consumer to allow another account to write or read directly, like an S3 bucket policy; the classic alternative is a cross-account IAM role.
• Encryption in flight: every API call uses HTTPS (TLS).
• Encryption at rest: server-side encryption with KMS, using the AWS managed key aws/kinesis or your own customer managed key. Producers and consumers then also need permission to use the key (lesson 10.8 explains KMS keys).
• Client-side encryption is possible too: you encrypt before PutRecord and decrypt in the consumer.
• Private access: an interface VPC endpoint (PrivateLink) lets producers and consumers in private subnets reach Kinesis without an internet gateway or NAT gateway (chapter 22).
• Audit: control-plane calls (create, delete, reshard, encryption changes) are logged in CloudTrail.

#What goes wrong

• Throttling with spare capacity: a hot partition key. Fix the key, not the shard count.
• Throttling when the stream is simply too small: add shards (split, UpdateShardCount) or switch to on-demand.
• Read throttling and rising iterator age with many polling consumers: move them to enhanced fan-out.
• A consumer falls behind and data expires: retention is too short or the consumer was down longer than it. Alarm on GetRecords.IteratorAgeMilliseconds (or the Lambda IteratorAge metric) and raise retention.
• A Lambda consumer stuck on one bad record: configure bisect on error, maximum retries, maximum record age and an on-failure destination.
• Out-of-order records after resharding: a custom consumer reads children before finishing the parent. Use the KCL.
• Out-of-order or duplicate records after retries: partial PutRecords failures retried out of order, lost responses retried. Send a key’s records in sequence and make consumers idempotent.
• Too many KCL workers do nothing: workers beyond the shard count hold no lease.
• AccessDeniedException on an encrypted stream: the role can use Kinesis but not the KMS key.
• Surprise bill: shards left provisioned for a peak that ended, long retention, enhanced fan-out consumers nobody uses.

#Kinesis Data Streams in system design

Common patterns:

• Real-time analytics + archive: producers → Kinesis Data Streams → (Lambda / KCL / Flink for real time) + (Firehose → S3 for the data lake).
• Event sourcing and ordered processing per entity: partition key = account ID, order ID or device ID.
• Log and metric aggregation: CloudWatch Logs subscription → stream → processing → OpenSearch.
• Change data capture: DMS or DynamoDB → stream → consumers that keep search indexes and caches in sync.
• Buffering a firehose of events in front of slow systems: unlike SQS, the stream keeps the data even after the slow system has read it, so a second consumer can be added later without changing producers.

When Kinesis is the wrong choice: independent jobs that each need to be done once by any worker (SQS), pushing notifications to people and endpoints (SNS), loading data into S3 or Redshift with no custom code and no need for replay or sub-second latency (Firehose on its own).

#Try it yourself

These commands use the AWS CLI v2. Kinesis charges per shard-hour (or per stream-hour on demand), so delete the stream at the end. In the CLI v2, --data is base64 unless you add --cli-binary-format raw-in-base64-out.

# 1. Create a provisioned stream with 2 shards, then wait until it is ACTIVE
aws kinesis create-stream --stream-name scores-example --shard-count 2
aws kinesis wait stream-exists --stream-name scores-example
aws kinesis describe-stream-summary --stream-name scores-example
# {
#   "StreamDescriptionSummary": {
#     "StreamName": "scores-example",
#     "StreamARN": "arn:aws:kinesis:ap-southeast-1:111122223333:stream/scores-example",
#     "StreamStatus": "ACTIVE",
#     "StreamModeDetails": { "StreamMode": "PROVISIONED" },
#     "RetentionPeriodHours": 24,
#     "OpenShardCount": 2,
#     "EncryptionType": "NONE", ...
#   }
# }

# 2. Write a record; the same partition key always lands in the same shard
aws kinesis put-record --stream-name scores-example --partition-key match-77 \
  --data '{"over":41.1,"runs":4}' --cli-binary-format raw-in-base64-out
# { "ShardId": "shardId-000000000001",
#   "SequenceNumber": "49650925471223049381123456789012345678901234567890" }

# 3. Write a batch and check for partial failures
aws kinesis put-records --stream-name scores-example --cli-binary-format raw-in-base64-out \
  --records Data='{"over":41.2}',PartitionKey=match-77 Data='{"over":41.3}',PartitionKey=match-77
# { "FailedRecordCount": 0, "Records": [ { "SequenceNumber": "...", "ShardId": "shardId-000000000001" }, ... ] }

# 4. See the shards and their hash key ranges
aws kinesis list-shards --stream-name scores-example \
  --query 'Shards[].[ShardId,HashKeyRange.StartingHashKey,HashKeyRange.EndingHashKey]' --output table

# 5. Read from the oldest record still in the shard (TRIM_HORIZON)
IT=$(aws kinesis get-shard-iterator --stream-name scores-example \
  --shard-id shardId-000000000001 --shard-iterator-type TRIM_HORIZON \
  --query ShardIterator --output text)
aws kinesis get-records --shard-iterator "$IT" --limit 10
# { "Records": [ { "SequenceNumber": "...", "PartitionKey": "match-77",
#                  "Data": "eyJvdmVyIjo0MS4xLCJydW5zIjo0fQ==", ... } ],
#   "NextShardIterator": "...", "MillisBehindLatest": 0 }
# Data is base64: echo eyJvdmVyIjo0MS4xLCJydW5zIjo0fQ== | base64 -d  →  {"over":41.1,"runs":4}

# 6. Replay from a point in time instead
aws kinesis get-shard-iterator --stream-name scores-example --shard-id shardId-000000000001 \
  --shard-iterator-type AT_TIMESTAMP --timestamp 2026-10-26T09:00:00Z

# 7. Reshard: double the shards (uniform scaling), or split one shard yourself
aws kinesis update-shard-count --stream-name scores-example --target-shard-count 4 \
  --scaling-type UNIFORM_SCALING
# { "StreamName": "scores-example", "CurrentShardCount": 2, "TargetShardCount": 4 }

# 8. Keep data 7 days, encrypt with KMS, register an enhanced fan-out consumer
aws kinesis increase-stream-retention-period --stream-name scores-example --retention-period-hours 168
aws kinesis start-stream-encryption --stream-name scores-example \
  --encryption-type KMS --key-id alias/aws/kinesis
aws kinesis register-stream-consumer --consumer-name live-app \
  --stream-arn arn:aws:kinesis:ap-southeast-1:111122223333:stream/scores-example

# 9. Switch to on-demand (allowed twice per 24 hours), then clean up
aws kinesis update-stream-mode --stream-mode-details StreamMode=ON_DEMAND \
  --stream-arn arn:aws:kinesis:ap-southeast-1:111122223333:stream/scores-example
aws kinesis delete-stream --stream-name scores-example --enforce-consumer-deletion

Also try in the console: open the stream’s Monitoring tab while you send records in a loop with one partition key, and watch one shard’s incoming bytes rise (enable enhanced shard-level metrics first). The Data viewer tab shows records per shard without writing code.

#Summary

Key takeaways

• Kinesis Data Streams collects and stores real-time streaming data (clickstreams, IoT, logs, events). Unlike SQS, records are not deleted when read: they stay for the retention period (24 hours default, up to 365 days), many consumers read the same data at their own pace, and they can replay. Records cannot be deleted before they expire.
• A record = partition key + data blob + sequence number. Max data 1 MB (the exam number and the default; configurable up to 10 MiB since 2025).
• Producers: SDK (PutRecord, PutRecords up to 500), KPL (batching, aggregation, retries), Kinesis Agent, AWS services. Consumers: SDK, KCL, Lambda, Data Firehose, Managed Service for Apache Flink.
• Shard limits: 1 MB/s or 1,000 records/s in, 2 MB/s out (shared by classic consumers, 5 GetRecords/s) or 2 MB/s per consumer with enhanced fan-out (push, about 70 ms). Shards needed = max(MB/s ÷ 1, records/s ÷ 1,000, shared reads ÷ 2). 3 MB/s → 3 shards.
• Partition key → MD5 → hash range → shard. Same key, same shard. Order is guaranteed per shard only, never for the whole stream. Delivery is at least once: make consumers idempotent.
• ProvisionedThroughputExceededException = over a shard’s limits; the shard is throttled, not crashed. Stream too small → add shards; hot shard (country as key, 95% Bangladesh) → use a high-cardinality key (user ID, transaction ID); retry with backoff. Splitting does not help a single giant key.
• Resharding: split (1 → 2, more capacity and cost), merge (2 adjacent → 1). Parents become closed, then expire; read parents before children (the KCL does). UpdateShardCount: ×2 or ÷2 per call, 10 times per 24 hours.
• Capacity modes: provisioned (you pick shards, pay per shard-hour, scale manually) vs on-demand (no planning, starts at 4 MB/s, scales to double the 30-day peak, pay per stream-hour and per GB). Switch twice per 24 hours.
• KCL: one worker per shard through leases, checkpoints in DynamoDB, no use for more workers than shards. Lambda: per-shard order, parallelization factor up to 10, bisect on error and on-failure destination against poison records.
• Security: IAM (roles), resource-based policies for cross-account, HTTPS in flight, KMS at rest, interface VPC endpoint for private access.

Self-check: Kinesis Data Streams32 questions