Instance Store: Fast and Ephemeral

By the end of this lesson you will be able to

• Explain what an instance store is and why it is much faster than EBS: local NVMe disks in the host, no network hop, millions of IOPS on storage optimised types.
• Recognise which instance types have one (the d in m6id or c6gd, the I and D families) and explain why its size is fixed and it can never be detached, moved or resized.
• Say exactly when instance store data survives (any reboot) and when it is lost (stop, hibernate, terminate, type change, host move, disk failure).
• Protect what you put there: copy out to S3 or EBS, stripe with RAID 0 knowing the risk, and rely on the built-in hardware encryption.
• Pick instance store for caches, buffers, scratch data and replicated datastores, and reject it for the only copy of important data.
• Answer the classic exam questions: highest IOPS for temporary data, data after a reboot, a fast database that must survive stops, and moving storage between instances.

#Why have a disk inside the host at all

Until now every disk in this course has been an EBS volume. An EBS volume is not inside the server your instance runs on. It lives on separate storage servers in the same Availability Zone, and every read and write travels over the AWS network to reach it. That design is what makes EBS so flexible: you can detach a volume, attach it to another instance, snapshot it, resize it, and it keeps its data when the instance stops.

The network hop has a cost, though. Each I/O makes a round trip, and each instance has a cap on how much EBS traffic it can push. For most workloads that does not matter. For a few it matters a lot: a cache that must answer in microseconds, a big data job that shuffles terabytes of temporary files, a NoSQL database doing millions of small random reads.

For those, EC2 offers the instance store: disks that are physically attached to the host computer your instance is running on.

#What an instance store is

An instance store provides temporary block-level storage for an instance. It consists of one or more instance store volumes, each one a real disk (or part of one) in the host machine, exposed to your operating system as a block device.

• Which disks: on current instance types they are NVMe SSDs. The D family and the older H1 use HDDs for cheap, dense capacity. A few very old types (C3, I2, M3, R3, X1) use non-NVMe SSDs.
• How many and how big: fixed by the instance type and size. An m6id.large gets one 118 GB NVMe SSD; an i8g.48xlarge gets twelve 3,750 GB NVMe SSDs (45 TB). You cannot pick another size.
• Price: there is no extra charge. The disks are included in the price of the instance.
• When they appear: only at launch. On instance types with NVMe instance store, every volume is attached automatically at launch. On the old non-NVMe types you had to list them in the block device mapping (as ephemeral0 to ephemeral23). Either way, you cannot add an instance store volume to an instance that is already running.
• State on arrival: raw and empty. Apart from the root volume, nothing is formatted or mounted for you.

#Why it is so fast

Some current storage optimised examples, from the EC2 instance type specifications:

You do not need to memorise these figures. Remember the shape: instance store reaches millions of IOPS on one instance, which EBS cannot, because there is no network in the way and many local disks work in parallel.

#Which instance types have it

Not every instance type has an instance store. You can spot the ones that do:

• A d in the family name (lesson 4.1): m5d, m6id, m7gd, c6gd, r6id… The d means “with local NVMe instance store”. The same type without the d (m6i, c6g) is EBS-only.
• Storage optimised I family: i3, i3en, i4i, i4g, i7i, i7ie, i8g, i8ge. Large NVMe SSDs and the highest random IOPS: NoSQL databases, in-memory databases with local persistence, search engines, data warehousing.
• Storage optimised D family (and the older H1): many large HDDs for the lowest cost per terabyte of local storage: Hadoop/HDFS data nodes, distributed file systems, log processing.
• Some GPU and accelerated types too (for example g4dn, g5, p4d), which use the local disks for training data and checkpoints.

Most other types, such as t3, m7i or c7g, have no instance store at all.

#When the data survives and when it is lost

This is the part the exam loves. One rule explains every row: instance store data lives only as long as your instance stays on the same host and that disk keeps working. A reboot keeps the instance on its host. Anything that releases the host erases the disks.

What actually happens on a stop: EC2 cryptographically erases every block of the instance store by destroying the disk’s encryption keys, then gives the host back to the pool. When you start the instance it usually lands on a different host, with new, empty, unformatted disks.

#Instance store as the root volume

The root volume is the disk the operating system boots from. Almost every instance today boots from an EBS-backed AMI, so its root is an EBS volume. Long ago, AMIs could also be instance store-backed (AWS now calls them Amazon S3-backed AMIs): the root volume was an instance store volume, filled at launch from an image template stored in S3.

For the exam, remember one line: an instance whose root is on instance store cannot be stopped, only rebooted or terminated. (It also cannot be hibernated, as lesson 4.5 showed: hibernation needs an encrypted EBS root.) Windows never supported an instance store root.

#Protecting what you put there

#No snapshots: backups are your job

EBS has snapshots, Data Lifecycle Manager and AWS Backup (lesson 5.1). The instance store has none of these. There is no snapshot API for it, and creating an EBS-backed AMI from the instance does not include instance store data.

If anything on it matters, you copy it somewhere durable:

• To S3 (aws s3 cp, aws s3 sync) for results, checkpoints and backups.
• To an EBS volume attached to the same instance, if it must stay block storage.
• To another node, when the application replicates by itself (the database pattern below).

#RAID 0 across instance store volumes

An instance with several instance store disks shows them as separate devices (nvme1n1, nvme2n1…). To use them as one large, fast volume, stripe them together with RAID 0 (mdadm on Linux). Capacity and throughput add up roughly linearly.

RAID 0 has no redundancy: if one disk fails, the whole array is lost, not just one disk’s share. On an instance store that is usually acceptable, because the data was temporary anyway. If you need redundancy, RAID 1 or RAID 10 halves the usable space, and still does not survive a stop.

#Encryption is built in

On NVMe instance store, the data is encrypted at rest with an XTS-AES-256 block cipher in a hardware module on the instance. The keys are generated by that hardware, are unique to each disk, and are destroyed when the instance stops or terminates. You cannot turn this encryption off, and you cannot bring your own key; it involves no KMS key and needs no setting. (EBS encryption with KMS keys is lesson 5.2.)

#Two performance tips

• SSD instance store has no space reserved for the controller’s housekeeping. AWS recommends leaving about 10% unpartitioned on write-heavy volumes, so write speed stays high as the disk fills.
• On types that support it, run fstrim periodically so the SSD knows which blocks are free.

#How it is used in system design

#Good fits

• Cache: a local cache of objects, images, query results or compiled assets. If it is wiped, the application refills it from the source.
• Buffers and queues of work in progress: data that is about to be processed or sent onwards.
• Scratch and temporary data: video rendering frames, intermediate files of big data jobs (Spark shuffle files, temporary output of Hadoop jobs), sort spill files, build directories. The input and the final result live in S3; only the middle steps live on the instance store.
• Replicated datastores: Cassandra, MongoDB replica sets, Kafka, Elasticsearch, HDFS. The application keeps several copies of every piece of data on different instances (ideally in different racks or AZs, lesson 4.5). When one node’s instance store is lost, the cluster keeps serving from the other copies and rebuilds the lost one on a new node. You get instance store speed, and durability comes from the cluster.
• Fleets of identical servers: for example web servers behind a load balancer that each keep a copy of the same static content, pulled from S3 at boot.

#The anti-pattern

Putting the only copy of anything important on an instance store: the single database server, uploaded user files, the only log of financial transactions. Sooner or later a stop, a type change, a host failure or a Spot interruption will erase it, and there is no snapshot to restore from.

#Instance store or EBS for a fast database?

So the decision is simple:

• A single high-performance database whose data must survive stops and starts → EBS Provisioned IOPS (io2 Block Express; io1 on older designs), however strongly the question stresses performance.
• A distributed database that replicates across nodes and needs the absolute maximum IOPS → instance store is a valid, common choice.
• Temporary data that needs extreme speed (cache, scratch) → instance store. For cheap, large, sequential data such as logs and data warehouses, the EBS answer is st1 (lesson 5.2).

The full three-way decision between EBS, EFS and instance store is in lesson 5.4.

#Try it yourself

# 1. Which instance types have an instance store, how big, SSD or HDD?
aws ec2 describe-instance-types \
  --filters Name=instance-storage-supported,Values=true \
  --query "InstanceTypes[].[InstanceType,InstanceStorageInfo.TotalSizeInGB,InstanceStorageInfo.Disks[0].Count,InstanceStorageInfo.Disks[0].Type]" \
  --output table

# Only the NVMe SSD ones, for one family
aws ec2 describe-instance-types \
  --filters Name=instance-storage-supported,Values=true Name=instance-storage-info.disk.type,Values=ssd \
            Name=instance-type,Values="i8g.*" \
  --query "InstanceTypes[].[InstanceType,InstanceStorageInfo.TotalSizeInGB,InstanceStorageInfo.EncryptionSupport]" \
  --output table

# 2. Launch one with a small instance store (m6id.large: 1 x 118 GB NVMe SSD)
aws ec2 run-instances --image-id $AL2023_AMI --instance-type m6id.large --subnet-id $SUBNET_A

# 3. On the instance: find the disks. The instance store shows the model
#    "Amazon EC2 NVMe Instance Storage"; the root is "Amazon Elastic Block Store".
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,MODEL

# 4. Format and mount (the disk arrives raw)
sudo mkfs.xfs -K /dev/nvme1n1          # -K skips the slow discard on a fresh disk
sudo mkdir -p /scratch
sudo mount /dev/nvme1n1 /scratch
df -h /scratch
# In /etc/fstab use "nofail", or mount from a boot script: after a stop and start the disk is blank again.

# 5. Prove the lifecycle
echo "hello" | sudo tee /scratch/test.txt
sudo reboot                            # reconnect, mount again: test.txt is still there
# aws ec2 stop-instances / start-instances  →  /dev/nvme1n1 is empty and unformatted again

# 6. RAID 0 across several instance store disks (on an i4i.8xlarge: 2 x 3,750 GB)
sudo mdadm --create /dev/md0 --level=0 --raid-devices=2 /dev/nvme1n1 /dev/nvme2n1
sudo mkfs.xfs -K /dev/md0
sudo mkdir -p /data && sudo mount /dev/md0 /data
cat /proc/mdstat                       # one disk failing = the whole array is lost

# 7. Back up what matters: instance store has no snapshots
aws s3 sync /data/results s3://my-results-bucket/run-42/
sudo fstrim -v /data                   # tell the SSDs which blocks are free

#Summary

Key takeaways

• An instance store is block storage on disks physically attached to the host your instance runs on: NVMe SSDs on current types, HDDs on the D family and H1. It is included in the instance price.
• It is the fastest storage on EC2: no network hop, many local disks in parallel, millions of IOPS on storage optimised types (about 7.2 million random read IOPS on an i8g.48xlarge, against 240,000 IOPS from EBS for the same instance).
• It comes only with certain types: a d in the name (m6id, c6gd, r6id), the I and D families. Size and number of disks are fixed by the type; it is available only at launch and can never be detached, moved or resized.
• Data survives any reboot (yours, AWS scheduled, after a power failure). It is lost on stop, hibernate, terminate, instance type change, retirement, automatic recovery to a new host, OS shutdown and disk failure.
• An instance with its root on instance store (an S3-backed AMI, end of life) cannot be stopped.
• No snapshots: copy anything valuable to S3 or EBS yourself. RAID 0 joins disks for speed but loses everything if one disk fails. Data is encrypted at rest by hardware (XTS-AES-256), always on, keys destroyed at stop.
• Use it for cache, buffers, scratch and temporary data, and replicated datastores (Cassandra, MongoDB, Kafka, HDFS). Never for the only copy of important data.
• A high-performance database that must survive stops → EBS io2. Moving storage between instances → EBS.

Self-check: instance store30 questions