Amazon EMR

By the end of this lesson you will be able to

• Explain what Amazon EMR (Elastic MapReduce) is, why big data needs a cluster at all, and which open-source frameworks it runs: Spark, Hadoop, Hive, HBase, Presto/Trino, Flink and more.
• Describe the three node types, primary, core and task, what each one runs, and what happens to the cluster and its data when each one is lost.
• Choose a purchasing option (On-Demand, Reserved, Savings Plans, Spot) for each node type, for long-running and transient clusters, and answer the classic exam question about Spot task nodes.
• Explain why HDFS is ephemeral and why input and output belong in S3 through EMRFS, and size HDFS with the replication factor.
• Compare uniform instance groups with instance fleets, and managed scaling with custom automatic scaling.
• Pick between EMR on EC2, EMR on EKS and EMR Serverless, and place EMR next to Athena, Redshift and Glue in a design.

#Why big data needs a cluster

Chapter 16 sorted databases by the questions they answer. Some questions are too big for any single machine. A mobile operator in Bangladesh writes a call-detail record (CDR) for every call, SMS and data session: billions of rows and several terabytes a night. Finance wants minutes per tower and per package by 6 am; the network team wants dropped calls per area; the fraud team wants patterns of illegal call bypass. One server, however large, would read those terabytes for many hours.

The answer, since Google’s papers in the 2000s, is to split the work over many ordinary machines. The programming model is MapReduce: cut the input into pieces, process every piece in parallel on its own machine (map), then combine the partial results (reduce). Apache Hadoop is the open-source implementation, with its own distributed file system (HDFS) and resource manager (YARN). Apache Spark later did the same work mostly in memory and much faster, and is what most teams write today.

Running such a cluster yourself is hard work: buy or rent dozens of servers, install matching versions of Hadoop, Spark and Hive, configure them to talk to each other, replace failed disks, patch them, and pay for them even when nothing runs.

#What Amazon EMR is

Amazon EMR (originally Elastic MapReduce) is AWS’s managed big-data platform. You say which frameworks and how much capacity you want; EMR provisions the EC2 instances, installs and configures the frameworks, wires them together, watches node health and tears it all down when you are done. A cluster can be a handful of instances or hundreds, even thousands, of EC2 instances.

EMR comes bundled with the main open-source big-data frameworks, at versions AWS has tested together in each EMR release (for example emr-7.x):

EMR also brings auto scaling (managed scaling) and deep integration with Spot Instances, and its own performance-tuned runtime for Spark and Hive. You still write the jobs; EMR runs the platform.

Use cases (the slides’ list, with what each looks like):

• Data processing and ETL: clean and reshape terabytes of logs, clickstream or CDRs into Parquet for Athena or Redshift.
• Machine learning: feature engineering and training on data too large for one machine, with Spark’s ML library (managed ML platforms are chapter 18).
• Web indexing: crawling and building search indexes over billions of pages, the job Hadoop was invented for.
• Big data analytics: large joins and aggregations, risk models, genomics, interactive SQL with Trino.

The exam keywords are simple: “Hadoop”, “Spark”, “Hive”, “HBase”, “big data framework”, “migrate an on-premises Hadoop cluster” → EMR.

#Inside a cluster: primary, core and task nodes

Every EMR cluster on EC2 has up to three kinds of node, each an EC2 instance. The notes mark this as very important for the exam.

#Primary node (formerly master node)

The primary node manages the cluster. It runs the coordinating parts of the frameworks: the YARN ResourceManager, which decides which job gets how much CPU and memory, and the HDFS NameNode, which knows which block of which file is on which machine. It tracks the status of submitted jobs and monitors the health of the other nodes. You can SSH to it to read logs and open the frameworks’ web interfaces.

AWS renamed “master node” to primary node; older questions and the notes still say master. A cluster has one primary node, or three for high availability (since EMR 5.23): if the active primary fails, EMR fails over to a standby and replaces the failed one. The primary does not need much power; m5.xlarge is enough for clusters up to about fifty nodes.

The notes say the primary “broadcasts jobs”. More precisely, it coordinates: it decides where work and data go, while the actual processing happens on core and task nodes.

#Core nodes

Core nodes do two jobs: they run tasks (YARN NodeManager, Spark executors, MapReduce tasks) and store data in HDFS through the DataNode service, on their instance store and EBS volumes. Because of the second job they are long-running for the life of the cluster: removing one means HDFS must first move its blocks elsewhere, and losing one suddenly can lose data. A cluster has one core instance group (or fleet), which can contain many instances.

#Task nodes (optional)

Task nodes only run tasks. They hold no HDFS data, only temporary files. They exist to add compute power when you need it: add twenty task nodes for a heavy night, remove them in the morning. With uniform instance groups you can have up to 48 task instance groups, each with its own instance type and purchasing option. Because losing one costs only time, task nodes are usually Spot.

Play the scenarios below: a normal job, then losing a Spot task node, a core node, the single primary, and one of three primaries.

When a Spot task node disappears, the running pieces of work are simply scheduled again. What must not disappear is the application master, the per-job coordinator that YARN starts somewhere in the cluster. EMR 5.x kept application masters on core nodes by default; from EMR 6.x this YARN node-label setting is off by default, and on EMR 7.x you can label nodes by market type and keep application masters on On-Demand (or core) nodes. If you run many Spot task nodes, turn it on.

#HDFS replication and capacity

HDFS keeps several copies of every block on different core nodes, so one failed disk does not lose data. EMR sets the default replication factor by the number of core nodes:

Usable HDFS capacity = (storage per core node × number of core nodes) ÷ replication factor. Ten core nodes with 800 GB of local storage each hold about 8,000 ÷ 3 ≈ 2,666 GB of data. Remember that input, intermediate and output data all have to fit while a job runs. If HDFS is too small: more or bigger core nodes, more EBS volumes, compression, or (best) keep the data in S3.

HDFS is rack-aware, which is why lesson 4.5 pairs Hadoop with partition placement groups.

#Purchasing options for each node type

EMR nodes are ordinary EC2 instances, so the EC2 purchasing options of lesson 4.4 apply. On top of the EC2 price you pay an EMR fee per instance-hour (for an m5.xlarge in us-east-1, $0.192 for EC2 On-Demand plus $0.048 for EMR), billed per second with a one-minute minimum. The EMR fee is the same whether the instance is On-Demand or Spot.

• On-Demand: reliable and predictable; AWS will not take the instance back. The default for anything that must not disappear.
• Reserved Instances (minimum 1 year) and Savings Plans: big discounts for capacity you run all year. Reserved Instances are a billing discount, so EMR uses them automatically when a cluster launches matching instances; there is nothing to configure in EMR. For a guaranteed slot in a specific AZ, EMR instance fleets can also use On-Demand Capacity Reservations.
• Spot Instances: up to about 90% cheaper, but EC2 can reclaim them with a two-minute notice. Cheap and less reliable: perfect for work that can be redone.

You cannot change a group’s purchasing option on a running cluster: for primary and core you must launch a new cluster; for task nodes you add a new task group or fleet and remove the old one.

Now build the cluster yourself. Pick the purchasing option for each node group and where the data lives, decide whether Spot is reclaimed two hours in, and compare what survives and what the night costs.

#Long-running and transient clusters

The slides end with a choice that changes everything else:

• A transient (temporary) cluster is created for one batch of work, runs its steps, and terminates itself when the last step finishes. You pay only for the hours of work. Data must live in S3, because the cluster’s disks disappear. This is the right model for nightly ETL, monthly reports and one-off backfills.
• A long-running (persistent) cluster stays up for many users and jobs: interactive notebooks in EMR Studio, ad-hoc Hive and Trino queries, an HBase table that applications read all day. Turn on termination protection, cover the baseline with Reserved Instances or Savings Plans, and let scaling follow the load.

Transient clusters take several minutes to start (instances, bootstrap actions, framework installation), so they suit jobs of an hour or more. For anything that must start in seconds, use a long-running cluster or EMR Serverless with pre-initialized capacity (below).

A step is a unit of work submitted to the cluster: a Spark application, a Hive script, a custom JAR. Each step has an action on failure: continue, cancel the remaining steps and wait, or terminate the cluster. Cluster logs can be pushed to an S3 log URI, so you can still read them after a transient cluster is gone. An auto-termination policy ends a cluster after the idle time you choose: the cure for the test cluster someone forgot over Eid.

#Storage: HDFS or S3 through EMRFS

An EMR cluster can store data in two very different places.

• HDFS lives on the core nodes’ instance store and EBS volumes. It is fast, local and replicated across core nodes, but ephemeral: when the cluster terminates, it is deleted (like the instance store of lesson 5.3). It is also sized by your core nodes.
• EMRFS (the EMR File System) lets Hadoop, Spark and Hive use S3 as a file system with s3:// paths. Data written there is persistent: it survives the cluster, can be read by the next cluster, by Athena or Redshift Spectrum, and is stored with S3’s eleven nines of durability. Since December 2020 S3 is strongly consistent, so the old “EMRFS consistent view” workaround (a DynamoDB table tracking S3 listings) is no longer needed; it reached end of standard support in 2023. From EMR 7.10 the default S3 connector is the open-source S3A; the idea, S3 as the cluster’s durable storage, is the same.

The modern pattern: input and output in S3, HDFS (or local disk) only for intermediate data such as Spark shuffle files. That gives you separation of storage and compute: clusters become disposable, any size or EMR release can read the same data, transient clusters cost nothing when idle, and an AZ outage or a reclaimed core node loses only scratch data. HBase can also keep its data in S3 (HBase on S3), so even a database cluster can be replaced.

#Instance groups and instance fleets

When you create a cluster you choose one of two ways to describe its instances, for the whole cluster:

Fleets are the better default for Spot-heavy clusters: diversifying across instance types and pools is the single most effective way to get Spot capacity and fewer interruptions (the same advice as Spot Fleet in lesson 4.4).

#Scaling: managed scaling and custom automatic scaling

• EMR managed scaling (EMR 5.30 and later) watches the cluster’s workload and adds or removes core and task capacity by itself. You set a minimum and a maximum, and optionally an On-Demand limit (capacity above it is Spot) and a maximum core limit (capacity above it is task nodes). The primary never scales. It works with instance groups and fleets, scales in carefully (it avoids nodes holding active Spark shuffle data or an application master), but understands only YARN applications such as Spark, Hive, Hadoop and Flink, not Presto/Trino or HBase.
• Custom automatic scaling policies are rules on CloudWatch metrics (for example YARN memory available below 15% → add two task nodes). They work only with instance groups, and are the option for non-YARN workloads.

Grow and shrink with task nodes and keep the core group steady: shrinking a core node means HDFS has to decommission it and copy its blocks first.

#Deployment options: EMR on EC2, EMR on EKS, EMR Serverless

Everything above is EMR on EC2, the original and still the most flexible model. Two newer options run the same EMR runtime without EC2 node groups to manage.

EMR Serverless in practice: create an application (a release label and a type, Spark or Hive), then submit job runs with an IAM runtime role that grants access to S3. The application starts automatically on the first job and stops after 15 idle minutes by default. Pre-initialized capacity keeps a warm pool of workers so jobs start in seconds (you pay for it while the application runs), and maximum capacity caps vCPU, memory and disk so a runaway job cannot run up the bill.

EMR on EKS suits a company that already runs everything on Amazon EKS (14.4): Spark jobs become pods next to the microservices, on EC2 (including Spot) or Fargate nodes, and share idle capacity.

#Security in one page

• Network: a cluster lives in a VPC subnet, best a private subnet with a NAT gateway or VPC endpoints for S3 and other services. EMR creates EMR-managed security groups for the primary and the other nodes; add your own for SSH or notebooks.
• IAM: three roles. The service role lets EMR create EC2 instances and other resources; the EC2 instance profile is what the cluster’s jobs use to read S3 and other services (on EMR Serverless and EMR on EKS, the runtime or execution role per job); and IAM policies decide which users may create clusters or submit work.
• Encryption is configured once in a security configuration: at rest for S3 data written through EMRFS (SSE-S3, SSE-KMS or client-side), for local disks and EBS volumes, and in transit with TLS between nodes. KMS keys are the security chapter’s KMS lesson.
• Fine-grained access: Kerberos for user authentication, and Lake Formation for table-, column- and row-level permissions on data in S3 (lesson 17.6).
• Catalog: point Hive and Spark at the Glue Data Catalog as their metastore, so table definitions outlive transient clusters and are shared with Athena and Redshift Spectrum.

#What goes wrong

• Results vanish when a transient cluster ends. They were written to HDFS. Write output to S3, or add a step that copies it (s3-dist-cp) before termination.
• The whole cluster dies at random. A single primary node on Spot was reclaimed. Keep it On-Demand, or use three primaries for long-running clusters.
• Jobs fail with missing HDFS blocks. Core nodes were on Spot, or there were three or fewer core nodes with replication factor 1. Core On-Demand, more core nodes, data in S3.
• Jobs fail when Spot task nodes go. The application master ran on a Spot node. Use YARN node labels to keep application masters on core or On-Demand nodes.
• A Spot core group never starts. Uniform groups wait until every requested Spot instance is available. Use instance fleets with several instance types and a timeout that switches to On-Demand.
• HDFS fills up mid-job. Input, intermediate and output must fit after dividing by the replication factor. Add core capacity or move input and output to S3.
• Managed scaling does nothing for Trino or HBase. It only scales YARN applications; use custom automatic scaling on instance groups.
• An AZ outage stops the night’s job. A cluster lives in one AZ. Keep data in S3 and relaunch in another AZ (fleets with several subnets make this automatic at launch).
• The bill keeps running. A forgotten test cluster or a step that failed without terminating the cluster. Auto-termination policy, “terminate cluster” on step failure, budgets and alarms (chapter 19, CloudWatch).
• Cluster creation fails with an EC2 limit. On-Demand vCPU quotas per Region apply to EMR too; request an increase before the big night.

#EMR in system design

Common patterns:

• Transient ETL cluster per run, orchestrated by Step Functions or Amazon Managed Workflows for Apache Airflow, data in S3, catalog in Glue. The default for batch.
• Long-running shared cluster for interactive analytics and notebooks, managed scaling with Spot task nodes, Reserved or Savings Plans for the baseline, three primaries.
• HBase on S3 for a large, read-heavy key-value store that must survive cluster replacement.
• Lift-and-shift Hadoop: an on-premises Hadoop cluster moves to EMR with the same Hive, Spark and HBase jobs; HDFS data goes to S3 (with DataSync or Snowball for very large volumes, chapter 12).
• Serverless Spark: EMR Serverless for teams that only want to submit jobs; Glue when they also want crawlers, catalog and job bookmarks.
• Streaming: Spark Structured Streaming or Flink on EMR reading Kinesis or MSK; the fully managed alternative is Managed Service for Apache Flink (17.7), and the full serverless pipeline is lesson 17.8.

How to choose quickly:

#Try it yourself

These commands create real resources that cost money; run them in a sandbox account and terminate everything afterwards.

1. Create the default roles (once per account):

aws emr create-default-roles

[
  { "RoleName": "EMR_DefaultRole", "Arn": "arn:aws:iam::111122223333:role/EMR_DefaultRole", ... },
  { "RoleName": "EMR_EC2_DefaultRole", "Arn": "arn:aws:iam::111122223333:role/EMR_EC2_DefaultRole", ... }
]

2. Launch a transient cluster with uniform instance groups: primary and core On-Demand, task Spot, a Spark step, logs in S3, terminate when done.

aws emr create-cluster \
  --name "cdr-nightly-2026-10-29" \
  --release-label emr-7.12.0 \
  --applications Name=Spark Name=Hadoop \
  --service-role EMR_DefaultRole \
  --ec2-attributes InstanceProfile=EMR_EC2_DefaultRole,SubnetId=subnet-0a1b2c3d4e5f67890 \
  --instance-groups \
    InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m5.xlarge \
    InstanceGroupType=CORE,InstanceCount=4,InstanceType=m5.xlarge \
    InstanceGroupType=TASK,InstanceCount=8,InstanceType=m5.xlarge,Market=SPOT \
  --log-uri s3://cdr-logs-example/emr/ \
  --steps Type=Spark,Name=aggregate,ActionOnFailure=TERMINATE_CLUSTER,\
Args=[s3://cdr-code-example/aggregate.py,--date,2026-10-29] \
  --auto-terminate

{
    "ClusterId": "j-2AXXXXXXGAPLF",
    "ClusterArn": "arn:aws:elasticmapreduce:ap-southeast-1:111122223333:cluster/j-2AXXXXXXGAPLF"
}

(Use the newest release your Region offers; aws emr list-release-labels lists them.)

3. Watch it and list its instances:

aws emr describe-cluster --cluster-id j-2AXXXXXXGAPLF \
  --query 'Cluster.{State:Status.State,AZ:Ec2InstanceAttributes.Ec2AvailabilityZone}'
aws emr list-instances --cluster-id j-2AXXXXXXGAPLF --instance-group-types TASK \
  --query 'Instances[].{Id:Ec2InstanceId,Market:Market,State:Status.State}' --output table

{ "State": "RUNNING", "AZ": "ap-southeast-1a" }
-------------------------------------------------
|                 ListInstances                 |
+----------------------+---------+--------------+
|          Id          | Market  |    State     |
+----------------------+---------+--------------+
|  i-0f1e2d3c4b5a69788 |  SPOT   |  RUNNING     |
|  i-0a9b8c7d6e5f43210 |  SPOT   |  RUNNING     |
|  ...                 |  ...    |  ...         |
+----------------------+---------+--------------+

Note the single Availability Zone.

4. On a long-running cluster, add a managed scaling policy (minimum 2, maximum 20, at most 4 On-Demand, at most 4 core) and a step:

aws emr put-managed-scaling-policy --cluster-id j-3BYYYYYYHBQMG \
  --managed-scaling-policy ComputeLimits='{UnitType=Instances,MinimumCapacityUnits=2,MaximumCapacityUnits=20,MaximumOnDemandCapacityUnits=4,MaximumCoreCapacityUnits=4}'

aws emr add-steps --cluster-id j-3BYYYYYYHBQMG \
  --steps Type=Hive,Name=daily-towers,ActionOnFailure=CONTINUE,\
Args=[-f,s3://cdr-code-example/towers.q]

{ "StepIds": [ "s-1K2L3M4N5O6P7" ] }

5. Terminate (a long-running cluster with termination protection needs it turned off first):

aws emr modify-cluster-attributes --cluster-id j-3BYYYYYYHBQMG --no-termination-protected
aws emr terminate-clusters --cluster-ids j-3BYYYYYYHBQMG

6. The same job on EMR Serverless: create an application with a maximum capacity, run a job, check it.

aws emr-serverless create-application \
  --name cdr-spark --type SPARK --release-label emr-7.12.0 \
  --maximum-capacity '{"cpu":"100vCPU","memory":"800GB"}'

{ "applicationId": "00fabcd1234efgh5", "name": "cdr-spark",
  "arn": "arn:aws:emr-serverless:ap-southeast-1:111122223333:/applications/00fabcd1234efgh5" }

aws emr-serverless start-job-run --application-id 00fabcd1234efgh5 \
  --execution-role-arn arn:aws:iam::111122223333:role/cdr-job-runtime-role \
  --job-driver '{"sparkSubmit":{"entryPoint":"s3://cdr-code-example/aggregate.py","entryPointArguments":["--date","2026-10-29"]}}'

aws emr-serverless get-job-run --application-id 00fabcd1234efgh5 --job-run-id 00fabcd9876xyz12 \
  --query 'jobRun.{state:state,vCPUHours:totalResourceUtilization.vCPUHour}'

{ "applicationId": "00fabcd1234efgh5", "jobRunId": "00fabcd9876xyz12", ... }
{ "state": "SUCCESS", "vCPUHours": 41.3 }

In the console, open the cluster’s Instances tab while a Spot task group runs, and the Steps tab to see each step’s logs. On EMR Serverless, the Spark UI link on a finished job run shows how many executors each stage used.

#Summary

Key takeaways

• Amazon EMR (Elastic MapReduce) is a managed big-data platform: it provisions clusters of EC2 instances (up to hundreds or thousands), installs Spark, Hadoop, Hive, HBase, Presto/Trino, Flink and more, and scales them. Use cases: data processing and ETL, machine learning, web indexing, big data analytics. Keywords: Hadoop, Spark, Hive, HBase, migrate Hadoop.
• Primary node (formerly master): YARN ResourceManager, HDFS NameNode, job and health tracking. One, or three for HA. Losing a single primary terminates the cluster.
• Core nodes: run tasks and store HDFS data. Losing one can lose blocks; HDFS replication factor 1 / 2 / 3 for 1–3 / 4–9 / 10+ core nodes. Task nodes (optional): compute only, no HDFS, usually Spot.
• Purchasing: On-Demand (reliable), Reserved (min 1 year, applied automatically) and Savings Plans (steady baseline), Spot (up to ~90% cheaper, two-minute notice). EMR fee on top, per second. Exam answer: primary and core On-Demand/Reserved, task Spot. Spot on the primary is allowed but not recommended except for test or cost-driven all-Spot clusters.
• Transient clusters run their steps and terminate (batch, pay per run); long-running clusters serve many users (termination protection, Reserved baseline, scaling). Auto-termination stops idle clusters.
• HDFS is ephemeral (core nodes’ disks, gone with the cluster). Keep input and output in S3 through EMRFS (S3A from EMR 7.10); S3 is strongly consistent, so “consistent view” is no longer needed.
• A cluster runs in one AZ; resilience = data in S3 + relaunch elsewhere.
• Instance fleets: up to 5 instance types (30 with an allocation strategy), On-Demand and Spot targets, several subnets, Spot timeout → On-Demand. Uniform instance groups: one type and one purchasing option per group, custom automatic scaling.
• Managed scaling: min, max, On-Demand limit, max core; YARN applications only. Grow with task nodes.
• EMR on EC2 (full control) vs EMR on EKS (Spark on your Kubernetes) vs EMR Serverless (Spark/Hive job runs, per-second billing, pre-initialized capacity, maximum capacity, auto-stop after 15 idle minutes).

Self-check: Amazon EMR32 questions