Docker, Containers vs VMs, and ECR

By the end of this lesson you will be able to

• Explain what Docker is and why “it works on my machine” stops being a problem once an application ships as a container.
• Tell an image (the frozen package) from a container (a running copy of it), read a Dockerfile, and explain image layers and the build cache.
• Follow the build → push → pull → run cycle, and name the registries an image can live in: Docker Hub, Amazon ECR private and ECR Public.
• Compare containers with virtual machines: shared kernel, start time, density and isolation.
• Run Docker by hand on EC2, and explain why that breaks down at a hundred containers and why orchestration (ECS, EKS) exists.
• Place ECS, EKS, Fargate, ECR and App Runner on one map of the container services on AWS.
• Use Amazon ECR well: repositories and URIs, IAM and repository policies, get-login-password, tags vs digests, tag immutability, basic vs enhanced scanning, lifecycle policies, replication and pull through cache.
• Pull images from a private subnet with VPC endpoints (or a NAT gateway), and troubleshoot CannotPullContainerError.

#Why containers: “it works on my machine”

A developer at a Dhaka e-commerce shop finishes the new checkout API on her laptop: Node.js 22, a particular version of an image library, a config file in a particular folder. It works. The operations team installs it on the servers, where Node.js 18 is installed and the image library is a different version. It crashes. Two days before the Eid sale, two teams are arguing about whose machine is “right”.

The application was never just the code. It was the code plus its runtime, its libraries, its settings and the operating system files it expects. A container ships all of that together.

#What Docker is

Docker is a platform for building, shipping and running applications in containers. Released in 2013, it made a set of Linux kernel features (namespaces to isolate what a process can see, cgroups to limit how much CPU and memory it may use) easy enough for every developer to use. Since 2015 the formats are open standards of the OCI (Open Container Initiative), so an image built with Docker also runs on containerd, Podman, ECS, Kubernetes or Lambda.

What you get:

• Runs the same everywhere: any machine with a container runtime, no compatibility issues, predictable behaviour.
• Any language, any stack: Node.js, Java, Python, PHP, Go, .NET; the runtime only sees a container.
• Less work to deploy and maintain: one artefact moves from laptop to test to production unchanged.
• Fast and light: a container starts in about a second and many fit on one server (see “Docker vs virtual machines” below).

Typical uses: microservices (each service in its own container, deployed on its own schedule), lift-and-shift of on-premises applications to AWS without rewriting them, batch jobs, CI build environments, and machine learning models packaged with their exact library versions.

#Image and container are not the same thing

The notes use “image” and “container” almost interchangeably. Keep them apart; the exam and every error message do.

A programmer’s comparison: the image is the class, the container is the object. A cook’s comparison: the image is the sealed recipe box with ingredients measured out; a container is one pot of biryani cooked from it. You can cook ten pots from one box, and spilling one pot does not change the box.

The last row matters in production: anything an application writes inside its container (uploaded files, a local database file, logs on disk) disappears when the container is replaced. Persistent data belongs in a volume or a managed service: EFS, EBS, S3 or a database (lesson 14.2 covers volumes for ECS tasks).

#The Dockerfile

An image is built from a Dockerfile, a short text file of instructions. The checkout API’s Dockerfile:

# 1. Base image: Linux files + Node.js 22, from a registry
FROM public.ecr.aws/docker/library/node:22-alpine

WORKDIR /app

# 2. Dependencies first: this layer is rebuilt only when package files change
COPY package.json package-lock.json ./
RUN npm ci --omit=dev

# 3. The application code: changes on every commit
COPY src/ ./src/

# 4. Metadata: which port the app listens on, which user, what to start
EXPOSE 3000
USER node
CMD ["node", "src/server.js"]

• FROM picks a base image (here the official Node.js image, served from ECR Public instead of Docker Hub).
• COPY and RUN add files and run commands; each one creates a layer.
• EXPOSE, USER, CMD add metadata: they do not add files.

#Layers and the build cache

Each file-changing instruction produces a read-only layer, a set of file changes stacked on the previous one. Layers are identified by a hash of their contents, which gives three big savings:

• Build cache. When you rebuild after changing only src/, Docker reuses the base and dependency layers and rebuilds only the last ones. That is why the Dockerfile copies package.json before the code: the slow npm ci layer is reused on most builds.
• Push and pull only what changed. A registry already holding the base and dependency layers says “layer already exists”; only the new code layer (a few MB) travels. A host that already has the old version downloads only that layer.
• Shared on disk. Ten containers from images that share a base layer store that base once per host.

When a container starts, the runtime puts one thin writable layer on top of the image’s read-only layers. That layer, and only that layer, belongs to the container, and it is deleted with it.

#Build, push, pull, run

The whole Docker workflow fits in four verbs:

1. Build

   docker build turns the Dockerfile and your source folder into an image on your machine (or in a CI pipeline).

2. Push

   docker push uploads the image’s layers and its manifest to a repository in a registry.

3. Pull

   On any host, docker pull downloads the manifest, then every layer the host does not have yet.

4. Run

   docker run creates a container from the image: a writable layer, an isolated process, network and limits, and starts the command from CMD.

#Where images are stored: registries

A registry is a server that stores images; a repository is one named collection inside it (one per application, usually), holding many versions of that image identified by tags and digests.

The full name of an image tells you exactly where it lives:

So the checkout image is 111122223333.dkr.ecr.ap-southeast-1.amazonaws.com/checkout-example:1.4.2. A name with no registry, such as nginx, means Docker Hub (docker.io/library/nginx), and a name with no tag means the tag latest.

#Docker vs virtual machines

Docker is sort of a virtualization technology, but not quite.

On AWS you usually get both: your containers run on EC2 instances (VMs), and Docker packs many containers onto each instance. Fargate takes this further: every Fargate task runs in its own isolated micro-VM that shares no kernel with other tasks, so you get container convenience with VM-level isolation (lesson 14.2).

#Docker by hand on EC2

#The empty plot of land

To run containers on an instance yourself:

# On an Amazon Linux 2023 instance (SSH or Session Manager)
sudo dnf install -y docker
sudo systemctl enable --now docker          # start the Docker daemon, now and at boot
sudo usermod -aG docker ec2-user            # let ec2-user run docker without sudo

# Log in to ECR (the instance role must allow ECR pulls), pull and run
aws ecr get-login-password --region ap-southeast-1 | docker login --username AWS \
  --password-stdin 111122223333.dkr.ecr.ap-southeast-1.amazonaws.com
docker run -d --name checkout -p 80:3000 --restart unless-stopped \
  111122223333.dkr.ecr.ap-southeast-1.amazonaws.com/checkout-example:1.4.2

-p 80:3000 maps port 80 of the host to port 3000 in the container. --restart unless-stopped asks the Docker daemon to restart the container if its process exits. For one small application on one instance, this is fine. (Elastic Beanstalk’s Docker platform, lesson 9.4, automates exactly this.)

#Why it breaks down at 100 containers

Now the shop has 12 microservices, 100 containers spread over 10 instances, and an Eid sale coming. By hand, someone must:

• Place each container: which instance has free CPU and memory, and which host ports are still free?
• Restart crashes. --restart restarts a process that exits on the same host, but it does not know whether the app is healthy (a frozen app keeps running), and it can do nothing when the host itself dies: every container on that instance is gone and nobody starts them elsewhere.
• Deploy new versions without downtime: stop and start 100 containers in the right order, and roll back if the new version fails.
• Scale with traffic: more containers before the sale, fewer after.
• Wire up the load balancer: register each new container’s host and port, deregister dead ones.
• Hand out secrets, logs and permissions to each container.

Doing this by logging in to servers is impossible to keep up with, and it fails exactly at 3 a.m. during a sale. The answer is orchestration: you declare what should run (“9 copies of checkout:1.4.2, each with 0.5 vCPU and 1 GiB, behind this load balancer”), and an orchestrator keeps reality equal to that declaration, around the clock. AWS offers two orchestrators: ECS and EKS.

#Containers on AWS: the menu

Two more places run containers: Elastic Beanstalk with its Docker platforms (lesson 9.4), and AWS Lambda, which can run a function packaged as a container image of up to 10 GB, pulled from ECR (chapter 15).

Think of it as layers: ECR stores the image; ECS or EKS decides what runs where and keeps it running; EC2 or Fargate provides the compute underneath.

#Amazon ECR: AWS’s own Docker Hub

Amazon Elastic Container Registry (ECR) is a fully managed registry for Docker and OCI images (and other OCI artifacts such as Helm charts and signatures). The notes call it AWS’s own Docker Hub, which is a good start, with two corrections: it has private and public repositories, and it does much more than store.

#Private registry, repositories and storage

• Every AWS account has one private registry per Region: <account>.dkr.ecr.<region>.amazonaws.com. Inside it you create repositories (checkout-example, payments/ledger-example…).
• Private repositories are reachable only with IAM credentials.
• Public repositories live in ECR Public and are listed in the ECR Public Gallery (gallery.ecr.aws). Anyone can pull them anonymously; you publish open-source tools or public SDK images there. They are created through the us-east-1 API, whatever Region your compute is in.
• Image layers are stored in Amazon S3 behind the scenes, so they inherit S3’s durability, and they are encrypted at rest (AES-256 by default, or a KMS key you choose when the repository is created; it cannot be changed later).
• You pay for storage (private: about USD 0.10 per GB-month) and for data transferred out of the Region. Pulls by EC2, Fargate, Lambda or App Runner in the same Region are free. ECR Public gives 50 GB of storage free each month.

#Access control: IAM, repository policies and the login token

ECR is “secured by IAM”, as the notes say. In practice there are three kinds of policy and one token:

The Docker CLI does not speak IAM. So you first call aws ecr get-login-password, which calls the GetAuthorizationToken API and returns a token valid for 12 hours, scoped to the permissions of the IAM principal that asked for it, and pipe it into docker login --username AWS. ECS, EKS and Lambda do this for you with their roles. If a pipeline that has run for 13 hours suddenly fails to push, the token expired.

A pull needs three actions: ecr:GetAuthorizationToken (on *), then ecr:BatchGetImage (the manifest) and ecr:GetDownloadUrlForLayer (the layers) on the repository. A push needs the upload actions too. When something is refused, the slides’ rule holds: permission errors mean a policy is missing, either IAM on the caller or the repository policy for another account.

#Tags, digests and tag immutability

A tag (1.4.2, latest, prod) is a movable label that points to an image. A digest (sha256:…) is the hash of the image manifest itself: it never changes and can never point to anything else.

By default, ECR tags are mutable: pushing a new image as 1.4.2 silently moves the tag, and the old image becomes untagged. That causes nasty surprises: two tasks of the same service, started an hour apart, run different code although both say 1.4.2.

Turn on tag immutability for the repository (--image-tag-mutability IMMUTABLE): pushing an existing tag again fails with ImageTagAlreadyExistsException, so 1.4.2 means the same bytes forever. Rules of thumb:

• Tag images with something unique: a version, the Git commit hash, a build number. Do not deploy latest; it is just a tag, not “the newest”.
• For full certainty, deploy by digest (checkout-example@sha256:…). ECS already resolves a tag to a digest when a deployment starts, so all tasks of one deployment run the same image.
• A newer option lets an immutable repository exclude a few tags by wildcard (for example keep latest movable) with IMMUTABLE_WITH_EXCLUSION, and a mutable repository protect a few tags the other way round.

#Image scanning: basic and enhanced

The notes’ exam trick: “store Docker images privately and scan for vulnerabilities” → Amazon ECR. ECR has two kinds of scanning, configured at the registry level with repository filters:

Basic scanning is a snapshot: an image that was clean on the day it was pushed is not re-checked when a new vulnerability appears next week. Enhanced scanning keeps watching, which is what a payment company needs for images that stay in production for months. Inspector itself is covered in lesson 21.7.

A scan does not block anything by itself. To stop a vulnerable image from reaching production, let the pipeline read the findings (or react to the EventBridge event) and fail the deployment on critical or high findings.

#Lifecycle policies

A CI pipeline that pushes on every commit fills a repository with thousands of old images, and storage is billed per GB-month. A lifecycle policy cleans up automatically, with rules such as:

• expire untagged images older than 7 days (the leftovers of moved tags);
• keep only the last 30 images tagged prod-* (imageCountMoreThan);
• expire images tagged dev-* 14 days after they were pushed (sinceImagePushed).

Rules are evaluated together and applied by priority (lower number wins); expired images are removed within 24 hours of matching, and each action is logged in CloudTrail. Always run the lifecycle policy preview first: a careless rule can expire the image production still runs.

#Replication: cross-Region and cross-account

ECR can replicate images from your private registry to other Regions and other accounts. You configure it once at the registry level (up to 25 destinations, optionally only for repositories with a given name prefix):

• Cross-Region: the Singapore registry copies every new image to Mumbai, so a disaster-recovery cluster there pulls locally and fast, with no cross-Region transfer at launch time.
• Cross-account: the tooling account where CI pushes copies images into the production account. The destination registry must have a registry policy allowing the source account to replicate (ecr:ReplicateImage, and ecr:CreateRepository if repositories should be created automatically).

Things to remember: only images pushed after replication is configured are copied (older ones need a backfill); repository names stay the same; most images arrive within 30 minutes; repository policies and lifecycle policies are not replicated (use repository creation templates to give new repositories the right settings); and replication never deletes anything in the destination.

#Pull through cache

A pull through cache rule makes your private registry a caching proxy for an upstream registry. You pull 111122223333.dkr.ecr.ap-southeast-1.amazonaws.com/docker-hub/library/nginx:1.27; the first time, ECR fetches the image from Docker Hub, stores it in a repository it creates for you, and returns it; later pulls come from your registry. ECR checks the upstream for a newer version of that tag at most once every 24 hours, and keeps serving the cached copy if the upstream is unreachable.

Supported upstreams include ECR Public, the Kubernetes registry and Quay (no credentials), Docker Hub, GitHub Container Registry, Azure Container Registry and others (credentials stored in a Secrets Manager secret whose name starts with ecr-pullthroughcache/), and another ECR registry. Why bother:

• Docker Hub rate limits. Hundreds of Fargate tasks behind one NAT gateway share one public IP, and Docker Hub counts anonymous pulls per IP: a scale-out during the sale can hit the limit and fail with toomanyrequests. Through the cache, Docker Hub sees one pull a day.
• Private subnets. After the first pull, tasks pull through the ECR VPC endpoints with no internet access (the very first pull needs a route to the internet).
• Scanning and control. Cached images live in your registry, so they are scanned and governed like your own.

#Pulling images from a private subnet

Containers usually run in private subnets, with no public IP. To pull an image they still need to reach three things:

There are two ways to give a private subnet that reach (lesson 3.4 and lesson 22.2):

• A NAT gateway in a public subnet. Simple, and it also reaches Docker Hub and other registries. But every byte of every layer goes through it and is billed per GB processed, and the subnet has outbound internet access, which some security teams forbid.
• VPC endpoints, no internet at all: two interface endpoints, ecr.api and ecr.dkr (with private DNS turned on, and a security group allowing HTTPS on 443 from the tasks), plus an S3 gateway endpoint on the subnets’ route tables, because the layers come from S3. The S3 gateway endpoint is free and carries most of the bytes.

ECS adds a few more: Fargate tasks that send logs with the awslogs driver need the CloudWatch Logs endpoint (logs), and tasks that read secrets need the Secrets Manager or SSM endpoints; EC2 container instances also need the ECS endpoints to talk to the ECS service (lesson 14.2).

The classic mistake: someone creates the two ECR interface endpoints, removes the NAT gateway, and forgets the S3 gateway endpoint. Login and manifest work, then every layer download times out and the task stops with CannotPullContainerError.

Try the combinations yourself:

#What goes wrong

#Containers and ECR in system design

• One artefact, many environments. Build once, push once, and promote the same digest from test to staging to production by adding tags or replicating to the production account. What you tested is exactly what runs.
• Separate accounts for build and run. CI pushes to a tooling account; production pulls through cross-account replication or a repository policy with read-only access. A compromised production role can never push.
• Multi-Region. Replicate images to every Region you run in (or your DR Region), so a scale-out or failover never depends on another Region’s registry.
• Private by default. Tasks in private subnets with ECR and S3 endpoints, no NAT for image pulls, and base images through pull through cache.
• Secure supply chain. Scan on push, enhanced scanning for what runs, immutable tags, signing, and a lifecycle policy so old vulnerable images do not linger.
• When not to containerize. A single small script that runs on an event is often simpler as a Lambda function; a static website belongs in S3 and CloudFront.

#Try it yourself

Create a repository, build and push an image, scan it, and add a lifecycle policy. Output is shortened.

# 1. A private repository with immutable tags and scan on push
aws ecr create-repository --repository-name checkout-example \
  --image-tag-mutability IMMUTABLE \
  --image-scanning-configuration scanOnPush=true --region ap-southeast-1
# "repositoryUri": "111122223333.dkr.ecr.ap-southeast-1.amazonaws.com/checkout-example"

REG=111122223333.dkr.ecr.ap-southeast-1.amazonaws.com

# 2. Log Docker in to the registry (token valid 12 hours)
aws ecr get-login-password --region ap-southeast-1 | \
  docker login --username AWS --password-stdin $REG
# Login Succeeded

# 3. Build for both CPU architectures, tag with a version, push
docker buildx build --platform linux/amd64,linux/arm64 \
  -t $REG/checkout-example:1.4.2 --push .
# => pushing layers ... done
# => pushing manifest for .../checkout-example:1.4.2@sha256:9f86d081...

# 4. Pushing the same tag again is refused (immutability)
docker push $REG/checkout-example:1.4.2
# tag invalid: The image tag '1.4.2' already exists in the 'checkout-example' repository
#   and cannot be overwritten because the repository is immutable.

# 5. Read the scan findings
aws ecr describe-image-scan-findings --repository-name checkout-example \
  --image-id imageTag=1.4.2 --region ap-southeast-1
# "imageScanStatus": { "status": "COMPLETE" },
# "findingSeverityCounts": { "HIGH": 1, "MEDIUM": 3 }

# 6. Lifecycle: expire untagged after 7 days, keep the last 30 release images
aws ecr put-lifecycle-policy --repository-name checkout-example \
  --lifecycle-policy-text file://lifecycle.json --region ap-southeast-1

# 7. Run it locally, then look at its layers
docker run -d -p 8080:3000 $REG/checkout-example:1.4.2
docker image history $REG/checkout-example:1.4.2

lifecycle.json:

{
  "rules": [
    { "rulePriority": 1, "description": "Expire untagged after 7 days",
      "selection": { "tagStatus": "untagged", "countType": "sinceImagePushed",
                     "countUnit": "days", "countNumber": 7 },
      "action": { "type": "expire" } },
    { "rulePriority": 2, "description": "Keep the last 30 releases",
      "selection": { "tagStatus": "tagged", "tagPatternList": ["1.*"],
                     "countType": "imageCountMoreThan", "countNumber": 30 },
      "action": { "type": "expire" } }
  ]
}

For a private subnet, the three endpoints (replace the IDs with yours):

for svc in ecr.api ecr.dkr; do
  aws ec2 create-vpc-endpoint --vpc-id vpc-0abc1234example --vpc-endpoint-type Interface \
    --service-name com.amazonaws.ap-southeast-1.$svc --private-dns-enabled \
    --subnet-ids subnet-0aaa1111example subnet-0bbb2222example \
    --security-group-ids sg-0ccc3333example
done
aws ec2 create-vpc-endpoint --vpc-id vpc-0abc1234example --vpc-endpoint-type Gateway \
  --service-name com.amazonaws.ap-southeast-1.s3 --route-table-ids rtb-0ddd4444example

#Summary

Key takeaways

• Docker packages code with its runtime, libraries and settings so it runs the same on any machine: the restaurant in a shipping container. Formats are open OCI standards.
• An image is a read-only, layered package built from a Dockerfile; a container is a running instance with its own writable layer, which is lost when the container is removed.
• Build → push → pull → run. Registries: Docker Hub (public, rate-limited), ECR private (IAM), ECR Public (gallery.ecr.aws). Layers are cached, so only changes are pushed and pulled.
• Containers vs VMs: containers share the host kernel, start in about a second and pack densely; VMs each run their own OS. EC2 is a VM; Fargate gives each task its own micro-VM.
• Docker by hand on EC2 (the empty plot of land) works for one app, not for 100 containers: no placement, no restart after a host failure, no rolling deploys, no scaling. Orchestration = ECS (AWS’s own) or EKS (Kubernetes), on EC2 or Fargate, pulling from ECR; App Runner is the simplest option.
• ECR: private and public repositories, stored in S3, encrypted, IAM identity policies plus repository policies (other accounts) and a registry policy (replication); get-login-password gives a 12-hour token; permission errors mean a policy is missing.
• Tags move, digests do not. Use tag immutability and unique tags; deploy by digest for certainty.
• Scanning: basic (OS packages, on push or manual) vs enhanced (Amazon Inspector, OS and language packages, continuous). “Store images privately and scan for vulnerabilities” → ECR.
• Lifecycle policies expire (or archive) old images; replication copies new images cross-Region and cross-account; pull through cache proxies Docker Hub and others.
• Private subnet pulls need ecr.api + ecr.dkr interface endpoints and an S3 gateway endpoint (or a NAT gateway). Otherwise: CannotPullContainerError.

Self-check: Docker and Amazon ECR30 questions