Virtual Memory, Paging and Swap

By the end of this lesson you will be able to

• Say what virtual memory really is (per-process address translation), and why “using the disk as memory” describes swap, a separate and optional feature.
• Explain virtual and physical addresses, the MMU, pages, frames and page tables, and translate an address by hand.
• Show why page tables are multi-level, and work out what a TLB hit, a TLB miss and a huge page each cost.
• Walk through a page fault step by step, and tell minor faults, major faults and invalid accesses apart.
• Describe how Linux reclaims memory, what swap and swappiness really do, why thrashing happens, and how to decide whether a server should have swap.
• Explain copy-on-write and fork, including why a Redis snapshot can double memory use.
• Explain mmap and the page cache, why Kafka and databases rely on them, and what survives a process crash versus a power cut.
• Explain overcommit and the OOM killer, read VSZ and resident memory correctly, and size memory limits for containers.

#Where this lesson fits

Lesson 1.2 said in one line that swapping makes everything slow and then the OOM killer ends the process. Lesson 3.1 said each process has its own address space, threads share one, and fork is cheap thanks to copy-on-write, a promise this lesson now explains. Lesson 3.2 listed the TLB among the hidden costs of a context switch; here you see why.

This lesson explains how one physical RAM is shared by hundreds of processes that all believe they own the machine, and what that means for design: why a restart makes a database slow, why Kafka is fast with a small heap, why a container dies while the host has free memory. How a hypervisor adds a second layer of translation under all this belongs to Chapter 4.

#What virtual memory really is

Virtual memory is a technique where each process gets its own private address space, and every address it uses is translated by hardware into a real location in RAM. A program never uses a physical RAM address. It uses virtual addresses, and the CPU and the kernel together decide where each one really lands.

       Process A                       Process B
  Virtual address space           Virtual address space
           |                               |
           v                               v
      A's page tables                 B's page tables
           |                               |
           +---------------+---------------+
                           v
                    Physical memory

Two programs can both use address 0x5000 and never collide, because their tables send that number to different places.

#Why it is needed

#Virtual addresses, physical addresses and the MMU

A virtual address is the number a process uses: every pointer you have ever printed is one. A physical address is the actual location on the RAM hardware. Processes never see it.

The MMU (Memory Management Unit) is the part of the CPU that translates one into the other on every single memory access: each instruction fetch, each load, each store. Since it happens billions of times per second, it has to be done in hardware. The kernel’s job is different: it writes the page tables that the MMU reads, and it handles the cases the MMU cannot.

A 64-bit process does not really have 64 bits to use. Common x86-64 CPUs use 48 virtual bits, or 256 TB, of which Linux gives the lower half to user space (128 TB); the upper half is reserved for the kernel. CPUs with five-level paging extend this to 57 bits, 128 PB. So VSZ can be enormous on a machine with little RAM, and that is not a contradiction: an address space is only a range of numbers.

#Pages, frames and the page table

Translating each byte separately would need a table bigger than memory itself, so memory is managed in fixed blocks:

• A page is a fixed-size block of virtual memory.
• A frame is a block of physical memory of the same size.
• The page table says which virtual page lives in which physical frame.

Virtual memory            Page table             Physical RAM
+--------+                                       +---------+
| Page 1 |  ------------> frame 3 ------------>  | Frame 1 |
+--------+                                       +---------+
| Page 2 |  ------------> frame 1 ------------>  | Frame 2 |
+--------+                                       +---------+
| Page 3 |  ------------> frame 2 ------------>  | Frame 3 |
+--------+                                       +---------+

The usual size is 4 KB. It is not universal: x86-64 also supports 2 MB and 1 GB pages, Arm servers can use 4, 16 or 64 KB as a base size, and Apple Silicon uses 16 KB. getconf PAGESIZE tells you, and portable code should ask rather than assume.

Each record in a page table is a page table entry (PTE), and it holds more than a frame number:

#Translating an address by hand

With 4 KB pages, 4,096 = 2^12, so the lowest 12 bits of an address are the offset inside the page and everything above them is the page number. In hexadecimal, 12 bits is exactly the last three digits, which makes this easy to do by eye.

Take virtual address 0x00403A7C, and suppose the page table maps page 0x403 to frame 0x1F2:

1. Split: page number 0x00403, offset 0xA7C.
2. Look up: read entry 0x403 of the table. It is an array index, not a search. Frame 0x1F2, present.
3. Keep the offset: a frame and a page have the same size, so the offset does not change.
4. Join: 0x1F2 followed by A7C gives physical address 0x1F2A7C.

The next byte, 0x00403A7D, is on the same page and needs no new lookup. That is why accessing memory in order is so cheap.

#Multi-level page tables

A single flat table is impossibly large. With 48-bit addresses and 4 KB pages there are 2^36 pages, and at 8 bytes per entry that makes 512 GB of page table for each process.

But the address space is mostly empty: a process has a few islands (code, heap, libraries, stack) in a vast sea. So page tables are a tree, and branches over empty regions simply do not exist. On x86-64 with four levels:

 47        39 38        30 29        21 20        12 11         0
+------------+------------+------------+------------+------------+
| L4 index   | L3 index   | L2 index   | L1 index   | offset     |
| 9 bits     | 9 bits     | 9 bits     | 9 bits     | 12 bits    |
+------------+------------+------------+------------+------------+

• Each table has 512 entries (2^9) of 8 bytes, so a table is exactly one 4 KB page. That neatness is deliberate.
• A CPU register (CR3 on x86) holds the physical address of the top table. Switching process means loading another value there.
• A small process needs only a handful of tables, so tens of kilobytes rather than hundreds of gigabytes.
• The price is time: a translation with no help takes four memory reads, plus the read of the data itself.

Page tables do use real memory, though, and it adds up. A 64 GB heap fully touched with 4 KB pages needs about 128 MB of last-level entries per process, and hundreds of processes each mapping a huge shared region can use gigabytes. PostgreSQL with very large shared_buffers and many connections is a known case, and huge pages are the usual fix.

#The TLB: making translation free

Five memory reads for each load would make computers unusable. The TLB (Translation Lookaside Buffer) is a small, fast cache of recent page-to-frame translations inside the CPU.

• TLB hit: the translation comes back in about a clock cycle. This is the overwhelmingly common case.
• TLB miss: a hardware walker reads the tree. Typically tens of nanoseconds, and more when the page-table reads also miss the CPU caches. It is not a fault, and no kernel code runs.
• Page not present: the walker finds the present bit off and raises a page fault. Only then does the kernel run.

TLBs are small. A core typically has a first level of a few dozen entries and a second level of one to a few thousand. What matters is TLB reach: entries × page size.

So a program that chases pointers randomly over a 64 GB heap misses the TLB constantly, and may spend a noticeable share of its time walking page tables even though no graph shows it as a problem. Use perf stat -e dTLB-load-misses to measure it. Under a hypervisor the walk is two-dimensional, as guest and host tables multiply; Chapter 4 explains that.

Two costs to remember for design:

• Process switches disturb the TLB. Page 0x403 means different things in different processes. Older hardware had to flush on every switch; with PCID (x86) or ASID (Arm), entries carry an address-space tag and can stay. Switching between threads of one process keeps the same tables, one concrete reason it is cheaper (lesson 3.2).
• TLB shootdowns. Each core has its own TLB. When a multi-threaded process unmaps or changes memory, x86 kernels must interrupt the other cores running it so they drop the stale entry. Frequent munmap on many cores is a real scalability bottleneck.

#Huge pages

A huge page ends the walk early, at level 2 for 2 MB or level 3 for 1 GB. The main win is TLB reach, and a smaller page-table tree also saves memory.

Trade-offs to weigh:

• Internal fragmentation: touching one byte can commit 2 MB, so sparse data structures waste memory.
• Latency: to build a 2 MB page the kernel may need to compact memory, stalling the faulting thread. Watch compact_stall in /proc/vmstat.
• Copy-on-write: one small write copies 2 MB, which makes the fork problem below worse.

Advice depends on the product and even the version: Redis warns about THP at startup, MongoDB’s recommendation has changed between versions, and many JVM workloads gain from it. Read your product’s documentation, then measure. cat /sys/kernel/mm/transparent_hugepage/enabled shows your setting; madvise (use THP only where the program asks) is a popular middle ground.

#Page faults

A page fault is the hardware exception raised when a translation cannot be completed. Despite the name, it is usually not an error: it is how the kernel gets control at the moment a page is needed. Allocating memory only records a range, and frames are given out on first touch. This is demand paging, and it is why starting a large program does not read the whole file into RAM.

When a process touches a page that is not currently in physical memory:

1. The kernel checks

   The MMU traps into the kernel with the faulting address. The kernel checks its list of valid regions for this process. If nothing covers the address, or the access breaks the permissions, it sends SIGSEGV and stops here.

2. It finds where the page should come from

   Zero-filled memory for a new heap page, the page cache or a file for a file-backed page, or a swap slot for a page paged out earlier.

3. It loads the page if needed

   It gets a free frame, evicting another page if it must, then fills it. If the data is on disk, the thread sleeps while other threads run.

4. It updates the page table

   The entry now holds the frame number, with the present bit on.

5. The instruction runs again

   The CPU restarts the same instruction from the beginning. The program never knows it was interrupted.

#Minor, major and invalid

Normal RAM access takes about 100 ns, so one major fault on NVMe costs as much as a thousand memory accesses. A request taking a few of these can go from 1 ms to 10 ms with nothing visibly wrong: no CPU spike, no error, and the average hardly moves. Major faults are a classic source of tail latency.

Design consequences:

• Cold starts. After a restart or a new deployment, code and data are not in memory, so the first requests take major faults. That is one reason to warm up an instance before sending it traffic, and to use a slow-start setting at the load balancer (Chapter 25).
• Latency-critical memory. mlock or mlockall pins pages in RAM, and pre-touching at startup (JVM -XX:+AlwaysPreTouch) moves the cost away from requests.
• Measure it. ps -o min_flt,maj_flt -p PID, pidstat -r 1, and perf all show fault rates.

#Swap and memory reclaim

When free memory runs low, the kernel reclaims frames, normally in the background through kswapd. If that cannot keep up, a thread that needs memory must do the reclaim itself (direct reclaim), and your application thread is suddenly stalled in the kernel. What it gives up depends on the kind of page:

To choose which pages, Linux keeps roughly least-recently-used lists, using the accessed bit from the PTE (and since kernel 6.1 it offers the multi-generational LRU as an option).

Swap is disk space used as a backing store for anonymous pages. Paging out and back in is what the notes call swapping. Historically “swapping” meant moving a whole process out; Linux moves individual pages, and people use both words for it. Since SSD and hard disks are orders of magnitude slower than RAM, a touched swapped-out page is painful.

#swappiness: not what most people think

vm.swappiness is commonly explained as “swap when free memory drops below this percentage”. It is not. It sets the relative preference between reclaiming anonymous pages and file pages when reclaim happens. The range is 0–200 since kernel 5.8, the default is 60, and 100 treats the two equally. swappiness=0 does not disable swap; the kernel still uses it as a last resort before killing a process. Kafka’s documentation suggests vm.swappiness=1, for example, so that idle heap goes only when there is no other choice.

#Thrashing

Thrashing is when the active working set no longer fits in RAM. Each page brought in pushes out a page needed a moment later, so the machine spends its time moving pages and almost none working. The symptoms: high si and so in vmstat, high I/O wait, collapsed throughput, a sluggish SSH session. Health checks then fail, the load balancer moves traffic to the other nodes, and they may go the same way. More swap does not help, because the problem is speed rather than space. Reduce the working set, add RAM, or shed load. Lesson 3.2 describes the same word for scheduling; the shape of the failure is identical.

#Should a server have swap?

Many cloud images ship with no swap, and Kubernetes historically required it to be off; newer versions can use it in a limited mode, so check yours. Clustered systems such as Elasticsearch ask you to disable swap or lock their memory, because a node paused by swap can be declared dead by its peers. A dead node is easier to handle than a slow one.

Alert on the rate, not the amount. A full swap with zero si is harmless: those pages really are idle. A small but steady si is a real problem. /proc/pressure/memory (PSI) directly reports the percentage of time tasks were stalled waiting for memory, and is the best single signal.

#Copy-on-write and fork

Lesson 3.1 introduced fork, which creates a child with a copy of the parent’s memory. Actually copying would be far too slow. So:

1. The kernel copies the page tables, not the frames, and marks every writable page read-only in both.
2. Parent and child share all the frames, and reads cost nothing.
3. When either one writes, the MMU raises a protection fault. The kernel recognises a copy-on-write (CoW) page, makes a private copy of that one page, repoints the writer, and restarts the instruction.

fork followed by exec, the way every shell starts a program, therefore copies almost nothing. The remaining cost is the page tables, and it grows with process size. Large processes that launch helpers should prefer posix_spawn or vfork.

The system design example: Redis persistence. BGSAVE forks. The child writes a snapshot from its frozen view of memory while the parent keeps serving. It is an elegant design, with two costs that catch people out:

• The fork pauses the server. Redis executes commands on one thread, and that thread is stuck in fork while the page tables are copied. For 24 GB there are about 48 MB of tables to copy; it is on the order of 10 ms per GB on normal hardware and can be much slower on some VMs. INFO stats shows latest_fork_usec.
• Memory can double. Each page updated during the save gets copied. If 30% of a 40 GB data set changes, you need about 52 GB. In the worst case, 80 GB. A machine sized at exactly the data set will hit the OOM killer during a save, typically at the worst moment. THP makes it worse by copying 2 MB per write.

This is why you never size Redis at 100% of RAM, and why Redis asks for vm.overcommit_memory=1: under the default heuristic, a big fork can be refused.

Copy-on-write also explains why pre-fork servers share less memory than you expect. Python and Ruby keep reference counts or garbage-collection marks inside objects, so merely reading an object can write to its page and un-share it. Instagram famously disabled parts of Python’s garbage collector for that reason, and gc.freeze() exists because of it.

#mmap and the page cache

#The page cache

The page cache is RAM the kernel uses to hold file contents. Every normal read and write passes through it, and it uses essentially all idle RAM, because empty RAM does nothing for you. It gives the memory back as soon as programs need it.

• Reads: a hit costs about a microsecond instead of 100 µs or several milliseconds. Sequential reads trigger read-ahead, so they rarely wait.
• Writes: data is copied into the cache and marked dirty, and write returns before anything reaches the disk. Flusher threads write it later, by default when it is about 30 seconds old or past vm.dirty_background_ratio. Past vm.dirty_ratio, writers are forced to wait, a common cause of write latency spikes.
• Durability: only fsync (or fdatasync) forces it to stable storage.
• Ownership: the cache belongs to the kernel. A process crash loses nothing that write accepted, while a power cut or kernel panic loses all dirty pages. The cache also survives a process restart, so a restarted database is warm if it relied on the page cache, and cold if it used only its own buffer pool.

#Why Kafka and databases care

Kafka keeps a small JVM heap, often single-digit gigabytes, and leaves the rest of RAM to the page cache. Messages are appended with ordinary writes, and consumers that keep up read data that is still cached, so no disk is touched. sendfile passes cached pages to the socket without a copy through the application (but TLS in user space loses this shortcut, unless kernel TLS is used). The cache survives a broker restart and stays out of garbage collection. Durability comes from replication rather than an fsync per message. The failure mode: one consumer reading far behind pulls old data in from disk and pushes fresh pages out, slowing everyone. Messaging itself is Chapters 30–32.

Databases differ, and an interviewer may ask why:

This is the reason “how much RAM does it need?” is really “how large is the working set?”. A database with a 2 TB file and 200 GB of hot data needs about 200 GB of cache. If the working set grows past your cache, performance doesn’t degrade gently; it falls off a cliff as hits become disk reads. Caching in general returns in Chapter 23.

#mmap

mmap asks the kernel to make a file, or plain memory, appear as a region of your address space.

It avoids a copy and a system call per read, and it lets you address files larger than RAM. The costs matter for storage engines:

• Hidden blocking. Any pointer dereference may take a major fault, and you cannot tell which one.
• Errors become signals. A disk error shows up as SIGBUS, not a return code.
• No control of eviction or of write timing. A database must guarantee its log reaches disk before its data pages, and the kernel may write mapped pages at any time.
• Shootdowns. Evicting mapped pages can need TLB shootdowns across cores.

MongoDB removed its mmap engine, MMAPv1, in version 4.2. A well-known 2022 paper, “Are You Sure You Want to Use MMAP in Your Database Management System?”, argues the case against it. Read-mostly systems such as LMDB and Lucene still do very well with it.

#Overcommit and the OOM killer

Linux, by default, overcommits: it hands out more virtual memory than exists, because most programs reserve far more than they touch. The setting vm.overcommit_memory picks the policy:

When pages are touched and nothing can be reclaimed, the page fault cannot be satisfied, and it is far too late for malloc to fail. The OOM killer picks a process, largely by memory use (see /proc/PID/oom_score), adjusted by oom_score_adj from -1000 (never) to 1000 (first), and sends SIGKILL. There is no graceful shutdown. dmesg or journalctl -k will show Out of memory: Killed process.

• The victim need not be the culprit. A leaking sidecar can get your database killed, so protect critical services with a low oom_score_adj and limit everything else.
• Malloc success proves nothing on Linux. Code that carefully handles a null return can still be killed later.
• The kernel’s OOM killer can act late, after the machine has been crawling for minutes. systemd-oomd and earlyoom act earlier, using pressure signals.

#Reading memory numbers correctly

Example: 8 pre-fork workers each show 2 GB of resident memory. That does not mean 16 GB is in use; if 1.5 GB is shared, the real total is about 1.5 + 8 × 0.5 = 5.5 GB. Pss shows each as about 0.69 GB.

Also, resident memory that stays high after free() isn’t necessarily a leak: allocators often keep freed memory for reuse. A leak is resident memory that keeps climbing under steady load.

#Memory limits in containers

A container isn’t a VM: it uses the host kernel with a cgroup limit (memory.max in cgroup v2). The kernel charges each frame the group uses, including page cache.

• Over the limit: first reclaim within the group, then an OOM kill within the group. Exit code 137 (128 + 9), and OOMKilled in Kubernetes. Free memory on the host doesn’t help. CPU limits only slow you down, but memory cannot be throttled like that, so memory limits kill. memory.high is a softer limit that slows the group under pressure.
• Page cache counts, but is reclaimable. A container reading big files can show usage near the limit and be perfectly healthy. Watch working set (usage minus inactive file cache), as the kubelet does. docker stats also excludes inactive cache.
• free and /proc/meminfo show the host’s memory, even inside a container. Read /sys/fs/cgroup/memory.current and memory.max instead.
• Runtimes must know the limit. A modern JVM is container-aware and by default sets max heap at 25% of the limit (-XX:MaxRAMPercentage). The heap isn’t the process, though: metaspace, thread stacks, direct buffers and JIT code all sit outside it. A 1 GB heap in a 1 GB limit will be killed. Go has GOMEMLIMIT, and Node has --max-old-space-size.
• Kubernetes requests vs limits. A request is used for scheduling and is a reservation in the scheduler’s accounts. A limit is enforced by the kernel. Setting requests equal to limits for memory avoids overcommitting the node, so it’s the common advice for stateful workloads.

#A short note on NUMA

On a multi-socket server, each socket has its own RAM. Memory attached to another socket takes longer to reach, by a factor of perhaps 1.3 to 2. That is NUMA (Non-Uniform Memory Access). Linux by default places a page on the node of the thread that first touches it, so a startup thread that initialises a big buffer can place all of it on one node. The classic failure: a database uses more than one node’s RAM, that node runs out, and the kernel swaps or reclaims even though the other node is free. Use numactl --hardware and numastat to look, and interleave memory (numactl --interleave=all) or pin processes to nodes to fix it. Most small cloud VMs show only one node, but big instances may not.

#What it means for system design

#Common production problems

#Interview angles

• “What is virtual memory?” Lead with translation and isolation. Mention swap only as an optional extra. This one answer separates candidates.
• “Why is Kafka fast?” Sequential I/O, the page cache and sendfile, not magic.
• “Why does Redis need spare memory?” Fork plus copy-on-write.
• “My container was killed but the node has free memory.” Cgroup limit, not host memory.
• “Is data safe after write?” Safe from a process crash, not from a power cut, until fsync.
• “Why is a thread switch cheaper than a process switch?” Same page tables, so the TLB stays useful.

#Try it yourself

All of this runs on any Linux machine, including Windows Subsystem for Linux. Read-only commands are safe on production; the drop_caches line is not for production.

# Page size and your memory picture
getconf PAGESIZE
free -h

# A process's virtual layout (code, heap, libraries, stack)
cat /proc/self/maps

# Virtual size vs resident memory for your shell
grep -E 'VmSize|VmRSS|VmSwap' /proc/$$/status

# Top memory users, with minor and major fault counts
ps -eo pid,comm,vsz,rss,min_flt,maj_flt --sort=-rss | head

# Swap activity and memory pressure: si and so should stay near 0
vmstat 1 5
cat /proc/pressure/memory

# Settings from this lesson
sysctl vm.swappiness vm.overcommit_memory vm.dirty_ratio
cat /sys/kernel/mm/transparent_hugepage/enabled

# NUMA layout, if numactl is installed
numactl --hardware

See demand paging. This Python script maps 1 GB and shows resident memory before and after touching it:

import mmap, re, resource

def resident():
    text = open('/proc/self/status').read()
    return re.search(r'VmRSS:\s+(\d+)', text).group(1) + ' kB'

def minor_faults():
    return resource.getrusage(resource.RUSAGE_SELF).ru_minflt

size = 1024 * 1024 * 1024
print('start          ', resident(), minor_faults())
m = mmap.mmap(-1, size)                  # anonymous mapping: 1 GB of address space
print('after mapping  ', resident(), minor_faults())
for offset in range(0, size, 4096):      # touch one byte in each 4 KB page
    m[offset] = 1
print('after touching ', resident(), minor_faults())

Mapping changes almost nothing. Touching grows resident memory by 1 GB, with about 262,000 minor faults (far fewer if THP gives you 2 MB pages, which is a nice way to see that it is on).

See the page cache. The second read is much faster:

dd if=/dev/zero of=$HOME/big.bin bs=1M count=1024  # create a 1 GB file
sync
echo 3 | sudo tee /proc/sys/vm/drop_caches          # empty the cache (not on production)
time cat $HOME/big.bin > /dev/null                  # slow: from disk
time cat $HOME/big.bin > /dev/null                  # fast: page cache
free -h                                             # buff/cache grew
rm $HOME/big.bin

See a container limit kill. Python allocates a 200 MB string under a 100 MB limit:

docker run --rm -m 100m --memory-swap 100m python:3-alpine \
  python -c "x = b'a' * (200 * 1024 * 1024); print('survived')"
echo "exit code: $?"        # 137, with no Python traceback

#Summary

Key takeaways

• Virtual memory is per-process address translation, not “disk as RAM”. Swap is optional, and many servers run with none.
• Addresses split into page number and offset. The page table maps pages to frames, and the offset stays the same. The MMU does this on each access.
• Page tables are multi-level because a flat one would be 512 GB per process. An unhelped lookup costs 4 reads plus the data.
• The TLB caches translations. Its reach is entries × page size, which is why huge pages help big heaps. THP can also cause latency spikes.
• A page fault is usually normal. Minor is about a microsecond, major is about 100 µs on NVMe or milliseconds on a hard disk, and invalid gets SIGSEGV.
• Reclaim drops clean pages free, writes dirty pages, and sends anonymous pages to swap. Thrashing happens when the working set doesn’t fit. swappiness is a preference, not a threshold.
• Copy-on-write makes fork cheap, but a Redis snapshot under heavy writes can need up to double the memory.
• The page cache uses all idle RAM. write succeeds before the disk does; a process crash keeps the data, a power cut doesn’t. Kafka and many databases depend on it. mmap shares those frames.
• Linux overcommits. When the bet fails the OOM killer sends SIGKILL, and in a container that happens at the limit, with exit code 137, whatever the host has free.
• Read available, not free. VSZ is mostly noise. Resident memory doesn’t add up across processes; Pss does.

Self-check: virtual memory30 questions