Concurrency vs Parallelism, and Context Switching

By the end of this lesson you will be able to

• Define concurrency and parallelism precisely, draw them on a timeline, and correct the common claim that concurrency only happens on one core.
• Explain why concurrency multiplies a server’s throughput on I/O-heavy work and does nothing for pure CPU work on one core, with the arithmetic.
• Walk through a context switch step by step, list what a context contains, and name what triggers one and what does not.
• Separate the direct cost of a switch (about 1–5 µs) from the larger indirect cost in caches and the TLB, and calculate when switching becomes thrashing.
• Describe how the Linux scheduler works: time slices, run queues, preemption, virtual runtime, CFS, EEVDF and nice.
• Tell CPU-bound from I/O-bound work, and voluntary from involuntary switches, and diagnose each with vmstat and pidstat.
• Use Amdahl’s law to predict the limit of adding cores, and tell real cores from hyper-threads.
• Explain Python’s GIL and free-threaded builds, and why a container at “40% CPU” can be throttled into terrible latency.

#Where this lesson fits

Lesson 3.1 defined processes and threads and the private state each thread owns. Lesson 1.2 showed the concurrency models of a server and Little’s law, and lesson 1.13 showed how web and app servers use workers.

This lesson answers the question those lessons left open: how does a machine with 16 cores run 3,000 threads, and what does that cost? Choosing the right number of threads comes straight after, in lesson 3.3, and how the TLB works in lesson 3.4. What goes wrong when threads touch the same data (races, locks, the memory model) is Chapter 14.

#Concurrency: many tasks in progress

Concurrency means managing several tasks so that their lifetimes overlap: each one makes progress over a period of time, and a new one can start before the last has finished. It does not mean they all execute at the same instant.

Take the web server you met in lesson 1.2. Three users send requests at the same moment:

User A → Request 1
User B → Request 2
User C → Request 3

• If the server handles one request to the end and makes the others wait, that is sequential processing.
• If it interleaves the work of several requests and moves all of them forward, that is concurrent processing.

On one core the scheduler creates this by switching quickly between tasks, so the timeline looks like this:

Time →
Task A | Task B | Task A | Task C | Task B

#Parallelism: many tasks executing at once

Parallelism means several instructions are really executing at the same instant, on separate cores or processors:

Core 1: Task A ───────────
Core 2: Task B ───────────
Core 3: Task C ───────────

So the two words answer different questions. Rob Pike’s line is the one to remember: concurrency is about dealing with lots of things at once; parallelism is about doing lots of things at once. Concurrency is a property of how the program is structured, and parallelism is a property of how it runs.

They are independent, so all four combinations exist: neither (a simple script), concurrent only (one core taking turns), parallel only (one big calculation split over cores with no task switching), and both (nearly every server you will ever run).

#Why concurrency is worth having

The notes list five reasons, and they all come down to the same thing: most of a task’s lifetime is not spent computing.

• Handling many users. Nobody should wait for a stranger’s request to end before theirs can begin.
• Doing other work during I/O waits. A waiting thread needs no CPU, so another can use it.
• Running background tasks, such as sending emails or cleaning up, without stopping the main work.
• Using CPU and I/O resources well. An idle core while requests queue up is wasted money.
• Raising server throughput, sometimes by a factor of ten with no new hardware.

Here is that last claim with numbers. A request needs 5 ms of CPU and 45 ms waiting for the database, on one core:

The same core does ten times the work and each request is no faster. It is just that the database calls overlap. With Little’s law from lesson 1.2, 200 req/s × 0.05 s means 10 requests in flight, so about 10 threads, which lesson 3.3 turns into a sizing rule. That only holds when the database can actually take those 10 queries at once.

Concurrency has real costs too. Interleaved code can touch shared data at bad moments, which creates races (Chapter 14), and a thread may be stopped between any two instructions, even on one core. It is also harder to debug, and each thread uses memory. And the switching itself costs CPU, which is the rest of this lesson.

#Context switching

A context switch is what happens when a core stops running one thread and starts running another: the current thread’s execution state is saved and the next thread’s state is restored. The core itself has no idea about “threads”: it simply executes whatever its registers say. Changing the registers changes the thread.

CPU
 |
 +---- Thread A
 |
 +---- Thread B
 |
 +---- Thread C

Say the CPU is running thread A and the scheduler decides thread B should run:

1. Enter the kernel

   A timer interrupt, a blocking system call or some other event brings the core into kernel mode. The thread’s user-level registers are stored first, so that the kernel can use them.

2. Choose the next thread

   The scheduler looks at the run queue and picks B. If nobody else is runnable there is no switch at all, and A just continues.

3. Save thread A

   A’s stack pointer, program counter and registers go to its kernel records, which are part of the process control block idea from lesson 3.1. A becomes ready (or blocked, if it was waiting).

4. Restore thread B

   B’s saved stack pointer and registers are loaded. If B belongs to another process, its page table is loaded as well.

5. Run thread B

   The kernel returns to user mode and B continues from exactly the instruction where it stopped. B cannot tell it was ever paused, except by reading a clock.

#What a context contains

The contents of memory are not saved or copied. Stack and heap stay where they are; the thread gets its stack back simply by getting its stack pointer back. That is why the direct cost is small and does not depend on how much memory the thread uses.

#What triggers a switch

Voluntary means the thread stopped because it could not go on. Involuntary means it could have gone on but the scheduler took the core. The difference is your first clue in a slowdown: lots of involuntary switches means too little CPU for the runnable threads; lots of voluntary switches means threads are waiting on I/O or locks.

The notes also list “interrupt” as a trigger. An interrupt enters the kernel, and it can lead to a switch when the thing it handled wakes a more urgent thread (as above), but on its own it is not a context switch.

#What a switch really costs

The notes say switching wastes CPU, hurts cache locality and affects the TLB. Those are three very different sizes:

These are ranges, not constants: they depend on the CPU, kernel settings and how much memory the thread touches. Measure yours (see “Try it yourself”). What matters is the indirect cost usually exceeds the direct cost, and the two together are small only compared with a long run between switches. Say each switch costs 25 µs in all:

#Thrashing

Thrashing here means the machine spends so much time switching and refilling caches that useful work falls, and adding threads makes it fall further. You see the CPU at 100%, a huge cs count in vmstat and throughput going down as load goes up. (Memory thrashing, from paging, is a different thing with the same name: see lesson 3.4.)

It comes in two forms, and the fixes are not the same:

• Many short wake-ups. Each thread wakes, does a very small job and blocks: thousands of times a second. Overhead eats the CPU. Fix: do more work per wake-up, using batching, an event loop or fewer threads.
• Many runnable threads. A thousand CPU-bound threads on 4 cores: each slice is still a few milliseconds, so the switch overhead looks modest, but each thread waits 250 × 3 ms = 750 ms for its next turn and always starts with cold caches. Fix: run about as many as you have cores, and let the rest wait in a queue. That queue is the thread pool, and this is the reason the notes point to it.

A third pattern looks similar: a lock convoy. Threads queue up on one busy lock, and every hand-over needs a switch, so throughput falls as threads are added, often with the CPU not even full.

#The scheduler: who runs next

The scheduler is the part of the kernel that decides which runnable thread each core runs, and for how long. On Linux it schedules threads individually: a process with one thread is just the simple case. (A thread blocked on I/O is not in the picture at all until it wakes.)

• Run queue. Each core has its own queue of ready threads. Separate queues keep scheduling fast, since cores do not fight over one shared list. Idle cores pull work from busy ones to balance the load, which costs cache warmth.
• Time slice. How long a thread may run while others wait: on a normal server, a few milliseconds. It is a compromise: short slices mean a lower wait for everyone but more switching, and long slices mean the reverse.
• Preemption. The kernel can take the core from a thread without its cooperation, because a timer interrupt lands it back in the kernel. This is why one buggy infinite loop does not freeze a machine.

#From CFS to EEVDF

CFS, the Completely Fair Scheduler, was the Linux default from kernel 2.6.23 in 2007. The idea is simple. Every thread has a virtual runtime: the CPU time it has used, scaled by its priority. CFS always runs the thread with the lowest virtual runtime, the one furthest behind. It kept runnable threads sorted in a red-black tree to find it quickly.

EEVDF, Earliest Eligible Virtual Deadline First, replaced it in kernel 6.6 (late 2023). It keeps the virtual-time fairness and changes how the next thread is chosen, in two steps:

1. Eligible: only threads that have not had more than their fair share (lag at or above zero) can be picked.
2. Earliest virtual deadline: among those, pick the one whose deadline is first, where the deadline is its virtual runtime plus its requested slice, scaled by weight.

The gain: a thread can ask for a shorter slice and get shorter, more frequent turns, which lowers its latency without increasing its share. CFS could only approximate this with many tweaks. Very few engineers tune the scheduler, but the names come up in interviews and in kernel changelogs, and you should know “CFS” is out of date for the choice of next thread. The term still survives in CFS bandwidth control, the mechanism behind container quotas later in this lesson. Since kernel 6.12 there is also sched_ext, which lets you load a custom scheduler written in eBPF.

#Priorities

• Nice runs from −20 (favoured) to 19. It changes a thread’s weight, and so its share under contention: at nice 0 the weight is 1024, and at nice 5 it is 335, so those two split a core about 75% to 25%. Each step is roughly 10% of CPU between competing threads. If nobody competes, a nice 19 thread still gets the whole core.
• Real-time classes (SCHED_FIFO, SCHED_RR, SCHED_DEADLINE) always run before normal threads. They are for audio or control loops. Linux normally keeps about 5% back for normal threads, so a spinning real-time thread does not lock up the core completely, but ordinary services should stay in the normal class.

#CPU-bound and I/O-bound work

Every design decision in this lesson depends on what your threads do with their time:

Most servers are mostly I/O-bound, so do not assume. A service is rarely all one kind, and the mix can be measured: pidstat -u shows how much CPU each thread uses, and a thread using 5% of a core spends the other 95% waiting.

The dangerous case is a mixture. A few CPU-heavy tasks in the same pool as quick I/O handlers fill the run queues, and the quick handlers then sit ready but not running. CPU charts show “100%”, and the p99 latency of cheap requests rises for no visible reason. Separate them into their own pool, process or service.

#Why more cores do not always help: Amdahl’s law

Parallelism only speeds up the part of the work that can run in parallel. Amdahl’s law (1967) puts a number on it. If a fraction s of the work must run serially, the speedup on N cores is:

Speedup(N) = 1 / (s + (1 − s) / N)
Limit as N grows = 1 / s

So a job that is 95% parallel can never run more than 20 times faster, and 64 cores give only 15.4×. The serial part of the work sets the ceiling, however small it looks.

On a server, “serial” usually does not mean one long job. It means anything each request must do one at a time: holding a global lock, one row every request updates, a single-threaded log writer, one connection to a shared service. Handle requests independently and they scale almost linearly, because this is a case of more work rather than the same work faster (Gustafson’s observation). Share one lock and you are back under Amdahl’s law: if a request holds it for 25% of its time, four cores is the end of the road.

Real systems can do worse than flat. Cores also spend time coordinating: fighting over a cache line, waking each other up, keeping data in step. The Universal Scalability Law adds that cost, and it grows with the number of pairs of cores, so throughput can reach a peak and then fall. This is why teams load test up to real size and why vertical scaling has an end long before the machine catalogue does.

#Cores, hyper-threads and vCPUs

nproc counts logical CPUs, not cores, so read it with care:

• Physical core: its own execution units and its own L1 and L2 caches.
• Hyper-thread (SMT): one physical core presents itself as two logical CPUs by keeping two sets of registers. Its execution units are shared. When one thread is waiting on memory the other can use them, so the pair gives a modest gain, often about 10–30%, and sometimes nothing, rather than double.
• vCPU: what a cloud sells you. On many x86 instance types one vCPU is one hyper-thread, so 2 vCPUs are often one physical core; on AWS Graviton one vCPU is a full core. Chapter 4 covers how hypervisors share them, including steal time.

Run lscpu and read “Thread(s) per core”. If a CPU-bound benchmark scales well up to 8 and then gets much flatter on a “16 CPU” machine, you are probably seeing hyper-threads. Some latency-sensitive and security-sensitive systems turn SMT off.

#When the language gets in the way: Python’s GIL

CPython, the standard Python, has a Global Interpreter Lock (GIL): a thread must hold it to run Python bytecode, so only one thread does so at a time, however many cores there are. It keeps the interpreter’s internals, such as reference counts, simple and safe. The result:

This is why Python web servers such as Gunicorn run several worker processes: one for each core, each with its own GIL.

Free-threaded builds make the GIL optional. PEP 703 introduced them as experimental in Python 3.13, and from 3.14 they are officially supported (PEP 779), although the default build still has the GIL. Binaries are usually named python3.14t. Threads there run in parallel with shared memory, at a cost: single-threaded code is slower (about 5–10% on 3.14), C extensions must declare support or the GIL comes back on import, and code that quietly relied on the GIL can now hit real races. It is a deliberate choice, and worth checking for each release.

Other languages have their own versions of this. Standard Ruby has a similar global lock. A Node.js program runs its JavaScript on one thread, as lesson 3.6 shows. Go and Java run threads in parallel with no such lock, and both offer lightweight user-level threads that switch without entering the kernel, also in lesson 3.6.

#Container CPU limits and throttling

Here is the production problem this lesson’s ideas explain best. In Linux a container’s CPU limit is not a smaller number of cores. It is a budget of CPU time per period, enforced by CFS bandwidth control through cgroups:

cpu.max = 200000 100000       # 200 ms of CPU per 100 ms period: "2 CPUs"

(--cpus=2 in Docker and limits.cpu: 2 in Kubernetes write this for you. cgroup v1 used cpu.cfs_quota_us and cpu.cfs_period_us.)

All threads in the container spend from one budget, and they can all run at once. On a large host, 8 busy threads burn 200 ms of budget in just 25 ms of real time. Then the kernel stops every thread in the container for the remaining 75 ms, even with idle cores all around. A request arriving then waits. The signs:

• p99 latency has steps of tens of milliseconds, while average CPU looks fine (say 40% of the limit), because a 15-second average cannot see a 100 ms period.
• cpu.stat shows nr_throttled and throttled_usec rising. In Kubernetes, the cAdvisor metric is container_cpu_cfs_throttled_periods_total.
• It gets worse the more threads there are, and with runtimes that size their worker pools from the host’s core count. Java has detected container limits for years, and Go has set its default GOMAXPROCS from the limit since 1.25, but older runtimes and hand-written pools may not. Check, and set the number yourself.

Requests are different from limits. A Kubernetes CPU request becomes a weight (cpu.weight): it decides your share when cores are contested and where the pod is placed, and it never wastes idle CPU. A limit is the hard budget, and it does.

What to do: size CPU-bound worker pools to the limit, not the host. Watch throttling on the same dashboard as CPU. Raise the limit for bursty services, or remove it for latency-critical ones and rely on requests, which is a common choice where you trust your neighbours (where you do not, keep the limits).

#Seeing it on a real machine

vmstat 1 gives you the whole-machine picture:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
14  0      0 812304  90212 410232    0    0     2    18 9120 48211 71 22  7  0  0

There is no universal bad number for cs: tens of thousands per second is normal on a busy multi-core server. What matters is the switches per core, the change from normal, and whether throughput falls.

pidstat -w then shows you who is doing it:

$ pidstat -wt -p 4213 1
   TID   cswch/s nvcswch/s  Command
  4213      0.00      0.00  java
  4218    820.00      3.00  |__http-worker-1
  4240      2.00    410.00  |__report-thread

cswch/s counts voluntary switches, and nvcswch/s involuntary ones. Here http-worker-1 mostly waits (I/O-bound), and report-thread keeps being pushed off the core (CPU-bound and competing).

#In system design

• Say what kind of work it is first. CPU-bound or I/O-bound decides everything after it. An interviewer asking “how many threads?” wants that question back.
• Choose the model per workload:

• Treat serial parts as your scaling limit. A shared lock or single primary database is Amdahl’s law in your architecture. Look for it before adding servers, and split it where you can (sharding, per-core data; Chapter 25 covers load balancing).
• Keep some CPU spare. Near full use, run queues grow fast, so latency grows long before CPU reads 100%. This is the same queueing behaviour lesson 1.2 described.
• Check the budget in containers. Worker counts, garbage collector threads and connection pools should follow the CPU limit.
• Interview angles: “Concurrency vs parallelism?” (structure vs execution; show both on a timeline). “Why can more threads reduce throughput?” (switching, cold caches, queueing, contention). “What does a context switch cost?” (direct 1–5 µs, indirect usually more, and it depends on work between switches). “Python threads for image resizing?” (no; processes). “Our Kubernetes service has 40% CPU and bad p99, why?” (throttling; check nr_throttled). “What limits scaling up?” (Amdahl, then coordination).

#Try it yourself

# Overall picture: watch r, cs, us, sy while you run something heavy
vmstat 1

# Logical CPUs, and whether they are hyper-threads
nproc
lscpu | grep -E 'Thread|Core|Socket|Model name'

# Switch rates per thread of one process (pidstat is in the sysstat package)
pidstat -wt -p <PID> 1

# The counters behind those numbers (one thread, here the main thread)
grep ctxt /proc/<PID>/status

# Measure context switching: two tasks pass a message through a pipe
perf bench sched pipe -l 100000
# The same, pinned to one core, so every message forces a switch
taskset -c 0 perf bench sched pipe -l 100000

# Count switches, migrations and cache misses of a command
perf stat -e context-switches,cpu-migrations,cache-misses -- sleep 5

# See which scheduler era you are on: 6.6 or later means EEVDF
uname -r

# Container: what is the limit, and has it been throttled? (inside, cgroup v2)
cat /sys/fs/cgroup/cpu.max
grep -E 'nr_periods|nr_throttled|throttled_usec' /sys/fs/cgroup/cpu.stat

# Priorities: run a job at low priority, or pin it to cores 0 and 1
nice -n 10 ./batch-job
taskset -c 0,1 ./batch-job

See Amdahl’s law and the GIL for yourself. Save this as gil.py and run it. With a default build, the thread pool takes about as long as the serial run, or longer, and the process pool is much faster on a multi-core machine:

import time
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor


def burn(n: int) -> int:
    total = 0
    for i in range(n):
        total += i * i
    return total


def timed(label: str, run) -> None:
    start = time.perf_counter()
    run()
    print(f"{label:<10} {time.perf_counter() - start:.2f} s")


if __name__ == "__main__":
    N, WORKERS = 5_000_000, 4
    timed("serial", lambda: [burn(N) for _ in range(WORKERS)])
    with ThreadPoolExecutor(WORKERS) as pool:
        timed("threads", lambda: list(pool.map(burn, [N] * WORKERS)))
    with ProcessPoolExecutor(WORKERS) as pool:
        timed("processes", lambda: list(pool.map(burn, [N] * WORKERS)))

Then change the body of burn to time.sleep(1): now the threads overlap perfectly, because the wait releases the GIL.

#Summary

Key takeaways

• Concurrency = tasks in progress at overlapping times (design); parallelism = instructions executing at the same instant (hardware). Concurrency is not limited to one core. Parallelism needs more than one.
• On one core, slicing pure CPU work gives fairness, not speed. Overlapping I/O waits can multiply throughput: 5 ms CPU + 45 ms wait goes from 20 to 200 req/s on the same core.
• A context switch saves one thread’s registers, stack pointer and program counter, and restores another’s. Memory is not copied. A process switch also changes the page table.
• Voluntary switches mean waiting on I/O or locks; involuntary switches mean competing for CPU. A system call or interrupt alone is only a mode switch.
• Direct cost is about 1–5 µs, and cache and TLB effects usually cost more. What matters is the work done between switches. Thrashing is when switching crowds out work, so more threads mean less throughput.
• The Linux scheduler keeps a run queue per core, uses time slices of a few milliseconds and can preempt anyone. CFS ran the lowest virtual runtime; EEVDF (since 6.6) runs the earliest deadline among eligible threads. Nice changes shares, only under contention.
• CPU-bound: threads ≈ cores. I/O-bound: more threads help until the core is full. Mixed: separate them.
• Amdahl: speedup is at most 1 / s; 95% parallel is capped at 20×. On servers the serial part is usually a lock or a shared resource. Coordination can make throughput fall.
• Hyper-threads add perhaps 10–30%, not double; a vCPU is often one hyper-thread.
• Python’s GIL blocks parallel Python bytecode in the default build: use processes for CPU work and threads for I/O. Free-threaded builds exist and are optional.
• A container CPU limit is a time budget per 100 ms, not cores: too many threads burn it early and then everything stops. Watch nr_throttled, not only CPU. Requests are weights, limits are caps.

Self-check: concurrency and context switching32 questions