Headers, Keep-Alive, Compression and CORS

By the end of this lesson you will be able to

• Read any HTTP header block: tell request, response and representation headers apart, and end-to-end from hop-by-hop headers; know why names are case-insensitive and what HTTP/2 pseudo-headers are.
• Use the headers that matter in production correctly: Host, Accept*, Content-Type, Content-Length, Authorization, cookies, X-Forwarded-For and Forwarded, request IDs and traceparent, and the main security headers.
• Explain keep-alive and persistent connections, size idle timeouts across a load balancer and an app so that they never cause 502s, and tell HTTP keep-alive from TCP keepalive.
• Explain how compression is negotiated, choose between gzip, Brotli and zstd with numbers, know what never to compress, and explain dictionary compression and the BREACH attack.
• Explain the same-origin policy and CORS end to end: origins, simple and preflighted requests, every Access-Control-* header, credentials, caching, and the misconfigurations that turn CORS into a data leak.
• Use all of this in design and debugging: where to set headers, how to avoid preflights, and what each typical symptom points to.

#Where this lesson fits

Lesson 1.1 showed the shape of an HTTP message (first line, headers, blank line, body) and a table of everyday headers. Lesson 1.3 taught Set-Cookie, Cookie and their attributes, lesson 1.9 taught HSTS, and lesson 1.10 explained why new TCP connections are slow. Lesson 2.1 followed HTTP from 1.0 to 3, and lesson 2.2 covered methods and status codes.

This lesson is about the metadata around every request and response: the headers, the connection they travel on, how the body is squeezed, and the browser rule that decides which pages may read which responses. All four are things you configure at web servers, load balancers and CDNs every week, and all four cause classic production incidents.

#Header anatomy

A header is a name–value pair of metadata: Name: value. In HTTP/1.1 each one is a text line; together they sit between the first line and the blank line that starts the body.

GET /api/orders?page=2 HTTP/1.1
Host: shop.example.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) … Chrome/141.0 Safari/537.36
Accept: application/json
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: bn-BD, en;q=0.8
Authorization: Bearer eyJhbGciOi…
Cookie: sid=9f2c…; lang=bn

Rules that trip people up:

• Names are case-insensitive. Content-Type, content-type and CONTENT-TYPE are the same header. Values usually are case-sensitive (a token, a cookie).
• Order does not matter between different headers, but it does for repeated ones.
• A header may repeat, and repeated values may be joined with commas: two Accept-Encoding lines equal one line with both values. The famous exception is Set-Cookie, whose values contain commas (in dates), so every cookie needs its own line.
• Quality values (;q=0.8) rank preferences from 0 to 1 in the Accept* headers; q=0 means “never”.
• Custom headers are allowed. The old X- prefix convention was officially deprecated in 2012, but X-Request-ID and X-Forwarded-For are too widespread to rename.

#Four kinds of header

Everything that is not hop-by-hop is end-to-end: it must reach the final recipient unchanged, through every proxy and cache. Hop-by-hop headers describe this link (keep the TCP connection open, the body on this link is chunked), so a proxy must remove them, plus any header whose name appears inside Connection, before forwarding. A proxy is also allowed to add headers of its own, which is how X-Forwarded-For and Via appear.

#Headers in HTTP/2 and HTTP/3

The meaning is identical, but the wire format is not text:

• The request line becomes pseudo-headers starting with a colon: :method, :scheme, :authority (replaces Host) and :path. A response has :status. Pseudo-headers come first.
• All names must be lowercase. An uppercase name makes the message malformed.
• Connection-specific headers are forbidden: Connection, Keep-Alive, Proxy-Connection, Transfer-Encoding and Upgrade. TE may only say trailers.
• Headers are compressed with HPACK (HTTP/2) or QPACK (HTTP/3), so repeating the same 2 KB of cookies on every request costs a few bytes after the first time (lesson 2.1).

Proxies routinely translate: HTTP/2 from browsers to the edge, HTTP/1.1 from the edge to the app.

#Size limits

There is no limit in the standard, but every server has one, and requests over it are rejected before your code runs: Nginx allows 8 KB per header line by default (large_client_header_buffers 4 8k), Apache 8,190 bytes per field, Node.js 16 KB for all headers together. The answer is 431 Request Header Fields Too Large, or 400 with a message such as “Request Header Or Cookie Too Large”. The usual culprit is cookies: every matching cookie rides on every request, including image requests. Limits differ per hop (CDN, load balancer, proxy, app), and the smallest one wins.

#The headers that matter

The everyday table is in lesson 1.1. Here is what each important header really does, and how it goes wrong.

#Who is the client really? X-Forwarded-For and Forwarded

Behind a load balancer, the app’s TCP peer is the load balancer, not the user (lesson 1.7: load balancers are NAT too). So the edge adds the client’s address in a header:

X-Forwarded-For: 203.0.113.7, 198.51.100.10
X-Forwarded-Proto: https
X-Forwarded-Host: shop.example.com
Forwarded: for=203.0.113.7;proto=https;host=shop.example.com, for=198.51.100.10

Each proxy appends the address it saw, so the list grows left to right: the original client first, the last proxy last. Forwarded (RFC 7239) is the standard, structured version of the three X-Forwarded-* headers; both are common.

The trap: the client can send its own X-Forwarded-For, and the proxy appends to it. The leftmost value is whatever the client wrote. Trust only the entries your own proxies added: with one trusted proxy in front, take the rightmost value; with two, the second from the right. Even better, let the outermost proxy overwrite the header. Frameworks have a setting for exactly this (“trusted proxies”, “number of hops”), and getting it wrong breaks rate limiting, geo-blocking, audit logs and IP allowlists.

#Following one request: X-Request-ID and traceparent

A single click may pass through a CDN, a load balancer, an API gateway and five services. To find every log line for it, give the request an ID at the edge and pass it along:

• X-Request-ID (or X-Correlation-ID): a random ID created by the first proxy if the client did not send one, forwarded to every service, written in every log line and echoed in the response so a user can quote it to support.
• traceparent (the W3C Trace Context standard, used by OpenTelemetry): 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 = version, 16-byte trace ID, 8-byte parent span ID, flags (01 = sampled). Each service keeps the trace ID and replaces the parent ID with its own span, so a tracing tool can draw the whole call tree. tracestate carries vendor extras. Chapter 46 builds on this.

#Security headers in brief

Most security headers are one line each and cost nothing, so set them centrally at the edge. Chapter 35 explains the attacks; here is what each header is for.

Also remove what leaks detail: version numbers in Server, and X-Powered-By: Express or PHP banners. The old X-XSS-Protection header is obsolete; use CSP instead.

#Keep-alive and persistent connections

Opening a connection is the most expensive part of a small request: one round trip for TCP, one more for TLS 1.3 (two for TLS 1.2), and then TCP starts with a small congestion window (lesson 1.10). On a 50 ms path that is 100 ms before the first request byte, for a response that may take 5 ms to produce.

• HTTP/1.0 closed the connection after every response. Some clients and servers added an unofficial Connection: keep-alive header to ask for reuse.
• HTTP/1.1 made persistence the default: the connection stays open after a response unless one side sends Connection: close. So Connection: keep-alive in an HTTP/1.1 request is redundant.
• The optional Keep-Alive: timeout=5, max=100 response header is a hint: “I will keep this connection for up to 5 idle seconds, and for at most 100 requests”. Clients may ignore it; it is not part of HTTP/2 or HTTP/3, which forbid it.
• What it saves: the TCP handshake and TLS negotiation on every request after the first, the server CPU those handshakes cost, the sockets left in TIME_WAIT, and slow start on a cold connection.
• The limit in HTTP/1.1: one request at a time per connection (pipelining was never enabled in browsers), so browsers open about 6 connections per host. HTTP/2 multiplexes everything onto one connection per origin; HTTP/3 does the same over QUIC.

#Idle timeouts, and the race that causes 502s

Idle connections hold memory, a file descriptor and TLS state, so every server closes them after an idle timeout, and many also after N requests. Typical defaults:

Trouble starts when two of these meet. A load balancer keeps a pool of connections to each backend and reuses them. If the backend’s timeout is shorter than the load balancer’s, the backend closes an idle connection at the very moment the load balancer sends a new request on it. The request meets a closed socket, the backend answers with a TCP reset, and the load balancer returns 502 Bad Gateway. It hits a small fraction of requests, more at low traffic, and nothing appears in the app’s logs. It is one of the most common causes of “random 502s” behind cloud load balancers.

The rule: along a chain, the side that sends requests must close idle connections first. So the backend’s keep-alive timeout must be longer than the load balancer’s idle timeout (for example 65 s behind a 60 s ALB), and a client’s pool timeout must be shorter than its server’s.

#Connection pools

Every serious client keeps a pool: browsers per origin, HTTP libraries per host, load balancers and reverse proxies per backend, and database drivers per database. Settings worth knowing:

• Maximum connections per host: caps load on the backend; too low and requests queue inside the client.
• Idle timeout: shorter than the server’s, for the reason above.
• Maximum lifetime: recycle connections every few minutes so that new backends behind DNS or a load balancer receive traffic, and long-lived connections do not pin all load to old servers after a scale-out.
• Validation or retry: a pooled connection may have been closed by the other side. Clients retry automatically only idempotent requests that failed before any response byte (lesson 2.2); a POST needs an idempotency key.
• Reverse proxy to backend: Nginx historically did not reuse upstream connections unless configured (keepalive 32; in the upstream block, plus proxy_http_version 1.1 and an empty Connection header). Without that, every request opens a new connection to the app, and under load you run out of ephemeral ports (lesson 1.9).

#HTTP keep-alive is not TCP keepalive

They share a name and nothing else:

Lesson 1.10 covers TCP keepalive and application heartbeats (HTTP/2 PING frames, WebSocket ping) in detail.

#Compression

Text compresses well: HTML, CSS, JavaScript, JSON, SVG and XML are full of repeated words and markup. Compressing them before sending saves bandwidth and makes pages load faster, especially on mobile networks.

#How it is negotiated

1. The client lists what it can decode: Accept-Encoding: gzip, deflate, br, zstd (optionally with q-values, e.g. br;q=1.0, gzip;q=0.8).
2. The server picks one it supports, compresses the body, and labels it: Content-Encoding: br. Content-Length is now the compressed size.
3. It adds Vary: Accept-Encoding so caches keep a separate copy per encoding (lesson 2.4 explains Vary).
4. The browser decompresses and renders. If the client sent no Accept-Encoding, the server should send the body as is (the identity coding).

Content-Encoding is end-to-end and describes the representation; the hop-by-hop Transfer-Encoding (chunked) is a different thing. Compressing request bodies (Content-Encoding on a POST) is possible but rare, and only works if the server explicitly supports it; otherwise expect 415.

#gzip vs Brotli vs zstd

A worked example from this lesson’s animation: a 412 KB JavaScript bundle becomes about 118 KB with gzip and 98 KB with Brotli at level 11. Precompress static files at build time (app.js.br, app.js.gz) and let the server pick the right file; compress dynamic responses on the fly at a moderate level. Many CDNs do all of this for you.

#What not to compress

• Already-compressed formats: JPEG, PNG, WebP, AVIF, GIF, MP4 and other video, MP3, WOFF2 fonts, ZIP, PDF with compressed streams. gzip saves 0–2% and can even grow them, while costing CPU on both ends.
• Tiny responses: below about 1 KB the saving is smaller than one packet, and the compression headers can make it larger.
• Responses that mix a secret with attacker-controlled text, unless you defend against BREACH (below).
• Default traps: Nginx has gzip off by default, and gzip_types covers only text/html until you add application/json, text/css, application/javascript and image/svg+xml. Many “our API is slow on mobile” tickets are uncompressed JSON.

#Compression dictionary transport

Normal compression starts from scratch for every response. Compression dictionary transport (standardised by the IETF in 2025; in Chrome since version 130) lets a previously downloaded file act as the dictionary for the next one:

1. The server sends app.v1.js with Use-As-Dictionary: match="/app.*.js". The browser stores it as a dictionary for matching URLs.
2. After a deploy, the browser requests app.v2.js with Available-Dictionary: :hash-of-v1: (the SHA-256 hash of the stored file) and adds the dictionary codings dcb (Brotli) and dcz (zstd) to Accept-Encoding.
3. If the server has v1, it compresses v2 against it and answers Content-Encoding: dcz: often only the changed bytes, a few KB instead of 100. Responses must also vary by Available-Dictionary.

It shines for frequently deployed JavaScript bundles and for API responses that share a big common structure (a custom dictionary built from typical responses).

#When compression leaks secrets: CRIME and BREACH

Compression finds repeated strings and replaces the second copy with a short back-reference. If an attacker can put a guess into the same compressed message as a secret, a correct guess makes the message shorter, and the length is visible even through TLS.

• CRIME (2012) attacked compression at the TLS level (and SPDY’s header compression) to steal cookies. TLS compression was then disabled everywhere and TLS 1.3 removed it; HPACK was designed so that header compression does not leak this way.
• BREACH (2013) attacks HTTP body compression. It needs three things together: the body is compressed, it contains a secret (a CSRF token), and it reflects attacker-controlled input (a search term in the URL). The attacker’s page makes the victim’s browser send thousands of guesses and watches response sizes on the network.

Defences, any of which breaks the attack: mask secrets per response (XOR the token with a fresh random value each time, as Django and Rails do), SameSite cookies so cross-site requests arrive logged out, do not compress responses that mix secrets with reflected input, and rate-limit. Nobody disables compression site-wide because of BREACH.

#CORS end to end

#The same-origin policy

The browser runs code from many sites side by side, all with the user’s cookies. Without a rule, a script on any page you visit could call your bank’s API with your cookies and read your balance. The same-origin policy (SOP) forbids that: a script may freely read responses only from its own origin.

An origin is scheme + host + port. Compared with https://shop.example.com (port 443 implied):

A site is broader: scheme plus registrable domain. shop.example.com and api.example.com are same-site but cross-origin. That is why SameSite cookies (lesson 1.3, which works per site) are still sent between them, while the same-origin policy (per origin) still applies.

The policy is narrower than people think. It does not stop a page from sending requests elsewhere or embedding other origins: <img>, <script>, <link rel=stylesheet>, <iframe>, <video> and HTML form submissions all work cross-origin, as they did in the 1990s. What it blocks is reading: a script cannot see the response of a cross-origin fetch, the pixels of a cross-origin image drawn on a canvas, or the inside of a cross-origin frame.

#CORS: the server opts in

Cross-Origin Resource Sharing (CORS) is the controlled exception. The browser tells the server which origin is asking (the Origin request header, set by the browser and impossible for page scripts to forge), and the server answers with Access-Control-* headers saying who may read. If the answer does not allow the page’s origin, the browser hides the response from the script.

So the same-origin policy is the wall, and CORS is the door the server decides to open. The notes’ example: a front-end at https://shop.example.com calling https://api.example.com. The browser checks whether the API has allowed that origin; only then may JavaScript read the response.

#Simple requests

Some requests were already possible with plain HTML forms before CORS existed, so CORS sends them directly, with no extra round trip. A request is simple when:

• the method is GET, HEAD or POST;
• it sets only CORS-safelisted headers: Accept, Accept-Language, Content-Language, Content-Type, Range;
• any Content-Type is one of application/x-www-form-urlencoded, multipart/form-data or text/plain.

The browser sends the request with Origin, the server answers, and the browser then checks Access-Control-Allow-Origin against the page’s origin. Two consequences surprise people:

1. A blocked request still reached the server. The handler ran, the database was queried, the 200 was logged. Only the page’s script was denied the result, with a generic TypeError: Failed to fetch; the reason is in the developer console, deliberately invisible to code.
2. A simple POST can change data even when blocked. That is why CORS does nothing against CSRF; that needs SameSite cookies, CSRF tokens or Origin checks (lesson 1.3, Chapter 35).

By default a cross-origin script can read only the safelisted response headers (Cache-Control, Content-Language, Content-Length, Content-Type, Expires, Last-Modified, Pragma). To read others, such as X-Request-ID, ETag or Location, the server lists them in Access-Control-Expose-Headers.

#Preflighted requests

Anything else (a PUT, PATCH or DELETE; an Authorization or custom header; Content-Type: application/json) could do things an old form never could. Old servers never expected such requests from other sites, so the browser first asks permission with an OPTIONS request, the preflight:

OPTIONS /users/42 HTTP/1.1
Host: api.example.com
Origin: https://shop.example.com
Access-Control-Request-Method: PUT
Access-Control-Request-Headers: authorization, content-type

HTTP/1.1 204 No Content
Access-Control-Allow-Origin: https://shop.example.com
Access-Control-Allow-Methods: GET, POST, PUT, DELETE
Access-Control-Allow-Headers: authorization, content-type
Access-Control-Max-Age: 600
Vary: Origin

The preflight must get a 2xx answer that allows the origin, the method and every listed header. Only then is the real PUT sent, and its response needs its own Access-Control-Allow-Origin too. The preflight never carries cookies, Authorization or a body, which leads to the most common CORS bug: an authentication middleware that runs before the CORS handler and answers the preflight with 401.

Preflights cost a round trip. Access-Control-Max-Age lets the browser cache the answer, but only per origin and URL, and browsers cap it (Chrome at 2 hours, Firefox at 24 hours; without the header, 5 seconds). A REST API with IDs in the path (/users/42, /users/43) pays a preflight for every new URL.

Access-Control-Allow-Origin cannot hold a list. To allow several origins, the server checks the request’s Origin against an allowlist and echoes the one that matched, adding Vary: Origin so caches do not serve one origin’s answer to another.

#Credentials and why * cannot go with cookies

By default fetch sends no cookies on cross-origin requests. The page must opt in with credentials: 'include' (or withCredentials on the old XMLHttpRequest). Then the response is readable only if:

• Access-Control-Allow-Credentials: true is present, and
• Access-Control-Allow-Origin names the exact origin, never *. With credentials, * in Allow-Headers, Allow-Methods and Expose-Headers is also treated as a literal name, not a wildcard.

The reason is simple: * means “any site may read this”. Combined with the user’s cookies it would let every website read every logged-in user’s data, so the specification forbids the combination. * is fine, and correct, for truly public, uncredentialed data: a public API, fonts or images on a CDN.

Cookies also have their own rules. For a same-site API (api.example.com from shop.example.com) a SameSite=Lax cookie is sent. For an API on another site, the cookie needs SameSite=None; Secure, and browsers that block third-party cookies may still refuse it. That is one more reason to keep a web app’s API on its own site, or better on its own origin.

#CORS is a browser protection, not server security

The notes’ best practices, with the reasons:

• Allow specific trusted origins, compared as whole strings against a fixed list.
• Do not use * in production for anything that is not public.
• Credentials cannot be combined with a wildcard origin, by design.
• Cookie-based authentication still needs CSRF protection, because simple requests are sent anyway.
• Do not treat CORS as a firewall.

#Common misconfigurations

#Designing around CORS

• Avoid it entirely: serve the API under the same origin through a reverse proxy or CDN path rule (https://shop.example.com/api/… → the API service). No preflights, no CORS configuration, first-party cookies.
• Keep calls simple where it is natural: a GET without custom headers needs no preflight.
• Set a sensible Access-Control-Max-Age (for example 600–7200 s) and, where possible, avoid unique IDs in the paths of hot preflighted endpoints, or send them as query parameters on GET instead.
• Configure CORS in one place, at the API gateway or in one middleware, from a reviewed allowlist.

#Headers, keep-alive, compression and CORS in system design

• Set cross-cutting headers at the edge. Security headers, X-Request-ID, X-Forwarded-* hygiene, compression and CORS are easiest to get right once, at the CDN, load balancer or API gateway, instead of in every service.
• Size timeouts as a chain. Client pool < load balancer idle timeout < backend keep-alive timeout; and drain backends with Connection: close (or HTTP/2 GOAWAY) during deploys.
• Reuse connections everywhere: browser to edge (HTTP/2 or HTTP/3), edge to backend (upstream keep-alive), service to service (client pools), app to database (connection pool, Chapter 18).
• Compress text, not media, precompress static files with Brotli, use gzip as the fallback, consider zstd for dynamic responses, and always send Vary: Accept-Encoding.
• Treat headers as untrusted input: X-Forwarded-For, Host and User-Agent are all written by the client unless your infrastructure overwrote them.
• Same origin beats CORS for your own web front-end; use CORS for genuinely separate consumers.

#Try it yourself

# See every request and response header (-v), or only the response headers (-I)
curl -v https://example.com -o /dev/null
curl -sI https://example.com

# Is compression on? Ask for Brotli or gzip and look at Content-Encoding and Vary
curl -sI -H 'Accept-Encoding: br, gzip' https://example.com | grep -i -E 'content-encoding|vary|content-length'

# Compare compressed sizes of the same resource
curl -s -o /dev/null -w '%{size_download} bytes\n' https://example.com/
curl -s -o /dev/null -w '%{size_download} bytes\n' -H 'Accept-Encoding: gzip' https://example.com/

# Keep-alive: two URLs in one curl call reuse the connection ("Re-using existing connection")
curl -v https://example.com/ https://example.com/ -o /dev/null -o /dev/null 2>&1 | grep -i -E 're-using|connected to'

# Simulate a CORS preflight against your own API and read the Access-Control-* answer
curl -i -X OPTIONS https://api.example.com/users/42 \
  -H 'Origin: https://shop.example.com' \
  -H 'Access-Control-Request-Method: PUT' \
  -H 'Access-Control-Request-Headers: authorization, content-type'

# Test for a reflecting misconfiguration: does the API approve an origin it should not?
curl -sI https://api.example.com/me -H 'Origin: https://evil.example.net' | grep -i access-control

In the browser, open Developer tools → Network, click a request and read Headers: request and response headers, the Origin header on cross-origin calls, and the OPTIONS preflight listed as a separate row (filter by method). The Console shows the exact CORS reason that your JavaScript cannot see. For HTTP/2 traffic, the pseudo-headers appear as :authority, :method, :path and :scheme.

#Summary

Key takeaways

• Headers are case-insensitive name–value metadata: request, response and representation headers. End-to-end headers must reach the recipient; hop-by-hop headers (Connection, Keep-Alive, Transfer-Encoding, TE, Upgrade) are removed by every proxy. HTTP/2 and HTTP/3 use lowercase names and pseudo-headers (:method, :scheme, :authority, :path, :status).
• Important headers: Host, User-Agent, Accept*, Content-Type, Content-Length, Authorization, Cookie/Set-Cookie, X-Forwarded-For/Forwarded (trust only what your proxies add), X-Request-ID/traceparent, and security headers (HSTS, CSP, nosniff, frame-ancestors, Referrer-Policy). Hide server versions; watch header size limits (431).
• Keep-alive: HTTP/1.1 connections persist by default; Connection: close ends them; Keep-Alive: timeout=5, max=100 is only a hint and is forbidden in HTTP/2 and HTTP/3. Reuse saves handshakes, CPU and slow start.
• Idle timeouts: the requester must close first, so the backend timeout must exceed the load balancer’s (65 s > 60 s), or the race produces random 502s. Pools need sane sizes, idle timeouts, lifetimes and idempotent-only retries. HTTP keep-alive ≠ TCP keepalive.
• Compression: Accept-Encoding → Content-Encoding + Vary: Accept-Encoding. gzip 70–80% smaller, universal; Brotli 15–25% smaller than gzip, browsers offer it only over HTTPS; zstd fast, in Chromium and Firefox since 2024. Do not compress images, video, fonts or archives. Dictionary transport sends only diffs; BREACH needs compression + a secret + reflected input.
• Same-origin policy: an origin is scheme + host + port; scripts may not read cross-origin responses, though they may send requests and embed resources. CORS lets the server opt in with Access-Control-* headers.
• Simple requests (GET/HEAD/POST, safelisted headers and content types) go straight out and are hidden afterwards; everything else gets an OPTIONS preflight without credentials. Max-Age caches it per URL (Chrome ≤ 2 h). Credentials need an exact origin, Allow-Credentials: true, and Vary: Origin; never *.
• CORS is a browser protection, not server security: curl ignores it, it does not stop CSRF, and reflecting any origin or allowing null leaks user data. Prefer serving the API from the same origin.

Self-check: headers, keep-alive, compression and CORS35 questions