CachingCaching

Part of the Backend Roadmap knowledge base.

Overview

A cache is a small, fast store that holds copies of data so that future requests for that data can be served without redoing the expensive work that produced it — a database query, a remote API call, a template render, an image resize, a cryptographic computation. Caching is the single highest-leverage performance technique in backend engineering: it trades a little memory and a little complexity for order-of-magnitude reductions in latency, load, and cost.

The intuition comes from a hard fact of computing: latency has a hierarchy, and each level is roughly 10–100× slower than the one above it.

Every layer of caching is an attempt to move data up this hierarchy — closer to the consumer, into faster storage — so you pay the slow cost once and amortize it over many reads.

But caching is famously treacherous. Phil Karlton’s line — “There are only two hard things in Computer Science: cache invalidation and naming things” — is a joke that every senior engineer has lived. A cache is a second copy of the truth, and the moment you have two copies you have a consistency problem. This note covers where caches live, how to use them correctly, and how to avoid the failure modes (staleness, stampedes, unbounded memory growth) that turn a cache from an asset into an outage.

Related notes: Web Servers (reverse proxies and edge caching), Database Design & Scaling (what you are usually caching in front of), and Scalability & Reliability (caching as a load-shedding and availability tool).

Fundamentals

Why cache — the latency/cost tradeoff

Every cache is a bet: the cost of storing and serving a copy is far less than the cost of recomputing the original, and the copy will be read enough times to pay off. Three benefits flow from a good cache:

• Lower latency. Serving from memory instead of re-querying a database turns a 10 ms operation into a 0.5 ms one — often the difference between a snappy and a sluggish product.
• Higher throughput / lower load. Every cache hit is a request the origin (database, upstream service) never sees. A cache with a 90% hit rate reduces origin load by 10×, which often means you avoid buying more database replicas or scaling out application servers.
• Lower cost. Cheap RAM offloading expensive database CPU, or a CDN edge absorbing bandwidth that would otherwise hit your origin, is directly cheaper.

The cost side of the bet:

• Memory / storage for the copies.
• Complexity: invalidation logic, an extra dependency to operate and monitor, new failure modes.
• Staleness risk: a cache can serve data that is out of date. Whether that is acceptable is a product decision, not just an engineering one.

What makes a good cache candidate

Not all data should be cached. The ideal candidate is:

Good examples: product catalog pages, a user’s profile, computed leaderboards, feature flags, session data, rendered fragments, results of expensive aggregations, geocoding/API lookups.

Poor examples: a stock-trading price at the instant of execution, a one-time password, data with a unique key per request and no reuse.

Cache hit ratio — the metric that matters

The hit ratio is hits / (hits + misses). It is the single most important cache metric because the value of a cache is almost entirely determined by it. A cache with a 50% hit rate roughly halves origin load; a 95% hit rate reduces it 20×. Always measure it. A cache with a poor hit ratio is pure overhead — you pay the memory and the lookup cost and rarely get the payoff. Common causes of low hit ratio: TTL too short, keyspace too large/unique, evictions too aggressive (cache too small), or caching data that is not actually re-read.

Key Concepts

The layers of caching

Caching is not one thing in one place. A single request can pass through half a dozen caches, each closer to the origin. The rule of thumb: the best cache is the one closest to the user, because it eliminates the most work and network.

• Client/browser cache is free and the fastest possible — a cache hit here means zero network. You control it with HTTP headers (below).
• CDN/edge cache serves content from a location physically near the user, cutting round-trip time and shielding your origin. Ideal for static assets, and increasingly for dynamic-but-cacheable content.
• Reverse-proxy cache (Nginx/Varnish) sits in front of your application and can serve whole responses without ever waking your app code.
• Application cache (Redis/Memcached, or an in-process map) is where backend engineers do most deliberate caching — caching database query results, session objects, and computed values.
• Database cache is mostly automatic (the buffer pool keeps hot pages in RAM), but you also opt into materialized views and result caching.

These compose. A well-architected system caches at multiple layers so that most requests are answered before they reach the expensive origin.

HTTP caching

HTTP has a rich, standardized caching model built into the protocol. Getting these headers right is often the cheapest, biggest win available, because it lets browsers and CDNs cache on your behalf.

There are two conceptual mechanisms:

1. Freshness — how long a response may be reused without contacting the server (Cache-Control, Expires).
2. Validation — how a cache checks whether a stale copy is still good with a cheap conditional request (ETag, Last-Modified).

Cache-Control

The primary, modern header. It is a list of directives.

Cache-Control: public, max-age=3600
Cache-Control: private, max-age=0, must-revalidate
Cache-Control: no-store
Cache-Control: public, max-age=31536000, immutable
Cache-Control: public, max-age=60, stale-while-revalidate=300

A common, powerful pattern is cache-busting via content hashes: name assets app.9f3a1c.js and serve them with Cache-Control: public, max-age=31536000, immutable. Because the filename changes when the content changes, you get aggressive caching and instant updates — the new HTML simply references a new filename.

ETag and If-None-Match (validation)

An ETag is an opaque version identifier (often a hash) the server attaches to a response. On a later request the client sends it back; if the resource is unchanged, the server replies 304 Not Modified with an empty body — saving bandwidth and render cost while still confirming freshness.

# First response
HTTP/1.1 200 OK
ETag: "a1b2c3d4"
Cache-Control: no-cache
Content-Length: 24506

<full body>

# Later conditional request from the client
GET /api/products/42 HTTP/1.1
If-None-Match: "a1b2c3d4"

# Server: unchanged
HTTP/1.1 304 Not Modified
ETag: "a1b2c3d4"
# (no body)

ETags come in two flavors: strong ("abc", byte-for-byte identical) and weak (W/"abc", semantically equivalent). Use weak when trivial byte differences (whitespace, gzip variance) shouldn’t count as a change.

Last-Modified and If-Modified-Since

An older, coarser validator based on timestamps (1-second resolution). The server sends Last-Modified; the client returns If-Modified-Since; the server answers 200 with a fresh body or 304 if unchanged. ETag is preferred (more precise), but Last-Modified is a fine fallback, especially for files.

HTTP/1.1 200 OK
Last-Modified: Wed, 15 Jul 2026 10:00:00 GMT

GET /page HTTP/1.1
If-Modified-Since: Wed, 15 Jul 2026 10:00:00 GMT

HTTP/1.1 304 Not Modified

Expires

The legacy freshness header — an absolute date (Expires: Wed, 21 Oct 2026 07:28:00 GMT). Superseded by Cache-Control: max-age, which wins when both are present. Only relevant for old clients.

Vary

Vary tells caches that the response depends on specific request headers, so they must key the cache entry by those headers too.

Vary: Accept-Encoding
Vary: Accept-Encoding, Accept-Language

Without Vary: Accept-Encoding, a cache might serve a gzipped body to a client that can’t decompress it. Be careful: Vary: Cookie or Vary: User-Agent can destroy your hit ratio by fragmenting the cache into near-unique entries. Vary only on what genuinely changes the representation.

The 304 revalidation flow

Client                          Cache (CDN/browser)                 Origin
  |-- GET /resource ------------->|
  |                               |  fresh? (within max-age)
  |                               |    yes -> serve from cache (no origin hit)
  |                               |    no  -> revalidate:
  |                               |-- GET /resource ----------------->|
  |                               |   If-None-Match: "etag"           |
  |                               |                          unchanged?
  |                               |<-- 304 Not Modified --------------|
  |<-- 200 (served from cache) ---|   (refresh freshness, keep body)
  |                               |   OR
  |                               |<-- 200 + new body ----------------|
  |<-- 200 (new body) ------------|

Freshness avoids the origin entirely; validation still contacts the origin but avoids re-sending the body. Both are wins over an uncached 200.

Application caching patterns

When you cache in your own code (typically with Redis or Memcached), there are four canonical patterns. The difference is who writes to the cache and when.

Cache-aside (lazy loading) — the default

The application manages the cache directly. On read: check cache; on miss, load from DB, populate cache, return. On write: update DB, then invalidate (or update) the cache entry.

def get_user(user_id):
    key = f"user:{user_id}"
    cached = redis.get(key)
    if cached is not None:
        return json.loads(cached)          # cache hit
    user = db.query("SELECT * FROM users WHERE id = %s", user_id)  # miss
    redis.set(key, json.dumps(user), ex=300)  # populate with 5-min TTL
    return user

def update_user(user_id, data):
    db.update("users", user_id, data)
    redis.delete(f"user:{user_id}")        # invalidate; next read repopulates

• Pros: simple, resilient (a cache outage just means more DB load, not failure), only caches what is actually read.
• Cons: each miss pays the full latency; there is a small window after a write where a concurrent reader can repopulate stale data (mitigate by deleting after the DB commit, or with versioned keys).

Read-through

The cache itself sits in front of the DB and loads on miss, via a provider or library. The application only ever talks to the cache; the cache knows how to fetch from the origin.

• Pros: application code is simpler (no explicit load logic); load logic is centralized.
• Cons: requires a cache that supports it (or a wrapper library); first read is still slow.

Write-through

On write, the application writes to the cache and the DB synchronously (cache first, which then writes to the DB, or both). The cache is always consistent with the DB for written keys.

def set_config(key, value):
    redis.set(f"config:{key}", value)      # write cache
    db.upsert("config", key, value)        # and DB, synchronously

• Pros: cache is never stale for written data; reads after writes are fast.
• Cons: every write pays cache + DB latency; caches data that may never be read (wasteful if writes outnumber reads).

Write-behind (write-back)

On write, update the cache immediately and return; the DB is updated asynchronously (batched, after a delay).

• Pros: fastest writes, absorbs write bursts, can batch/coalesce DB writes.
• Cons: risk of data loss if the cache dies before flushing; complex; eventual consistency. Used for high-write workloads (metrics, counters, analytics) where some loss is tolerable.

Patterns comparison

Cache-aside plus TTL is the workhorse for the vast majority of backends. Reach for the others only when you have a specific reason.

Eviction policies and memory management

A cache has finite memory. When it fills, it must decide what to remove. This is the eviction policy.

TTL (time-to-live) is orthogonal to and combined with the above: every entry can carry an expiry, and expired entries are removed regardless of the eviction policy. TTL is your first line of defense against staleness — even if you never explicitly invalidate, data self-heals after the TTL. Choose TTLs by how much staleness the data can tolerate: seconds for volatile data, hours or days for stable reference data.

Redis exposes eviction via maxmemory + maxmemory-policy:

maxmemory 2gb
maxmemory-policy allkeys-lru      # evict any key by approximated LRU
# Other policies:
#   noeviction         -> return errors on write when full (default)
#   allkeys-lru        -> LRU across all keys
#   allkeys-lfu        -> LFU across all keys (Redis 4+)
#   volatile-lru       -> LRU only among keys with a TTL set
#   volatile-ttl       -> evict keys with the nearest expiry first
#   allkeys-random     -> random

A key operational decision: with noeviction, a full cache starts rejecting writes (fail loud); with allkeys-lru, it silently makes room (fail soft). Choose deliberately. Also set per-key TTLs even under LRU, so nothing lives forever by accident.

The hard problems

Cache invalidation

Keeping the cache consistent with the source of truth is the central difficulty. Strategies:

• TTL expiry — the simplest: let entries expire and repopulate. Accepts bounded staleness. Almost always your baseline.
• Explicit invalidation — on a write, delete/update the affected keys (cache-aside write path). Precise but you must know every key affected by a change, which is easy to get wrong (e.g. a product update should also bust the category-listing cache).
• Write-through — keep cache and DB in lockstep on write (see above).
• Versioned / key-namespaced invalidation — embed a version in the key (user:42:v7) and bump the version to invalidate whole groups at once, letting old entries age out via TTL. Great for “invalidate everything related to X” without tracking individual keys.
• Event-driven invalidation — publish change events (e.g. via Redis pub/sub, a message queue, or DB change-data-capture) and have caches subscribe and invalidate. Scales to many caches/instances.

Prefer delete over update on writes: deleting is idempotent and avoids races where two writers set the cache in the wrong order. The next read repopulates from the source of truth.

Cache stampede (thundering herd)

When a popular (hot) key expires or is missing, many concurrent requests all miss at once and simultaneously hammer the origin to recompute the same value — potentially overwhelming the database. This is a classic self-inflicted outage: the cache that was protecting the DB suddenly stops, and the full crowd rushes in.

Mitigations:

• Locking / mutex (single-flight): the first request to miss acquires a lock (e.g. SET key:lock 1 NX EX 10) and recomputes; others wait briefly and then read the freshly populated value instead of also recomputing.
• Request coalescing / single-flight in-process: within one process, deduplicate concurrent identical loads so only one hits the origin and the rest await its result (Go’s singleflight, or a per-key in-flight map).
• Jittered / randomized TTL: never expire many related keys at the same instant. Add randomness — ttl = base + random(0, base*0.1) — so expirations spread out and no synchronized wave forms.
• Early / probabilistic recomputation: refresh a hot key before it expires (e.g. XFetch — probabilistically recompute as expiry nears), so it never actually goes cold.
• stale-while-revalidate: serve the stale value immediately and refresh in the background, so users never wait on a miss.

# Cache-aside with a lock to prevent stampede
def get_expensive(key):
    val = redis.get(key)
    if val is not None:
        return val
    # try to become the single recomputing worker
    if redis.set(f"{key}:lock", "1", nx=True, ex=10):
        try:
            val = compute_expensive()          # only one caller does this
            # jittered TTL to avoid synchronized expiry later
            redis.set(key, val, ex=300 + random.randint(0, 30))
            return val
        finally:
            redis.delete(f"{key}:lock")
    else:
        time.sleep(0.05)                       # someone else is computing
        return redis.get(key) or compute_expensive()  # brief wait, then read

Two adjacent problems worth naming:

• Cache penetration: requests for keys that don’t exist in the cache or the DB (e.g. querying random non-existent IDs) always miss and hit the DB. Mitigate by caching the negative result (a “not found” sentinel with a short TTL) or a Bloom filter to reject impossible keys.
• Cache avalanche: a large fraction of the cache expires or the cache server dies all at once, dumping full load on the origin. Mitigate with jittered TTLs, high availability for the cache, and origin rate-limiting / circuit breakers.

Staleness and consistency

A cache is eventually consistent by nature. Decide explicitly how much staleness each dataset tolerates, and encode it as TTL. For data that must be strongly consistent (do not serve stale), either don’t cache it, use write-through, or read from the source of truth on the critical path. Beware the read-after-write anomaly: a user updates their profile, then immediately reads it and sees the old cached value. Fixes include invalidating on write, writing through, or briefly reading from the primary after a user’s own writes.

CDN caching: static vs dynamic content

A CDN caches responses at edge locations around the world.

• Static content (images, CSS, JS, fonts, videos) is the classic CDN use case: immutable, shared across all users, long TTLs, content-hashed filenames. Cache aggressively (max-age=31536000, immutable). This is nearly free performance.
• Dynamic content is trickier but increasingly cacheable:
  • Cacheable-but-dynamic: pages that are the same for many users for a short window (a news homepage, a product page). Use short s-maxage + stale-while-revalidate, and validation (ETag).
  • Personalized: differs per user. Cache the shared shell and fetch personalized fragments separately (edge-side includes, client-side hydration), or don’t cache the personalized parts.
  • Purging: CDNs let you purge/invalidate specific paths or tags when content changes, giving you long TTLs and fast updates.
  • Edge compute (Cloudflare Workers, Lambda@Edge) can generate or assemble responses at the edge, blurring the static/dynamic line.

Key CDN knobs: honor Cache-Control/s-maxage, set sensible defaults, use Vary sparingly, and always version static assets so you never fight the cache with manual purges.

Redis vs Memcached

The two dominant application-cache stores.

For a pure “put this string/JSON behind a key” cache, both are excellent and Memcached’s multithreading can edge ahead at very high throughput. In practice Redis is the default choice for most teams because its data structures and features let one system do many jobs.

Redis is more than a cache

Redis’s data structures make it a general-purpose in-memory toolkit, not just a cache:

• Strings / counters — page views, rate-limit counters (INCR, INCRBY).
• Hashes — store an object’s fields compactly (HSET user:42 name Alice age 30).
• Lists — queues, recent-items feeds (LPUSH/RPOP).
• Sets — membership, tags, unique visitors (SADD, SISMEMBER).
• Sorted sets (ZSET) — leaderboards, priority queues, time-series by score (ZADD, ZRANGE).
• Streams — append-only logs / lightweight event streaming with consumer groups.
• Pub/Sub — real-time messaging, cache invalidation fan-out.
• HyperLogLog — approximate unique counts in tiny memory (PFADD, PFCOUNT).
• Geospatial — radius queries (GEOADD, GEOSEARCH).
• TTL/expiry — distributed locks, session stores, ephemeral tokens.

# A few illustrative commands
SET session:abc123 "{...}" EX 1800          # session with 30-min TTL
INCR page:home:views                         # atomic counter
HSET user:42 name "Alice" plan "pro"         # object as a hash
ZADD leaderboard 4200 "player:7"             # add score
ZREVRANGE leaderboard 0 9 WITHSCORES         # top 10
EXPIRE user:42 300                           # set/refresh a TTL
TTL user:42                                   # inspect remaining TTL
SET lock:job "1" NX EX 10                     # simple distributed lock

This versatility is why Redis often becomes a caching layer, a session store, a rate limiter, a job queue, and a leaderboard engine all at once.

Best Practices

• Cache-aside + TTL is your default. Start there; adopt write-through/write-behind only for a specific, justified need.
• Always set a TTL. Never let entries live forever by accident. TTL is your safety net against unbounded staleness and memory growth.
• Add jitter to TTLs for related keys to prevent synchronized expiry and avalanches.
• Measure the hit ratio (and latency, evictions, memory) per cache. A low hit ratio means the cache is overhead — fix the TTL/keyspace or remove it.
• Prefer delete over update on writes — it’s idempotent and race-safe; let the next read repopulate.
• Design keys carefully. Use a consistent, namespaced scheme (entity:id:field, user:42:profile). Include a version segment when you need group invalidation.
• Never cache sensitive data in shared caches. Use Cache-Control: private/no-store for personal or secret responses; be careful with Vary.
• Protect against stampedes on hot keys with locks/single-flight, and against penetration with negative caching or Bloom filters.
• Treat the cache as optional, not required. The system must still function (slower) if the cache is down — cache-aside naturally gives you this; guard against making the cache a single point of failure.
• Cache at the highest layer you can. Browser/CDN caching beats application caching, which beats no caching. A 304 beats a full response; a full response from the edge beats one from your origin.
• Version static assets by content hash and serve them immutable — aggressive caching with instant updates.
• Make caching observable. Log/emit hit/miss, and expose whether a response was served from cache (e.g. an X-Cache: HIT/MISS header) so you can debug.
• Invalidate deliberately, not hopefully. Know exactly which keys a write affects; when in doubt, use versioned keys or shorter TTLs rather than guessing.

References

• MDN — HTTP caching
• MDN — Cache-Control
• RFC 9111 — HTTP Caching
• Redis documentation
• Redis — Key eviction (maxmemory policies)
• Memcached wiki
• AWS — Caching best practices
• Cloudflare — What is a CDN / how caching works