Intermediate13 min readLevel 2

Caching Fundamentals

Store the answer so you don't have to compute it twice.

Introduction

A cache is a fast, usually in-memory store that keeps copies of expensive-to-produce data so future requests are served quickly. Caching is the single highest-leverage performance tool in system design — and also the source of some of its subtlest bugs.

Why it exists

Databases and computations are comparatively slow and have limited throughput. If the same data is read far more often than it changes, recomputing or re-fetching it every time wastes latency and load. A cache trades a small amount of memory (and some staleness risk) for large gains in latency and capacity.

Analogy

It's like keeping frequently used tools on your workbench instead of walking to the storeroom each time. The bench (cache) is tiny and fast; the storeroom (database) is huge and slow. You only walk to the storeroom when the tool isn't on the bench.

How it works

On each read the application first checks the cache. A cache hit returns immediately; a cache miss falls back to the source of truth, then stores the result for next time. Entries expire via a TTL or are pushed out by an eviction policy (LRU/LFU) when memory is full.

The dominant read pattern is cache-aside: the application owns the logic — check cache, on miss read DB and populate cache. Alternatives shift responsibility to the cache layer (read-through) or change write behavior (write-through writes cache+DB synchronously; write-behind writes cache now and DB asynchronously; write-around writes only the DB and lets reads populate the cache).

The two numbers that matter are hit ratio (fraction of reads served from cache) and the cost of a miss. A high hit ratio is what turns a cache from decoration into a load-bearing part of the system.

Cache-aside

1
Application receives a read request and computes the cache key.
2
It checks the cache. On a hit, it returns the cached value immediately.
3
On a miss, it reads from the database (the source of truth).
4
It writes the result into the cache with a TTL.
5
It returns the value; subsequent reads hit the cache until the TTL expires or the key is invalidated.

Interactive

Architecture

Cache-aside read path

Application

Check first

Cachehit → return

On miss only

Databaseread, then populate cache

Hit: one fast hop. Miss: cache → DB → write back to cache. Use 'Simulate failure' to see the fallback to the DB.

In code

TypeScriptCache-aside with a TTL

import { redis } from "@/lib/redis"
import { db } from "@/lib/db"

const TTL_SECONDS = 300

export async function getProduct(id: string) {
  const key = `product:${id}`

  const cached = await redis.get(key)
  if (cached) return JSON.parse(cached)          // cache hit

  const product = await db.product.findById(id)   // cache miss → source of truth
  if (product) {
    await redis.set(key, JSON.stringify(product), "EX", TTL_SECONDS)
  }
  return product
}

C#Caching an expensive aggregate in .NET

public async Task<DashboardStats> GetStatsAsync(string tenantId)
{
    var key = $"stats:{tenantId}";
    if (_cache.TryGetValue(key, out DashboardStats cached))
        return cached;                              // hit

    var stats = await _repo.ComputeExpensiveStatsAsync(tenantId); // miss
    _cache.Set(key, stats, TimeSpan.FromMinutes(5));
    return stats;
}

In the real world

Where you've seen this

A product page shows the same catalog data to thousands of users per minute but the data changes a few times a day. Caching product records in Redis with a short TTL takes the database from thousands of reads/second to a handful, while a CDN caches the rendered page and static assets even closer to users.

When to use it

Reach for it when

Read-heavy workloads where the same data is requested repeatedly.
Expensive computations or aggregations whose inputs change slowly.
Data that tolerates a little staleness (bounded by a TTL).
Smoothing spikes so the origin isn't hit by every request.

Avoid it when

Write-heavy data that changes on nearly every read (low hit ratio).
Data that must be perfectly fresh and consistent on every read.
Tiny datasets where the DB is already trivially fast.

Trade-offs

Advantages

Large latency reduction on the hot path.
Big drop in load on the origin, raising overall capacity.
Absorbs read spikes and protects slow backends.

Disadvantages

Introduces staleness — the cache can lag the source of truth.
Invalidation is genuinely hard to get right.
Adds a component that can fail or fill up.
Cold caches and stampedes can hammer the origin.

Caching is fundamentally a trade of freshness for speed. A longer TTL means higher hit ratios and less origin load but more staleness; a shorter TTL is fresher but stampede-prone and lower hit ratio. Write-through keeps the cache fresh but slows writes; write-behind is fast but risks losing writes on a crash; cache-aside is simple but has a race window on concurrent write+read. There is no free lunch — you choose where the staleness and complexity live.

Read & write strategies	Cache-aside	Read-through	Write-through	Write-behind
Who loads on miss	Application	Cache layer	n/a (write path)	n/a (write path)
Write latency	DB only	DB only	Cache + DB (slower)	Cache now, DB later (fast)
Freshness	Race window	Race window	Strong	Weak until flushed
Risk	Stale/stampede	Stale/stampede	Slower writes	Lost writes on crash

Common mistakes

Watch out for

Caching without a TTL and never invalidating, so data goes permanently stale.
Treating the cache as the source of truth instead of a copy.
No plan for a cache miss storm when a hot key expires (stampede).
Caching per-user data under a shared key (leaking data across users).
Ignoring what happens when the cache is down.

Failure thinking

What breaks it

What happens if Redis (the cache) goes down?

Every read becomes a miss and falls through to the database, which may suddenly see 10–100x its normal load and fall over — a classic cascading failure. Design for it: use short timeouts on cache calls, treat a cache error as a miss, add request coalescing so one origin call serves many waiters, and protect the DB with a circuit breaker and load shedding. The cache should be an optimization, not a hard dependency.

A very popular key expires and 5,000 requests miss at once.

That's a cache stampede: all 5,000 recompute the value and slam the DB simultaneously. Fix with single-flight/request coalescing (only one recomputes, others wait), a short lock, or stale-while-revalidate (serve the old value while one worker refreshes). Randomizing TTLs avoids many keys expiring together (avalanche).

Think like a senior

Senior Engineer Insight

The famous line 'there are only two hard things in computer science: cache invalidation and naming things' is about this chapter. Anyone can add a cache; the seniority signal is having a concrete answer for staleness, stampede, and cache-down.

Senior Engineer Insight

Always state your consistency requirement first. 'Can this be 60 seconds stale?' changes the entire design. Caching a bank balance and caching a product description are not the same problem.

Remember

A cache is a copy, never the source of truth.

Remember

Every cache needs an answer for three questions: how does it get stale, what happens on a stampede, and what happens when it's down.

Interview questions

Walk me through cache-aside and its main pitfall.

Active recall

Check yourself

Caching Fundamentals · Question 1 / 1Medium

In the cache-aside pattern, who is responsible for populating the cache on a miss?

Practical challenges

Medium1–3 hours

Product API with cache-aside

Build a product read API over PostgreSQL + Redis using cache-aside, then measure the hit ratio.

Products are read constantly and updated rarely. You want p99 latency down and DB load reduced.

Requirements

Implement cache-aside reads with a TTL.
Invalidate (or update) the cache on product writes.
Expose a metric for cache hit ratio.

Constraints

Cache errors must degrade to a DB read, not a 500.
Per-product keys, never a shared blob.

Acceptance criteria

Repeated reads of the same product hit the cache.
After an update, the next read reflects the new data.
Killing Redis still serves reads (from the DB).

Edge cases

Product not found (cache the negative result briefly?)
Concurrent update + read.

Bonus

Add stale-while-revalidate.
Add request coalescing to prevent a stampede.

Reflection

What hit ratio makes this worthwhile, and how does TTL affect it?

Failure drill1–2 hours

Prevent a cache stampede

Reproduce a stampede, then fix it so only one request recomputes a hot key.

A single hot key expires and hundreds of concurrent requests all miss and recompute simultaneously.

Requirements

Simulate N concurrent misses on one key.
Show the DB being hit N times (the problem).
Apply a fix so the DB is hit once.

Acceptance criteria

After the fix, exactly one recomputation occurs; others wait or serve stale.

Edge cases

The single recomputing request fails — others must not wait forever.

Bonus

Compare a mutex/single-flight approach vs stale-while-revalidate.

Reflection

Which fix is better for a value that is very expensive vs cheap to compute?

Architecture challenge

Design the caching layer for a product API backed by PostgreSQL that must serve 20,000 reads/sec with p99 < 50ms, where products change a few times per day. Decide the strategy, TTLs, and keys; then explain exactly what happens (and how the DB survives) when the cache node restarts and every key is cold.

Flashcards

Flashcards1 / 4

Summary

Caching stores copies of expensive data in a fast layer so most reads avoid the origin. Cache-aside is the workhorse pattern; TTLs and eviction bound staleness and memory. The real engineering is in the failure modes — staleness, stampedes, and cache-down — which you must design for explicitly rather than assume away.

Key takeaways

Cache-aside: check cache, on miss read source and populate with a TTL.
Hit ratio and miss cost determine whether a cache earns its keep.
Trade freshness for speed — pick your TTL with the consistency requirement in mind.
Always design for stampede and cache-down; the cache must be an optimization, not a hard dependency.

Your notes

Saved to this device