Failure Training

The interview question that separates seniors: 'what happens if this fails?' Step through each cascade one failure at a time, then reveal how real systems contain the blast radius.

Caching

A hot cache key expires under load

Trigger

A popular key (e.g. the homepage feed) expires while receiving 50k req/s.

The cascade

  1. 1Every request misses the cache simultaneously
  2. 2All of them hit the database at once (thundering herd)
  3. 3DB connection pool saturates, queries queue up
  4. 4Latency spikes → upstream timeouts → retries → more load
  5. 5Database falls over, cascading to every dependent service
Step 1 of 5

Blast radius

Entire read path for anything backed by that database.