ByteByteGo · Distributed Systems

Must-Know Failure Modes

Six failure patterns every distributed system engineer should recognize. Run each simulation to see how partitions, ambiguity, gray failures, and cascades behave in practice.

Network Partition

1
Cluster A
A1A2A3
Cluster B
B1B2B3

Each side stays internally consistent but cannot see the other half of the cluster.

Split-Brain

2
Leader A PRIMARY
user.balance = $100
Leader B PRIMARY
user.balance = $50

When the partition heals, which value wins? Two valid histories cannot always be merged cleanly.

Partial Failures

3

From the client's view, four outcomes look identical — the request may or may not have succeeded.

Lost in transit — request never arrived.
Crashed before processing — server died first.
Reply lost — work succeeded, response dropped.
Still processing — server will reply later.

Gray Failures

4
Self health check Healthy
User-facing latency45 ms

Alive by every internal check, yet failing real workloads (slow disk, GC pause, broken NIC).

Amplification Loop

5
1. Trigger → 2. Degraded state → 3. System responds → 4. Amplifies problem

Retries, failover, and autoscale can make an incident worse instead of better.

Cascading Failures

6

Load shifts from failed nodes to survivors until they overload too. Adding nodes mid-cascade can spread failure further.

← Back to all guides