NOTE
1.14 Distributed-System Failures
Failure detection and common resilience behaviors including failover, fail-fast, fail-safe, fail-silent, failback, hedged/forked calls, and broadcast calls.
This is a historical learning note and may contain outdated or incomplete understanding.
1. What Is a Distributed-System Failure?
Failures include crashed or overloaded nodes, broken network paths, long pauses, and partial reachability. A distributed caller often cannot immediately distinguish a dead peer from a slow or partitioned peer.
2. Detecting Failures
2.1 Heartbeats
Nodes exchange periodic heartbeats or health signals. Missing several expected heartbeats can make a peer suspected unavailable, but timeouts are not proof of failure.
2.2 Gossip-Based Detection
Gossip can spread membership and failure-suspicion information across a cluster.
3. Common Failure-Handling Behaviors
- Failover: route work to another healthy replica.
- Fail-fast: reject quickly when success is unlikely instead of consuming resources until timeout.
- Fail-safe: degrade gracefully when a non-critical dependency fails.
- Fail-silent: temporarily stop calling a peer believed unhealthy; circuit breakers implement a related idea.
- Failback: restore traffic to a recovered primary or preferred replica.
- Forking / hedged requests: issue equivalent requests to multiple replicas and use the first acceptable response, trading extra load for tail-latency reduction.
- Broadcast: send an operation to multiple targets when the operation semantically requires all or many of them to process it.
Retries and duplicate requests require idempotency or deduplication when the operation has side effects.