Designing Highly Available Systems

1. Availability Is an End-to-End Property

Adding two servers does not automatically make a system highly available. Availability depends on every critical dependency and on whether failover actually works under real failure modes.

Define a target SLO first, such as request success/latency over a time window.

2. Remove Single Points of Failure

Use redundancy across independent failure domains:

• multiple application instances;
• load balancers/gateways with HA;
• replicated storage;
• multi-zone placement where the requirement justifies it.

Independence matters: two replicas on the same rack, credential, quota, or control plane may fail together.

3. Bound Failures

Every remote dependency needs a timeout. Without one, a slow dependency can consume all threads/connections and become a cascading failure.

Use:

• bounded retries with backoff/jitter;
• circuit breakers when useful;
• concurrency limits/bulkheads;
• load shedding and admission control.

Retries multiply load, so they need budgets.

4. Graceful Degradation

Not every dependency deserves to take down the request.

Possible fallback behaviors:

• stale cache;
• reduced feature set;
• asynchronous completion;
• default/partial response;
• queue for later processing.

Fallback must have explicit correctness limits.

5. Data Availability

Choose replication/failover based on RPO/RTO and consistency needs. More replicas do not guarantee zero data loss if replication is asynchronous.

Backups are separate from replicas: replication copies mistakes too.

6. Observe and Test

Monitor user-facing SLOs, saturation, dependency errors, queue/replication lag, and failover state.

Regularly test:

• instance loss;
• zone/network partition;
• dependency latency;
• bad deploy/config;
• backup restoration.

A failover design that has never been exercised is an assumption, not a capability.