1.13 Distributed-System Upgrades and Rollbacks

1. Why Upgrades and Rollbacks Matter

A distributed service usually runs multiple instances. A release therefore replaces or changes a fleet rather than a single process. The deployment strategy must control compatibility, traffic exposure, failure blast radius, and the path back to a known-good version.

2. Deployment Strategies

2.1 Downtime Deployment

Stop the current version, deploy the new version, then restart service. It is simple and avoids mixed-version behavior, but causes user-visible downtime.

2.2 Blue-Green Deployment

Maintain two environments. Traffic stays on the current environment while the new version is deployed and verified in the other one, then traffic switches over. Rollback can be fast, but the duplicated environment costs more and stateful migrations still need careful handling.

2.3 Rolling Deployment

Replace instances gradually. Capacity remains available, but old and new versions coexist, so APIs, schemas, messages, and configuration must remain compatible during the rollout.

2.4 Canary Deployment

Send a small fraction of real traffic to the new version, observe metrics and logs, then expand gradually if healthy. Canary rollout is primarily a release-risk control mechanism.

2.5 A/B Testing

Run multiple product variants concurrently and compare user or business outcomes. A/B testing is primarily an experiment, not a deployment-safety mechanism, although the infrastructure can overlap with canary delivery.

3. Rollback Strategies

• redeploy the previous artifact or container image;
• switch traffic back to the old environment;
• disable risky behavior through a feature flag;
• roll forward with a small corrective release when data/schema changes make rollback unsafe.

A reliable release process should make both deployment and rollback observable and repeatable.