System_Design

An overview of why distributed systems are needed and how replication, partitioning, consistency, transactions, communication, and common distributed components fit together.

A repeatable system-design method: clarify requirements, quantify scale, define data and invariants, identify bottlenecks, choose components, and validate failure modes.

Requirements and failure cases for locks across processes or machines, with Redis- and ZooKeeper-style approaches and the importance of ownership and fencing.

Strategies for large-scale data: partitioning, external sorting, hashing, Bloom filters, streaming aggregation, distributed storage, and choosing algorithms from access patterns.

Design goals and trade-offs for globally unique IDs using database sequences, UUIDs, Redis counters, and Snowflake-style timestamp/worker/sequence layouts.

Availability engineering through redundancy, failure domains, timeouts, retries, load shedding, replication, failover, graceful degradation, observability, and recovery testing.

Why process-local sessions break under horizontal scaling and how replication, shared session stores, and centralized authentication address the problem.

A practical path from reducing work per request to caching, horizontal scaling, partitioning, asynchronous processing, backpressure, and hotspot control.

The core building blocks of distributed storage: partitioning, replication, consistency, metadata, routing, and failure recovery.

Choosing data models from invariants and access patterns: relational, document, key-value, search/index, event log, graph, denormalization, and derived views.

BASE as an availability-oriented distributed-systems design idea: basic availability, soft state, eventual consistency, and its relationship to flexible transactions.

Refactoring as behavior-preserving design improvement, with test protection, incremental changes, observability, and boundaries between cleanup and architectural rewrite.

The CAP theorem: consistency, availability, partition tolerance, and the C/A trade-off a distributed system faces when a network partition occurs.

Vertical vs. horizontal scaling, statelessness, load balancing, autoscaling signals, warm-up, downstream capacity, and stateful partitioning.

What cluster metadata represents, why routing and ownership depend on it, and centralized versus peer-to-peer metadata management.

A methodology for technical planning: business goals, current bottlenecks, target architecture, risk reduction, migration stages, observability, and measurable outcomes.

Why replicated distributed systems face consistency problems, how consistency models define observable guarantees, and how consensus algorithms help nodes agree on state.

A cache abstraction layer for routing, serialization, TTL policy, observability, stampede protection, multi-level caching, and graceful fallback.

A compact introduction to distributing computation across machines, with batch and stream processing as two common execution models.

Connection-pool lifecycle, capacity, health checks, timeouts, idle eviction, backpressure, and protocol-safety considerations.

Synchronous and asynchronous service communication, RPC versus resource-oriented HTTP APIs, API compatibility, and message encoding choices.

RPC framework architecture: IDL/contracts, serialization, transport, multiplexing, discovery, load balancing, timeouts, retries, observability, and compatibility.

How state placement changes routing, scalability, recovery, and storage dependencies in distributed services.

Fixed/sliding windows, token and leaky buckets, distributed rate-limit state, fairness, burst handling, local/global layers, and failure behavior.

Deployment strategies for multi-instance services: downtime, blue-green, rolling, canary rollout, A/B testing, and practical rollback mechanisms.

Public API contracts, authentication, authorization, idempotency, versioning, errors, pagination, quotas, signatures, observability, and backwards compatibility.

Failure detection and common resilience behaviors including failover, fail-fast, fail-safe, fail-silent, failback, hedged/forked calls, and broadcast calls.

Deadline propagation, timeout budgets, retry safety, exponential backoff/jitter, retry storms, idempotency, hedging, and observability.

How distributed nodes exchange cluster metadata and the trade-offs between centralized coordination and peer-to-peer dissemination.

Stable error taxonomy, machine-readable codes, retryability, user-facing messages, wrapping/context, cross-service mapping, and observability.

Why transactions become harder across services and data stores, and how 2PC, TCC, Saga, reliable messaging, reconciliation, and best-effort notification address different consistency requirements.

A general model for thread, connection, object, and worker pools: lifecycle, sizing, queue bounds, validation, fairness, and overload behavior.

How two-phase commit coordinates atomic commit across transactional resources, and why blocking, coordinator failure, and long-held resources are its main costs.

Cache layers, cache-aside/read-through/write-through/write-back, consistency races, invalidation, stampedes, penetration, hot keys, TTL, and source-of-truth recovery.

Application-level distributed transactions using Try, Confirm, and Cancel operations, including reservations, compensation, retries, idempotency, and business-code cost.

How a durable business change and a durable outgoing message are coupled so downstream services can converge through retries and idempotent consumption.

A notification pattern for external or loosely coupled systems using bounded retries, durable result lookup, and reconciliation instead of atomic cross-system commit.

Long-running distributed transactions as a sequence of local transactions plus compensating actions, with orchestration/choreography and isolation trade-offs.

The CanCommit, PreCommit, and DoCommit structure of 3PC, why it was proposed to reduce blocking, and why partitions still make it uncommon in practice.

A conceptual comparison between resource-level two-phase commit and application-level TCC, both of which separate preparation from the final decision.

How to commit business state and an outgoing event in one local database transaction, then asynchronously publish with retries and idempotent consumption.

A compact implementation-oriented view of two-phase commit and XA resource coordination.

An implementation-oriented summary of application-level Try/Confirm/Cancel transactions and their idempotency and compensation requirements.

Using an MQ for retryable notifications while keeping an authoritative query path for receivers that miss or cannot process a callback.

RocketMQ's half-message, local-transaction, commit/rollback, and transaction-check protocol for coupling producer-side local state with message visibility.

Why distributed nodes need consensus, how Paxos, Raft, ZAB, and Gossip differ, and how consensus relates to strong or eventual consistency.

Basic Paxos roles and two-phase decision flow, why contention is expensive, and how Multi-Paxos uses a stable leader to make repeated consensus practical.

An entry point to ZooKeeper Atomic Broadcast (ZAB), ZooKeeper's leader-based protocol for ordered, reliable state updates.

Raft's leader, follower, and candidate roles; leader election; replicated-log operation; quorum commitment; and recovery after leader failure.

How gossip protocols spread information through randomized peer-to-peer exchanges, why they scale well, and the redundancy and convergence trade-offs they introduce.

A practical map of linearizability, eventual and causal consistency, read-your-writes, session consistency, monotonic reads and writes, and consistent-prefix reads.

Why distributed systems keep multiple replicas of the same data, how replication improves availability and read capacity, and the major design dimensions behind replication.

Why multiple leaders are useful across regions or offline-capable systems, and how concurrent writes create conflicts that require prevention, detection, or resolution.

Leaderless replication with multi-replica reads/writes, quorum-style coordination, version reconciliation, read repair, and anti-entropy.

How leader/follower replication handles writes, reads, initial synchronization, incremental catch-up, leader election, and failover.

Physical and logical change logs, statement- versus row-oriented replication, and why deterministic replay and schema/storage coupling matter.

Synchronous, asynchronous, and semi-synchronous replication and how acknowledgement rules change latency, durability, availability, and replica lag.

The three common replication topologies: single-leader, multi-leader, and leaderless replication, and the trade-offs each introduces.

How partitioning splits a dataset across nodes, why it scales storage and read/write throughput, and how partitioning differs from replication.

How to choose a partition key and compare explicit, random, range, modulo-hash, consistent-hash, virtual-node, and hash-slot partitioning.

How shard keys determine targeted routing, why missing shard keys cause scatter/gather, and the difference between local and global secondary indexes.

Three ways to locate a partition owner: client-side routing, a dedicated proxy, or server-side forwarding/redirects.

How logical partitions are assigned to machines, static versus dynamic placement, and what changes when a cluster rebalances.