1.1 Message Queues and Event Logs

1. Why Use a Message System?

A message broker decouples producers from consumers in time and topology.

Instead of service A synchronously calling B, C, and D, A can publish an event and let consumers process it independently.

Main benefits:

• decoupling — producers do not need to know every consumer;
• asynchrony — user-facing work can finish before downstream processing;
• buffering/backpressure — bursts can accumulate in the broker while consumers process at sustainable speed;
• replay/retention in log-oriented systems.

The cost is additional distributed-system state: broker availability, duplicate/lost messages, lag, ordering, retries, and observability.

2. Queue vs. Pub/Sub / Log

Work Queue

A message is normally processed by one worker from a consumer pool.

Pub/Sub

Multiple independent subscriptions can each receive the event.

Durable Log

Kafka-style systems retain records for a configured period/size and let consumer groups track positions independently. Multiple groups can replay the same partition history.

These models overlap in modern brokers, so product behavior matters more than one textbook label.

3. Push vs. Pull

Push can minimize delivery latency but requires the broker to respect consumer backpressure.

Pull lets consumers control fetch rate and batch size. Pull need not mean inefficient busy polling: long-polling/blocking fetch can wait efficiently for data.

4. Delivery Semantics

Common goals are:

• at-most-once — no retry after uncertain delivery; loss possible;
• at-least-once — retry until acknowledged; duplicates possible;
• exactly-once within a defined scope — requires stronger coordination/idempotence and does not automatically make external side effects exactly once.

Most business consumers should be idempotent even when the broker has stronger features.

5. Reliability Layers

A message can fail at several boundaries:

1. producer → broker;
2. broker storage/replication;
3. broker → consumer;
4. consumer side effect → acknowledgement/offset commit.

A complete design handles all four.

6. Ordering

Global order and horizontal scalability conflict. Most scalable brokers guarantee order only within a partition/queue/shard.

Route events that require relative order by the same entity key to the same ordered partition, then process that partition sequentially or preserve key-level sequencing.

7. Consistency with Databases

Publishing an event and committing a database transaction are two separate writes. Use patterns such as a transactional outbox/CDC when the business invariant requires the database state and emitted event to stay consistent.