Message Broker & Xử lý bất đồng bộMessage Brokers & Async Processing

Part of the Backend Roadmap knowledge base.

Overview

Most backend systems start with synchronous request/response calls: a client asks, a service does the work, the client waits for the answer. This is simple and easy to reason about, but it couples the caller’s fate to the callee’s. If the downstream service is slow, the caller is slow. If it is down, the caller fails. If a burst of traffic arrives, both must scale in lockstep.

Asynchronous messaging breaks that coupling. Instead of calling a service directly, a producer writes a message to a message broker — a piece of middleware that stores and routes messages — and returns immediately. One or more consumers read those messages later, at their own pace. The broker acts as a buffer, a router, and a shock absorber between parts of your system.

This shift unlocks a family of patterns — background job processing, event-driven architecture, streaming, and data pipelines — that are essential for building scalable, resilient backends. This note covers why and when to go async, the two core messaging models (queues and pub/sub), delivery semantics, a deep comparison of RabbitMQ vs Kafka, the wider ecosystem of brokers, and the production patterns (outbox, saga, competing consumers, fan-out) you need to use them safely.

Related notes: Software Architecture (where messaging fits in an overall design), Real-time Communication (pushing messages to clients), and Scalability & Reliability (using async to absorb load and survive failure).

Fundamentals

Why async? The core benefits

Synchronous vs asynchronous communication

Async is not free: you trade simplicity and immediate consistency for eventual consistency, harder debugging, and new failure modes (duplicate messages, out-of-order delivery, poison messages). Use it where the benefits above matter; keep synchronous calls for reads that need an answer now.

The actors

• Producer (publisher) — creates messages and sends them to the broker.
• Broker — the middleware that receives, stores, routes, and delivers messages. It provides durability, routing rules, and delivery guarantees.
• Consumer (subscriber / worker) — reads messages and processes them.
• Message — a self-contained unit of data: a payload (the body, e.g. JSON/Avro/Protobuf) plus metadata/headers (a message ID, timestamp, content type, routing key, correlation ID, etc.).

A well-designed message is self-describing and versioned. Prefer explicit schemas (JSON Schema, Avro, Protobuf) and a schema registry so producers and consumers can evolve independently.

Key Concepts

Messaging models: queue vs pub/sub

There are two foundational models. Most brokers support one or both.

1. Message queue (point-to-point) — a message is delivered to exactly one consumer among those listening. Multiple consumers on the same queue compete for messages, which spreads work (the “competing consumers” pattern). This is the model for task/job queues.

Producer ──> [ Queue ] ──> Consumer A   (each message goes to exactly one)
                       └──> Consumer B

2. Publish/subscribe (pub/sub) — a message published to a topic is delivered to every subscriber. Each subscriber gets its own copy. This is the model for event broadcasting / fan-out.

Note that Kafka blurs the line: a topic is pub/sub across consumer groups, but within a single group, partitions are distributed among members like competing consumers (see below).

Delivery semantics

The single most important thing to understand about any broker is what guarantees it makes about delivery.

“Exactly-once” is subtle. True end-to-end exactly-once delivery across a network is impossible in the general case. What brokers actually offer is exactly-once processing within specific boundaries — e.g. Kafka’s transactions give exactly-once semantics for read-process-write cycles inside Kafka. In practice, you achieve effective exactly-once by combining at-least-once delivery with idempotent consumers. Do not assume “exactly-once” removes the need for idempotency.

Acknowledgements (acks)

Delivery guarantees hinge on acks. After a consumer processes a message, it sends an ack to the broker, which then removes/advances past the message. If the consumer crashes before acking, the broker re-delivers.

• Auto-ack (ack on receive, before processing) → at-most-once.
• Manual ack after processing → at-least-once.
• A nack (negative ack) or timeout tells the broker to redeliver or route to a dead-letter queue.

On the producer side, publisher confirms (RabbitMQ) or acks=all (Kafka) ensure the broker durably persisted the message before the producer considers it sent.

Idempotent consumers

Because at-least-once means duplicates, consumers must be idempotent — processing the same message twice has the same effect as once. Techniques:

• Dedup by message ID — store processed message IDs (in Redis/DB) and skip ones you’ve seen.
• Idempotency keys — natural keys or client-supplied keys that make writes safe to retry (INSERT ... ON CONFLICT DO NOTHING, upserts).
• Conditional updates — UPDATE ... WHERE status = 'pending' so a replay is a no-op.

def handle(msg):
    if seen.exists(msg.id):          # already processed → skip
        ack(msg); return
    with db.transaction():
        apply(msg.payload)           # do the work
        seen.add(msg.id, ttl=7*DAY)  # record it (ideally in same tx)
    ack(msg)

Ordering

Global ordering is expensive. Most brokers guarantee ordering only within a partition (Kafka) or a single queue with one consumer (RabbitMQ). To keep related messages ordered, route them to the same partition/queue using a partition key (e.g. userId), accepting that unrelated messages may interleave. Adding parallel consumers usually sacrifices strict ordering.

Dead-letter queues (DLQ)

A message that repeatedly fails processing is a poison message — retrying it forever blocks the queue and wastes resources. A dead-letter queue is where such messages are sent after N failed attempts (or on TTL expiry / queue overflow). This isolates bad messages so healthy ones keep flowing, and gives you a place to inspect, alert on, and manually replay failures.

Best practice: retry a few times (often with exponential backoff, via a delay/retry queue), then DLQ. Monitor DLQ depth — a growing DLQ is an incident signal.

RabbitMQ vs Kafka — deep comparison

These are the two dominant brokers, built on fundamentally different models. RabbitMQ is a smart broker / dumb consumer message queue (AMQP); Kafka is a dumb broker / smart consumer distributed commit log.

RabbitMQ model (AMQP)

Producers publish to an exchange, not directly to a queue. The exchange routes to queues based on bindings and a routing key. Consumers read from queues. Once a message is acked, it is gone.

Producer ──> [ Exchange ] ──(binding, routing key)──> [ Queue ] ──> Consumer

Exchange types:

RabbitMQ excels at complex routing, per-message priorities, flexible topologies, and low-latency task distribution. Messages are typically consumed and deleted (a queue is not a durable log you can rewind).

Kafka model (log)

Kafka is a distributed, append-only, partitioned commit log. A topic is split into partitions; each partition is an ordered, immutable sequence of records. Producers append to partitions (chosen by key hash). Consumers track their own offset (position) and read forward. Messages are retained (by time or size) regardless of consumption — so multiple consumers can read the same data, and you can replay from any offset.

                       Partition 0: [r0 r1 r2 r3 ...]  ← offset
Producer ──> [ Topic ] Partition 1: [r0 r1 r2 ...]
                       Partition 2: [r0 r1 ...]
Consumer group members each own a subset of partitions.

Consumer groups: all consumers in a group share the topic’s partitions — each partition is read by exactly one member (competing consumers → parallelism). Different groups read the topic independently (pub/sub → fan-out). Parallelism is capped by the partition count.

Comparison table

When to use which

Choose RabbitMQ when:

• You need task/job queues and background work distribution.
• You need complex routing logic (topic/header-based).
• You want per-message priority, TTLs, and flexible ack semantics.
• Messages are transient commands to be processed once and discarded.
• Lower operational complexity for moderate volumes.

Choose Kafka when:

• You are building event streaming / an event-driven backbone.
• You need to replay history or have multiple independent consumers of the same stream.
• You need very high throughput and durable event storage (event log as source of truth).
• You are doing stream processing, analytics, log aggregation, or CDC.
• Ordering per key at scale matters.

Many organisations run both: Kafka as the streaming backbone/event log, RabbitMQ (or SQS/Celery) for command-style task queues.

Other brokers (brief)

For managed cloud stacks, SQS/SNS or Google Pub/Sub remove most operational burden and are excellent defaults.

Streaming & event-driven architecture

In event-driven architecture (EDA), services communicate by emitting and reacting to events (“something happened”, e.g. PaymentReceived) rather than issuing commands. This maximises decoupling: the emitter doesn’t know or care who consumes.

• Event notification — a thin event (“order 123 changed”); consumers call back for details.
• Event-carried state transfer — the event carries all needed data, so consumers don’t call back.
• Event sourcing — the event log is the source of truth. State is derived by replaying events, not stored as current-value rows. This gives a full audit trail and time-travel, at the cost of complexity and needing projections/read models. Kafka’s replayable log makes it a natural fit.

Change Data Capture (CDC) turns a database’s commit log (e.g. Postgres WAL, MySQL binlog) into a stream of change events, typically via tools like Debezium into Kafka. This lets you propagate data changes to other systems (search indexes, caches, data warehouses) without dual writes — and it underpins the outbox pattern below.

Backpressure, throttling, and load shedding

When producers outpace consumers, the system must cope or it will fall over.

• Backpressure — signalling upstream to slow down. A pull-based consumer (Kafka) naturally applies backpressure by fetching only what it can handle. RabbitMQ uses prefetch (QoS) to limit unacked messages per consumer and flow control to slow fast publishers when memory is high.
• Throttling / rate limiting — deliberately cap the rate at which producers publish or consumers process, protecting downstream resources (e.g. a fragile third-party API).
• Load shedding — under overload, deliberately drop lower-priority messages to protect the system (better to lose some non-critical work than crash entirely). Often paired with priority queues.
• Load shifting / leveling — move work in time: enqueue during peaks, process during troughs. This is a primary reason to use queues at all.

Watch queue depth / consumer lag as your key health metric. Growing lag means consumers can’t keep up — scale out, optimise, throttle producers, or shed load. Also bound your queues: unbounded queues just move the failure from “overloaded service” to “out-of-memory broker”.

Patterns

Outbox pattern

The classic problem: you must update your database and publish an event atomically. If you write to the DB then publish, a crash in between loses the event (dual-write problem). The outbox pattern solves it:

1. In the same local DB transaction as your business change, insert the event into an outbox table.
2. A separate relay (a poller or CDC via Debezium) reads unpublished outbox rows and publishes them to the broker, marking them sent.

Because the business write and the outbox insert share one transaction, they succeed or fail together. The relay provides at-least-once publishing to the broker.

BEGIN;
  UPDATE orders SET status = 'paid' WHERE id = 42;
  INSERT INTO outbox (id, topic, payload) VALUES (gen_random_uuid(), 'OrderPaid', '{"orderId":42}');
COMMIT;
-- relay later: SELECT * FROM outbox WHERE published = false; publish; mark published.

Saga

A saga manages a distributed transaction across services without a global 2-phase commit. It is a sequence of local transactions, each publishing an event that triggers the next step; if a step fails, compensating transactions undo the prior ones.

• Choreography — each service reacts to events and emits the next; no central coordinator. Simple but the flow is implicit and can be hard to follow.
• Orchestration — a central orchestrator tells each service what to do and handles compensation. Explicit and easier to reason about; the orchestrator is a dependency.

Example (order saga): OrderCreated → reserve inventory → charge payment → ship. If payment fails, emit compensations: release inventory, cancel order.

Competing consumers

Multiple consumers read from one queue to process messages in parallel. The broker delivers each message to only one of them, so throughput scales with the number of workers. This is the workhorse pattern for job queues. Combine with idempotency (redeliveries) and be aware it trades away strict ordering.

Fan-out

One message reaches many consumers, each doing something different (billing, email, analytics). Implemented via pub/sub topics, a fanout exchange, or SNS→multiple SQS queues. The key benefit: adding a new consumer requires no change to the producer.

Task/job queues for background work

Application-level job queues sit on top of a broker to run background work (send email, resize image, generate report). They add developer ergonomics: retries, scheduling, result storage, and worker management.

Example (Celery):

from celery import Celery
app = Celery("tasks", broker="amqp://localhost", backend="redis://localhost")

@app.task(bind=True, max_retries=3, acks_late=True)
def send_welcome_email(self, user_id):
    try:
        email.send(user_id)
    except TransientError as exc:
        raise self.retry(exc=exc, countdown=2 ** self.request.retries)

# producer
send_welcome_email.delay(user_id=42)   # returns instantly; runs in a worker

Use acks_late=True (ack after work, not before) for at-least-once semantics, keep tasks idempotent, set max_retries with backoff, and route poison messages to a DLQ.

Best Practices

• Default to at-least-once + idempotent consumers. Assume every message can arrive more than once and out of order. Design handlers to be safe under replay.
• Use dead-letter queues. Never let a poison message block a queue. Retry with backoff, then DLQ, then alert on DLQ depth.
• Make messages self-describing and versioned. Include a message ID, timestamp, type, and schema version. Use a schema registry (Avro/Protobuf) for safe evolution; add fields, don’t repurpose them.
• Solve dual writes with the outbox pattern, not “write DB then publish”. Never rely on two independent systems both succeeding without a transaction or CDC bridging them.
• Keep messages small. Put large payloads (files, images) in object storage and send a reference/pointer, not the bytes (claim-check pattern).
• Bound your queues and monitor lag. Consumer lag / queue depth is the vital sign. Alert on growth; unbounded queues just relocate the outage.
• Choose ordering deliberately. Only require ordering where you truly need it, and scope it to a partition/key. Global ordering kills parallelism.
• Design for backpressure and overload. Use prefetch/QoS, rate limiting, and (where appropriate) load shedding of low-priority work. Decide what to drop before you’re on fire.
• Set correlation and causation IDs. Propagate a correlation ID through async flows so you can trace a request across services and time (distributed tracing).
• Separate retryable from non-retryable errors. Retry transient failures (timeouts) with backoff; send permanent failures (validation errors) straight to the DLQ — don’t retry them.
• Prefer managed brokers when you can. SQS/SNS, Google Pub/Sub, and managed Kafka/RabbitMQ remove huge operational burden; self-host only when you must.
• Pick the model to the job. Queue for distributing work; pub/sub for broadcasting events. RabbitMQ for routing/tasks; Kafka for streaming/replay. Don’t force one tool to do everything.
• Test failure paths. Deliberately kill consumers mid-process, inject duplicates, and delay messages in staging to verify idempotency, retries, and DLQ behaviour.

References

• RabbitMQ — Tutorials & Concepts
• RabbitMQ — AMQP 0-9-1 Model Explained
• Apache Kafka — Documentation
• Apache Kafka — Introduction & Design
• Confluent — What is Event-Driven Architecture
• Microservices.io — Transactional Outbox & Saga patterns
• AWS — SQS vs SNS vs EventBridge messaging
• Enterprise Integration Patterns (Hohpe & Woolf)