Khả năng mở rộng & Độ tin cậyScalability & Reliability

Part of the Backend Roadmap knowledge base.

Overview

Scalability is a system’s ability to handle more work — more users, more requests, more data — by adding resources, ideally without a rewrite and ideally at a cost that grows no faster than the load. Reliability is a system’s ability to keep doing the right thing correctly even when parts of it fail, slow down, or behave unexpectedly. The two are deeply linked: a system that cannot scale eventually falls over under load (a reliability failure), and a system that is not designed for failure cannot scale safely because every added component is one more thing that can break.

For a backend engineer these are not “later” concerns. The architectural decisions that make a system scalable and reliable — statelessness, load balancing, timeouts, retries, idempotency, redundancy — are cheap to design in early and brutally expensive to retrofit. This note covers how to reason about scale, the patterns that keep systems standing under stress, the failure modes those patterns defend against, and the metrics (SLA/SLO/SLI, error budgets, the “nines”) that let you manage reliability as an engineering discipline rather than a hope.

A useful mental model comes from Michael Nygard’s Release It!: every integration point is a potential failure, and failures propagate. Your job is not to prevent all failure — that is impossible in a distributed system — but to contain it so that one slow database, one dead node, or one traffic spike does not take down everything.

Related notes in this knowledge base:

• Database Design & Scaling — replication, sharding, read replicas.
• Caching — the cheapest way to buy scalability and reduce load.
• Message Brokers & Async — decoupling and load leveling with queues.
• Observability & Monitoring — you cannot manage reliability you cannot measure.
• Load balancer internals and networking: ../network-engineer roadmap material.

Fundamentals

Vertical vs horizontal scaling

There are two fundamental ways to add capacity.

The practical rule: scale up first because it is simple, but design so you can scale out, because vertical scaling always hits a wall and gives you no redundancy. Real systems combine both — moderately sized nodes, many of them.

Statelessness: the key that unlocks horizontal scale

A service is stateless when any request can be served by any instance, because the instance holds no client-specific data between requests. State is pushed to shared backing stores: databases, caches (Redis), object storage, and the request/token itself (e.g. a JWT).

Why it matters: if instance A holds a user’s session in local memory, that user must return to A. You cannot freely add, remove, or replace instances — you have lost the whole point of horizontal scaling, and you have created a single point of failure per user.

Rules for statelessness:

• No in-memory sessions — use a shared session store or stateless tokens.
• No local file writes that later requests depend on — use object storage (S3).
• No “sticky” in-process caches that must be consistent across nodes — use a shared cache.
• Any node must be safely killable at any time (this also enables autoscaling and rolling deploys).

The scaling cube (AKF Scale Cube)

The AKF Scale Cube describes three orthogonal axes of scaling:

• X is the easiest and where you start — it needs statelessness and a load balancer.
• Y aligns with team boundaries and lets you scale hot subsystems independently.
• Z is sharding — it removes the single-database bottleneck (see Database Scaling).

Mature systems use all three.

Performance vs scalability

These are often confused but are different properties:

• Performance = how fast a single request is served (latency) at low load.
• Scalability = how well the system maintains acceptable performance as load grows.

A system can be fast for one user and collapse at a thousand (great performance, poor scalability). Another can be moderately fast per request but hold that latency flat from 1 to 1,000,000 users (the scalable one). Optimize for the shape of the curve, not just the starting point. Always measure tail latency (p95/p99), not the average — the average hides the users having the worst experience, and at scale the tail is many real users.

Little’s Law — the capacity intuition

L = λ × W, where L = concurrent requests in the system, λ = arrival rate (req/s), W = average time in system (s). If requests arrive at 500/s and each takes 0.2s, you have ~100 requests in flight on average and need enough threads/connections/workers to hold them. When W rises (a slow dependency), L rises, resources exhaust, and you queue or fall over — this is the mechanism behind most overload collapses.

Key Concepts

Load balancing

A load balancer (LB) sits in front of your instances and distributes incoming requests across them. It is the linchpin of horizontal scaling and also a reliability tool: it routes traffic away from unhealthy nodes.

L4 vs L7 load balancing

Use L7 for HTTP microservices (routing /api/orders to one service, /api/users to another; canary by header). Use L4 when you need raw throughput, non-HTTP protocols, or want the backend to terminate TLS itself.

Balancing algorithms

Power of two random choices deserves a mention: picking two servers at random and sending to the less loaded one gives near-optimal balancing without any global state, which is why it is popular in large systems.

Health checks

The LB must know which backends are alive. Two kinds:

• Liveness / passive — mark a node bad after N failed responses; stop sending it traffic.
• Readiness / active — periodically poll a health endpoint (GET /healthz) and only route to nodes that pass.

Make the health check deep enough to be meaningful but shallow enough not to cascade: a /healthz that checks the database will mark every node unhealthy the moment the database blips, taking down a service that could have served cached traffic. A common split: /livez (is the process up?) vs /readyz (can it serve, incl. critical deps?).

Sticky sessions

Sticky sessions (session affinity) pin a client to one backend (via cookie or IP hash). They are a crutch for stateful apps — they undermine even load distribution, break when a node dies (the pinned users lose their session), and complicate deploys. Prefer statelessness with a shared session store. If you must use stickiness, use cookie-based affinity with a short TTL and ensure your app degrades gracefully when re-pinned. See load balancer internals in the ../network-engineer material.

Resilience patterns

These are the defensive patterns from Nygard’s Release It! and the cloud reliability canon. Each contains a specific failure.

Timeouts

Never make a network call without a timeout. The default in many libraries is infinite, which means one hung dependency can exhaust all your threads/connections as requests pile up waiting forever (Little’s Law in action). Set timeouts at every layer: connect timeout, read/socket timeout, total request timeout. Budget them: if your SLA is 300ms and you call three services, they cannot each have a 1s timeout. Timeouts turn a hang (fatal, unbounded) into a fast error (recoverable).

Retries with exponential backoff + jitter

Transient failures (a brief network blip, a rebooting node) are worth retrying. But naive retries are dangerous:

• Retry only idempotent operations (or use idempotency keys — see below). Retrying a non-idempotent POST /charge can double-charge.
• Bound the retries (e.g. max 3). Infinite retries turn a blip into a storm.
• Back off exponentially: wait 100ms, 200ms, 400ms… so you do not hammer a struggling dependency.
• Add jitter (randomize the delay). Without jitter, all clients that failed at the same instant retry at the same instant — a synchronized retry storm / thundering herd that keeps the recovering service down.

import random, time

def call_with_retry(fn, max_attempts=3, base=0.1, cap=2.0):
    for attempt in range(max_attempts):
        try:
            return fn()
        except TransientError:
            if attempt == max_attempts - 1:
                raise
            # full jitter: sleep uniformly in [0, min(cap, base * 2**attempt)]
            backoff = min(cap, base * (2 ** attempt))
            time.sleep(random.uniform(0, backoff))

AWS’s recommended default is exponential backoff with full jitter. Also consider retry budgets (cap retries to e.g. 10% of total requests) so retries can never more than modestly amplify load.

Circuit breaker

A circuit breaker stops calling a dependency that is clearly failing, giving it room to recover and failing fast instead of piling up doomed, slow calls. It is a state machine:

Without a breaker, a slow dependency causes every caller thread to block on its timeout, exhausting the pool and cascading the failure upstream. The breaker converts that into an instant, cheap failure (paired ideally with a fallback). Libraries: Resilience4j (Java), Polly (.NET), and service meshes (Envoy/Istio) provide it at the infra layer.

Bulkhead

Named after ship compartments that stop one flooded section from sinking the vessel. Isolate resources so one overloaded dependency cannot consume all of a shared pool. Example: give calls to the flaky recommendations service their own thread pool / connection pool of 10, separate from the checkout pool. If recommendations hangs, it exhausts its 10 threads and no more — checkout keeps working. Bulkheads turn a total outage into a partial, contained degradation.

Rate limiting & throttling

Rate limiting caps how many requests a client (or the whole system) may make in a window, protecting you from abuse, buggy clients, and traffic spikes. Throttling is the act of rejecting/delaying requests over the limit (usually HTTP 429 Too Many Requests with a Retry-After header).

Common algorithms:

Rate limit per key (API key, user, IP) and enforce it in a shared store (Redis) so the limit is global across instances, not per-node.

Backpressure

Backpressure is a system telling its upstream to slow down when it cannot keep up, instead of silently buffering until it runs out of memory. Instead of accepting unbounded work, a component signals “I’m full” — via a bounded queue that blocks/rejects when full, TCP flow control, HTTP 429, or reactive-streams demand signals. The alternative — unbounded queues — just moves the failure to an out-of-memory crash and hides growing latency. Bounded queues are a feature, not a limitation. Async systems (see Message Brokers) provide natural backpressure: producers slow when the queue backs up.

Load shedding

When overloaded, it is better to serve most requests well than all requests badly (or none, after a crash). Load shedding deliberately drops or rejects a fraction of incoming work to stay within capacity — for example, rejecting low-priority requests first, or returning 503 once concurrency exceeds a threshold. Combined with priority (shed batch/analytics traffic before user-facing checkout), it keeps the system alive and the important work flowing under overload. Google SRE emphasizes shedding before you saturate, based on measured load.

Graceful degradation & fallbacks

Design so that when a non-critical dependency fails, the system loses a feature, not the whole page. Examples:

• Recommendations service down → show a static “popular items” list instead of personalized ones.
• Cache down → serve from the database (slower but working), or serve slightly stale data.
• Payment provider A down → failover to provider B.
• Fresh data unavailable → serve last-known-good cached value (“stale-while-revalidate”).

A fallback is the specific alternative returned when the primary path fails (often paired with a circuit breaker). Decide fallbacks deliberately per dependency; a silent empty result can be worse than a clear error.

Failure modes & designing for failure

You must understand the ways distributed systems fail, because the resilience patterns above each defend against a specific one.

Cascading failures

One component’s failure overloads another, which fails, which overloads the next — a domino chain. Classic path: DB slows → app threads block waiting → thread pool exhausts → app stops responding to health checks → LB removes nodes → remaining nodes get more load → they die faster. Defenses: timeouts, circuit breakers, bulkheads, load shedding, and health checks that do not amplify.

Thundering herd

Many clients hit the same resource at the same instant. Common triggers:

• A popular cache key expires → thousands of requests simultaneously miss and stampede the database. Fix: request coalescing / single-flight (one request recomputes, others wait), staggered TTLs with jitter, or stale-while-revalidate. See Caching.
• A service restarts and every client reconnects at once → jittered reconnect backoff.
• A cron job fires the same second everywhere → spread the schedule.

Retry storm

A dependency slows; clients retry; retries add load; the dependency slows more; more retries — a positive feedback loop that keeps a recovering system down. Defenses: backoff with jitter, retry budgets, circuit breakers (which stop retries entirely when a dependency is clearly down).

Single point of failure (SPOF)

Any component whose failure takes down the whole system: a single database primary, a single load balancer, a single AZ, a shared config service, even a single deploy pipeline. Find them by asking “what happens if this dies?” for every box in your architecture diagram. Remove with redundancy (see below).

Chaos engineering

You do not truly know your system tolerates failure until you cause failure and watch. Chaos engineering is the disciplined practice of injecting controlled failures in production (or production-like environments) — killing instances, adding latency, dropping a dependency, partitioning the network — to verify your resilience holds and to find weaknesses before they find you. Pioneered by Netflix’s Chaos Monkey. Do it with a hypothesis, a small blast radius, and an abort switch. It turns “we think we handle a node death” into “we verified it Tuesday.”

Reliability metrics

You manage reliability numerically, not by feel.

Availability — the nines

Availability = uptime / total time, usually stated as “nines”:

Each extra nine is roughly 10× harder and more expensive. Availability multiplies across dependencies: a service that depends on three components each at 99.9% has a ceiling of ~99.7% (0.999³) — this is why deep dependency chains are fragile and why redundancy and graceful degradation matter.

SLI, SLO, SLA

The Google SRE vocabulary, from the inside out:

Rule of thumb: SLA < SLO — your public promise (SLA) should be looser than your internal target (SLO) so you have margin before you owe anyone money. Pick SLIs that reflect the user’s experience (latency, error rate, availability, freshness), not internal proxies like CPU.

Error budgets

If your SLO is 99.9%, you are permitted 0.1% unreliability — that is your error budget (~43 min/month). This reframes reliability as a resource to spend:

• Budget remaining → ship features fast, take risks.
• Budget exhausted → freeze risky launches, prioritize reliability work.

It resolves the eternal dev-vs-ops tension with a number both sides agree on, and it acknowledges that 100% is the wrong target — it is impossibly expensive and users cannot tell the difference between 100% and 99.99% given their own network flakiness.

MTBF / MTTR and RTO / RPO

Modern reliability thinking prioritizes lowering MTTR over maximizing MTBF — you cannot prevent all failures, so invest in recovering fast. RTO/RPO drive your disaster-recovery and backup strategy: an RPO of 5 minutes means you must replicate/back up at least that often.

High availability & redundancy

High availability (HA) means eliminating single points of failure through redundancy so the system survives component loss.

Active-active vs active-passive

Failover

Failover is automatically switching to a redundant component when the active one fails. Key concerns: fast and reliable failure detection (health checks, heartbeats), avoiding split-brain (two nodes both think they are primary — prevent with quorum/consensus like Raft or a fencing token), and testing that failover actually works (untested failover usually does not).

Multi-AZ and multi-region

• Multi-AZ (Availability Zone): spread instances across isolated datacenters within one region. Survives a datacenter failure; low latency between AZs; the standard baseline for production HA.
• Multi-region: replicate across geographic regions. Survives a whole-region outage and serves users closer to them (lower latency), but introduces cross-region latency, data-consistency, and cost complexity. Usually reserved for the highest availability tiers or global user bases.

The trade-off ladder: single instance → multi-instance (one AZ) → multi-AZ → multi-region, each step buying more availability at more cost and complexity.

Idempotency & consistency under scale

At scale, exactly-once delivery is essentially impossible — networks drop responses, clients retry, and messages get redelivered. The practical answer is idempotency: design operations so that doing them twice has the same effect as doing them once.

• Use idempotency keys: the client sends a unique key with a request (e.g. Idempotency-Key: <uuid>); the server records the key + result, and on a repeat key returns the stored result instead of re-executing. This is how Stripe makes payment retries safe.
• Prefer naturally idempotent designs: PUT (set state) over POST (append), upserts, “set balance to X” over “add X”.
• For consumers of a message queue, expect at-least-once delivery and dedupe on a message ID.

Consistency under scale is a spectrum (see Database Scaling for CAP and replication): strong consistency costs latency and availability; eventual consistency scales better but forces you to handle stale reads. Choose per use case — a bank balance needs strong; a “like” count tolerates eventual.

The fallacies of distributed computing

Peter Deutsch’s classic list of false assumptions that cause distributed systems to fail. Internalize them:

1. The network is reliable. (It is not — design for dropped/duplicated messages.)
2. Latency is zero. (Remote calls are orders of magnitude slower than local; batch and cache.)
3. Bandwidth is infinite. (Payload size matters at scale.)
4. The network is secure. (Assume hostile; encrypt and authenticate.)
5. Topology doesn’t change. (Nodes come and go; do not hardcode addresses — use discovery.)
6. There is one administrator. (Many teams and dependencies you don’t control.)
7. Transport cost is zero. (Serialization, bandwidth, and infra all cost.)
8. The network is homogeneous. (Mixed hardware, protocols, versions.)

Nearly every resilience pattern in this note exists because one of these fallacies is false.

Capacity planning, autoscaling & profiling

Capacity planning

Estimate the resources needed to meet demand at target reliability. Steps: measure current usage and per-request cost, forecast growth and peaks (including seasonal/launch spikes), load test to find the real breaking point (do not guess), and provision with headroom (commonly target ~60-70% utilization so you have room to absorb spikes and lose a node). Plan for peak, not average.

Autoscaling

Automatically add/remove instances based on load. Requires statelessness (new nodes must be interchangeable).

• Reactive / metric-based: scale on CPU, memory, request rate, or queue depth. Simple but lags — scaling up takes time (boot + warmup), so it can be too slow for sharp spikes.
• Scheduled: pre-scale for known patterns (business hours, a sale at noon).
• Predictive: ML-forecast demand and scale ahead of it.

Pitfalls: cold starts (new nodes are slow until warm — pre-warm or keep a warm pool), flapping (scaling up and down rapidly — use cooldowns and hysteresis), and scaling a tier while its downstream DB cannot keep up (more app nodes → more DB connections → DB melts). Scale-out is not free downstream. Queue depth is often a better autoscaling signal than CPU for async workers.

Profiling to find bottlenecks

You cannot optimize what you have not measured, and intuition about bottlenecks is usually wrong. Use profilers (CPU, memory, allocation), flame graphs, distributed tracing (Observability), and load tests (k6, Gatling, Locust) to find the actual constraint. Apply Amdahl’s Law: the speedup from optimizing one part is limited by how much of the total time that part represents — optimize the biggest contributor to tail latency first. Common bottlenecks: N+1 queries, missing indexes, lock contention, synchronous calls that should be async, undersized connection pools, and serialization overhead.

Patterns summary table

Best Practices

• Design stateless from day one. Externalize sessions and state. It is the single highest-leverage decision for scale, and painful to retrofit.
• Put a timeout on every network call, and budget them so the sum fits your latency SLO. Infinite timeouts are the most common cause of cascading failure.
• Retry only idempotent operations, with bounded exponential backoff + full jitter and a retry budget. Never retry blindly.
• Wrap flaky dependencies in a circuit breaker with a defined fallback. Fail fast, do not pile up.
• Isolate resources with bulkheads so one bad dependency cannot drain the whole thread/connection pool.
• Bound your queues and buffers to get backpressure instead of out-of-memory crashes.
• Shed load intelligently under overload — reject low-priority work first, keep the core path alive; serving most users well beats crashing for all.
• Make health checks meaningful but non-cascading — separate liveness from readiness; do not let a DB blip mark every node dead.
• Kill single points of failure: redundant LBs, multi-AZ, DB failover. Ask “what if this one box dies?” for every component.
• Design idempotent APIs (idempotency keys, upserts) — assume every request may be delivered twice.
• Cache aggressively but plan invalidation and stampede protection (single-flight, jittered TTLs). See Caching.
• Define SLIs/SLOs from the user’s perspective and manage an error budget. Do not chase 100%. Measure p95/p99, never just the average.
• Optimize MTTR, not just MTBF — fast rollback, runbooks, good observability, and automated recovery beat trying to never fail.
• Load test to find your real breaking point and profile to find real bottlenecks — never guess. Provision with headroom.
• Practice chaos engineering and test your failover — untested redundancy is not redundancy.
• Remember availability multiplies — minimize deep synchronous dependency chains; prefer async and graceful degradation.
• Autoscale on the right signal (often queue depth over CPU), guard against cold starts and flapping, and watch downstream capacity.

References

• AWS Well-Architected Framework — Reliability Pillar
• Google SRE Book — Site Reliability Engineering (see chapters on SLOs, Handling Overload, Addressing Cascading Failures)
• Google SRE Workbook — Implementing SLOs
• Michael T. Nygard — Release It! Design and Deploy Production-Ready Software (2nd ed., Pragmatic Bookshelf)
• Timeouts, Retries, and Backoff with Jitter — AWS Builders’ Library
• Using Load Shedding to Avoid Overload — AWS Builders’ Library
• Martin Fowler — CircuitBreaker
• The AKF Scale Cube and Fallacies of Distributed Computing (Wikipedia)