Dự phòng & High AvailabilityRedundancy & High Availability

Part of the Network Engineer Roadmap knowledge base.

Overview

Every network eventually breaks. A power supply dies, an optic goes dark, a config change fat-fingers a routing table, a switch reboots for a firmware bug at 3 a.m. The difference between a network that hiccups and a network that goes down is redundancy — having a second path, a second device, or a second data center ready to carry the load — combined with fast, automatic failover so that the spare takes over before users (or SLAs) notice.

High Availability (HA) is the discipline of engineering systems to keep serving despite component failures. It is not a single feature you switch on; it is a property that emerges from decisions at every layer:

• Physical: dual power feeds, dual supervisors, dual NICs, diverse fiber runs.
• Link layer: link aggregation, redundant uplinks, loop-free topologies (STP/RSTP — see ./08-switching.md).
• Network layer: multiple routers, First Hop Redundancy Protocols (FHRP), equal-cost multipath (ECMP), dynamic routing convergence.
• Application/service layer: load balancers distributing across a pool of servers, health checks pulling dead nodes, geographic redundancy across regions.

The unifying goal is the elimination of single points of failure (SPOFs). A SPOF is any element whose failure takes down the whole service. If your two redundant firewalls both plug into one switch, that switch is a SPOF. If your two switches share one power strip, that strip is a SPOF. HA design is largely the systematic hunt for and removal of SPOFs.

This note covers the math of availability, First Hop Redundancy Protocols (HSRP, VRRP, GLBP), failover concepts (active-passive vs active-active, heartbeat, split-brain, preemption), load balancing in depth (L4 vs L7, algorithms, health checks, session persistence, SSL termination), link and path redundancy, and how to assemble these into resilient topologies. Related notes:

• Network devices (including load balancers as appliances): ./04-network-devices.md
• Switching, VLANs, STP, EtherChannel: ./08-switching.md
• Traffic management and QoS: ./13-traffic-management-and-qos.md
• Cloud networking (managed LBs, multi-AZ/region): ./16-cloud-networking.md

Fundamentals

Why HA matters: SPOFs, cost of downtime, and blast radius

Downtime is expensive and asymmetric. A few seconds of packet loss during a clean failover is invisible; a 30-minute outage during business hours can cost revenue, breach an SLA, and trigger contractual penalties. The business drivers for HA are:

• Revenue protection — e-commerce, payments, and SaaS lose money per minute of downtime.
• SLA compliance — contracts promise “three nines” or “four nines”; missing them incurs credits/penalties.
• Safety and compliance — hospitals, industrial control, and financial systems have regulatory uptime requirements.
• Reputation — repeated outages erode customer trust faster than almost anything else.

The engineering job is to shrink the blast radius of any single failure so that no one component failing can take everything down.

The “nines”: availability math

Availability is the fraction of time a system is operational, usually expressed as a percentage and colloquially as a number of “nines”:

Availability = Uptime / (Uptime + Downtime)
             = MTBF / (MTBF + MTTR)

Two crucial insights:

1. Nines are multiplicative in series. If a request must traverse a firewall (99.99%), a load balancer (99.99%), and an app server (99.9%), the end-to-end availability is 0.9999 × 0.9999 × 0.999 ≈ 99.88% — worse than any single component. Chains of dependencies drag availability down.
2. Redundancy in parallel multiplies the failure probability. Two independent components each with 99% availability (1% failure) placed in parallel fail together only 0.01 × 0.01 = 0.0001 of the time → combined availability 99.99%. This is why redundancy is so powerful — but only if the components fail independently (no shared power, no shared fiber, no shared bug triggered by the same input).

MTBF and MTTR

• MTBF (Mean Time Between Failures) — average operational time between failures. Improved by higher-quality hardware, better cooling, redundant internal components. You have limited control over MTBF once you buy the gear.
• MTTR (Mean Time To Repair/Recover) — average time to restore service after a failure. This is where operators have enormous leverage: automated failover, good monitoring, spare parts on site, runbooks, and practice all crush MTTR.

Because Availability = MTBF / (MTBF + MTTR), halving MTTR improves availability just as much as doubling MTBF — and is usually far cheaper. Fast, automatic failover is the highest-ROI HA investment. A sub-second FHRP failover turns a device reboot from an outage into a non-event.

RTO and RPO (the DR context)

HA (keeping running) is adjacent to Disaster Recovery (DR) (recovering after a major loss). Two DR metrics frame the requirements:

• RTO (Recovery Time Objective) — the maximum acceptable time to restore service after a disaster. “We must be back within 15 minutes.”
• RPO (Recovery Point Objective) — the maximum acceptable data loss measured in time. “We can lose at most 5 minutes of transactions.”

HA mechanisms (FHRP, load balancers, clustering) target near-zero RTO for component failures. DR mechanisms (backups, replication, standby sites) target bounded RTO/RPO for catastrophic failures. A complete design uses both: HA for the common case (a device dies), DR for the rare case (a site is lost).

Key Concepts

First Hop Redundancy Protocols (FHRP)

Hosts are configured with a single default gateway IP. If that gateway (router/L3 switch) dies, every host on the subnet loses off-subnet connectivity — a classic SPOF. First Hop Redundancy Protocols solve this by letting two or more routers share a virtual IP (VIP) and virtual MAC that acts as the gateway. Hosts point at the VIP; the routers elect an active forwarder; if it fails, a standby takes over the VIP/MAC with no reconfiguration on the hosts.

The magic is that the virtual MAC moves with the role. Because hosts have the VIP→virtual-MAC mapping in their ARP cache, and the new active router starts answering for that same virtual MAC, the hosts’ ARP entries stay valid — traffic just starts flowing to a different physical box. Failover is transparent to end stations.

HSRP (Hot Standby Router Protocol) — Cisco proprietary

• Cisco proprietary (defined in RFC 2281, but Cisco-owned).
• Routers form a group; one is Active, one is Standby, others are Listen.
• Election by priority (default 100, higher wins); tie broken by highest IP.
• Virtual MAC: 0000.0C07.ACxx (v1) where xx is the group number; v2 uses 0000.0C9F.Fxxx.
• Hello timer default 3 s, hold timer default 10 s → failover ~10 s by default (tunable to sub-second).
• Communicates via multicast 224.0.0.2 (v1) / 224.0.0.102 (v2), UDP 1985.
• Supports interface tracking (decrement priority if an uplink dies) and preemption (higher-priority router reclaims Active when it returns).
• Load sharing is achieved only by running multiple groups and pointing different hosts/VLANs at different VIPs (Multiple HSRP / MHSRP).

! Cisco HSRP — two routers, VIP 10.1.1.1, group 10
! Router A (primary)
interface Vlan10
 ip address 10.1.1.2 255.255.255.0
 standby version 2
 standby 10 ip 10.1.1.1
 standby 10 priority 110
 standby 10 preempt
 standby 10 timers msec 250 msec 750      ! hello 250ms / hold 750ms
 standby 10 track GigabitEthernet0/1 20   ! -20 priority if uplink down
!
! Router B (secondary) — same VIP, lower priority
interface Vlan10
 ip address 10.1.1.3 255.255.255.0
 standby version 2
 standby 10 ip 10.1.1.1
 standby 10 priority 100
 standby 10 preempt

VRRP (Virtual Router Redundancy Protocol) — open standard, RFC 5798

• Open standard (RFC 5798 for VRRPv3, covering IPv4 and IPv6; RFC 3768 was VRRPv2). Multi-vendor.
• Roles are Master and Backup (analogous to HSRP Active/Standby).
• Election by priority (1–254, default 100, higher wins); the address owner (a router whose real interface IP equals the VIP) uses priority 255 and always wins.
• Virtual MAC: 0000.5E00.01xx where xx is the VRID (Virtual Router ID).
• Advertisement interval default 1 s; master-down interval ≈ 3 × interval + skew → default failover ~3 s (tunable to sub-second with centiseconds in v3).
• Communicates via multicast 224.0.0.18, IP protocol 112.
• Preemption is ON by default (unlike HSRP where you must enable it).
• Load sharing again requires multiple VRRP groups (VRIDs).

! VRRP (works across vendors; Cisco syntax shown), VIP 10.1.1.1, VRID 10
interface Vlan10
 ip address 10.1.1.2 255.255.255.0
 vrrp 10 ip 10.1.1.1
 vrrp 10 priority 110
 vrrp 10 timers advertise 1
 vrrp 10 preempt          ! on by default; shown for clarity

GLBP (Gateway Load Balancing Protocol) — Cisco, active-active load sharing

• Cisco proprietary. Its differentiator: native load balancing across multiple routers using a single VIP.
• One router is elected AVG (Active Virtual Gateway); it hands out up to four different virtual MACs to hosts as they ARP for the VIP.
• Each participating router is an AVF (Active Virtual Forwarder) owning one virtual MAC and actually forwarding the traffic sent to it.
• Load-balancing methods: round-robin (default), weighted (proportional to configured weight), host-dependent (a given host always gets the same MAC → sticky).
• If an AVF fails, another AVF takes over its virtual MAC → still transparent to hosts.
• Multicast 224.0.0.102, UDP 3222.

! GLBP — active-active, single VIP 10.1.1.1, group 10
! Router A
interface Vlan10
 ip address 10.1.1.2 255.255.255.0
 glbp 10 ip 10.1.1.1
 glbp 10 priority 110
 glbp 10 preempt
 glbp 10 load-balancing weighted
 glbp 10 weighting 100
! Router B
interface Vlan10
 ip address 10.1.1.3 255.255.255.0
 glbp 10 ip 10.1.1.1
 glbp 10 priority 100
 glbp 10 preempt
 glbp 10 load-balancing weighted
 glbp 10 weighting 80

FHRP comparison

Tip: All three support object/interface tracking so a router that loses its upstream link lowers its priority and steps aside, avoiding the “black hole” where the active gateway is up on the LAN side but has no path out.

Failover concepts

Active-passive vs active-active

A key active-active caveat: if two nodes each run at 70% and one dies, the survivor needs 140% — it will be overloaded. Plan active-active capacity so any single node can absorb the failover load.

Heartbeat

Redundant peers exchange periodic heartbeat messages (FHRP hellos, LB cluster keepalives, firewall HA links) to know each other is alive. Losing N consecutive heartbeats declares the peer dead and triggers failover. Tuning is a trade-off:

• Aggressive timers (sub-second) → fast failover but risk false positives (declaring a healthy-but-busy peer dead during a CPU spike or transient link flap), causing flapping.
• Conservative timers → stable but slow failover.

Use BFD (Bidirectional Forwarding Detection) for fast, low-overhead liveness detection that many FHRPs and routing protocols can hook into for sub-second convergence without over-aggressive protocol timers.

Split-brain

Split-brain is the dangerous failure where the heartbeat/communication link between peers breaks but both peers are actually alive. Each concludes the other is dead and both become active — two masters answering for the same VIP, or two firewalls both holding the active state, causing MAC flaps, duplicate IPs, corrupted shared state, and asymmetric routing. Mitigations:

• Dedicated, redundant heartbeat links (separate from the data path) so a single data-link failure doesn’t isolate the control plane.
• Quorum / witness / tie-breaker — a third voter (an odd number of nodes, or an external witness) so a minority partition steps down.
• Fencing / STONITH (“Shoot The Other Node In The Head”) — forcibly power off or isolate the peer before taking over shared resources.

Preemption

Preemption governs what happens when a higher-priority device recovers:

• Preempt enabled — the recovered primary reclaims the active role. Restores intended traffic distribution but causes a second failover event (another brief blip).
• Preempt disabled — whoever is currently active stays active until it fails. Fewer disruptions, but traffic may run “backwards” on the intended-secondary indefinitely.

Best practice: enable preemption with a delay (preempt delay minimum) so a rebooting router fully converges its routing table and forwarding plane before grabbing the active role — otherwise it becomes active while still black-holing traffic.

Load balancing (in depth)

A load balancer (LB) presents a single virtual IP (VIP) to clients and distributes incoming connections/requests across a pool of backend servers (“real servers”). It provides scale (add servers behind one address), availability (remove failed servers automatically via health checks), and flexibility (rolling deploys, SSL offload, routing).

L4 vs L7 load balancing

L4 is chosen for raw speed and non-HTTP protocols; L7 is chosen when you need to route by URL/host, do TLS termination, rewrite headers, or apply application-aware policy.

Load balancing algorithms

Rule of thumb: start with round-robin (or least-connections if request durations vary a lot), add weights for uneven hardware, and reach for hashing/consistent hashing only when you need stickiness or cache affinity.

Health checks

Health checks are what make an LB an availability tool rather than just a distributor. The LB continuously probes each backend and removes unhealthy ones from rotation:

• Passive — observe live traffic; mark a server down after N failed real requests (fast to react, but a few users hit the failure first).
• Active — the LB sends synthetic probes on an interval:
  • L3: ICMP ping (server reachable?).
  • L4: TCP connect to the port (is the service listening?).
  • L7: HTTP GET to a /healthz endpoint expecting 200 OK (is the app actually working, DB reachable, etc.?).
• Tunables: interval, timeout, rise (consecutive successes to mark healthy), fall (consecutive failures to mark unhealthy). Deep L7 checks catch “up but broken” servers that L3/L4 checks would keep in rotation.

Session persistence (sticky sessions)

Some applications keep per-user state in server memory, so a user’s requests must return to the same backend. Persistence methods:

• Source IP affinity — hash client IP → same server. Simple, no app cooperation, but breaks fairness behind large NATs and breaks on IP change (mobile).
• Cookie-based — LB inserts its own cookie (e.g., HAProxy SRV=s1) or learns an existing app cookie (e.g., JSESSIONID) to pin the session. Precise and per-session.
• SSL session ID — pin by TLS session (less common now).

Best practice: prefer stateless backends (store session in a shared store like Redis or a signed token) so you can load balance freely and lose no sessions on failover. Sticky sessions are a crutch that hurts even distribution and complicates rolling deploys.

SSL/TLS termination

• TLS termination — the LB holds the certificate, decrypts TLS, and talks plaintext HTTP to backends. Offloads crypto CPU from app servers, centralizes cert management, and enables L7 inspection/routing. Traffic between LB and backends should stay on a trusted segment.
• TLS re-encryption (bridging) — LB terminates, inspects, then re-encrypts to the backend. Needed for end-to-end encryption / compliance.
• TLS pass-through — LB forwards encrypted bytes untouched (L4). Backends do their own TLS; LB cannot inspect L7. Used when the LB must not see plaintext or when using SNI-based routing at L4.

Hardware vs software vs cloud load balancers

# HAProxy — L7, round-robin with health checks, cookie stickiness, TLS termination
global
    maxconn 20000

defaults
    mode http
    timeout connect 5s
    timeout client  30s
    timeout server  30s
    option httpchk GET /healthz

frontend web_in
    bind :80
    bind :443 ssl crt /etc/haproxy/certs/site.pem   # TLS termination
    redirect scheme https if !{ ssl_fc }
    default_backend web_pool

backend web_pool
    balance roundrobin
    cookie SRV insert indirect nocache               # sticky sessions
    server web1 10.0.1.11:8080 check inter 2s fall 3 rise 2 cookie s1 weight 100
    server web2 10.0.1.12:8080 check inter 2s fall 3 rise 2 cookie s2 weight 100
    server web3 10.0.1.13:8080 check inter 2s fall 3 rise 2 cookie s3 weight 50

# NGINX — least-connections upstream with health-relevant options
upstream app {
    least_conn;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.13:8080 backup;      # only used if others are down
}
server {
    listen 443 ssl;
    ssl_certificate     /etc/nginx/site.crt;
    ssl_certificate_key /etc/nginx/site.key;
    location / {
        proxy_pass http://app;
        proxy_next_upstream error timeout http_502 http_503;
    }
}

Link and path redundancy

FHRP protects the gateway; the paths between devices need their own redundancy.

Link aggregation (LAG / EtherChannel / bond)

Bundle multiple physical links between two devices into one logical link (IEEE 802.3ad / LACP; Cisco EtherChannel; Linux bonding). Benefits: higher aggregate bandwidth and link-level redundancy — if one member fails, traffic redistributes over the survivors without STP reconvergence. Traffic is hashed across members (by MAC/IP/port), so a single flow does not exceed one member’s speed. See ./08-switching.md.

! Cisco LACP EtherChannel
interface range GigabitEthernet0/1 - 2
 channel-group 1 mode active        ! LACP
interface Port-channel1
 switchport mode trunk

Redundant uplinks and STP

Two uplinks from an access switch to two distribution switches create a physical loop. STP/RSTP blocks the redundant path to prevent broadcast storms, then unblocks it in ~seconds if the primary fails (RSTP converges much faster than classic STP). Modern designs prefer MLAG / vPC / stacking so both uplinks are active (no STP-blocked port) while still being loop-free — active-active uplinks instead of active-passive. Details in ./08-switching.md.

ECMP (Equal-Cost Multi-Path)

At Layer 3, when the routing table has multiple equal-cost paths to a destination, the router installs all of them and hashes flows across them — active-active load sharing and redundancy at the routing layer. If one next-hop fails, its share redistributes over the remaining paths on the next routing update (or immediately with BFD). ECMP underpins spine-leaf data-center fabrics and anycast. Flows are hashed per-5-tuple so a single flow stays on one path (avoids packet reordering).

Comparing the redundancy layers

Designing redundant topologies

HA is layered — each tier removes SPOFs at its level:

1. Component redundancy — dual power supplies (on separate PDUs/feeds), dual supervisors/route processors, dual NICs bonded, hot-swappable fans.
2. Device redundancy — pairs of switches/routers/firewalls/LBs, tied together with FHRP (gateways), HA clustering (firewalls/LBs), and MLAG (switches).
3. Link/path redundancy — dual uplinks (MLAG/ECMP), LACP bundles, and physically diverse fiber runs so a single backhoe cut can’t sever both.
4. Site/geographic redundancy — replicate the service across data centers / cloud regions/AZs. Steer traffic with GSLB / DNS-based load balancing, anycast, or provider global LBs. This is where RTO/RPO and DR meet HA.

Canonical enterprise pattern — collapsed-core / two-tier: dual distribution/core switches (MLAG), each access switch dual-homed to both, FHRP (or an SVI on the MLAG pair) for the gateway VIP, dual firewalls in an HA pair, dual routers to two ISPs running BGP for internet redundancy.

Rules of thumb:

• Remove SPOFs pair-by-pair; after each pair, ask “what single thing failing still takes us down?”
• Ensure failures are independent: separate power feeds, separate physical paths, avoid shared fate. Two “redundant” links in the same conduit are one link.
• Test failover regularly (game days / chaos drills). Untested failover is a hypothesis, not a feature — standby configs rot, certs expire, and preemption misfires are only found by pulling cables.
• Size active-active tiers for N+1: any one node’s death must be absorbable by the rest.
• Automate and monitor: alert on the loss of redundancy (a failed-but-covered link), not just on outages — otherwise you silently run without a safety net until the second failure.

Best Practices

• Eliminate SPOFs systematically. Diagram the traffic path and, for each element, ask “if this dies, are we down?” Redundant-pair anything that answers yes — including power, cabling, and cooling, not just the boxes.
• Optimize MTTR, not just MTBF. Fast automatic failover, good monitoring, and rehearsed runbooks buy more availability per dollar than premium hardware.
• Tune FHRP timers deliberately. Defaults (~3–10 s) are conservative. Use sub-second timers or BFD for fast failover, but not so aggressive that transient load spikes cause false failovers/flapping.
• Always enable interface/object tracking on FHRP so a gateway that loses its uplink steps down instead of black-holing traffic.
• Enable preemption with a delay so a recovering device converges before reclaiming the active role.
• Guard against split-brain with dedicated redundant heartbeat links plus quorum/witness or fencing.
• Make backends stateless. Externalize sessions (Redis/token) so you can load balance freely and lose nothing on failover; treat sticky sessions as a last resort.
• Use deep (L7) health checks hitting a real dependency-aware endpoint, so “up but broken” servers are pulled from rotation.
• Terminate TLS at the LB for CPU offload and central cert management; re-encrypt to backends where compliance requires end-to-end encryption.
• Plan active-active for N+1 headroom. If losing one node overloads the rest, you have a slower outage, not redundancy.
• Ensure independent failure domains. Separate power feeds, physically diverse fiber, and beware shared components/bugs that defeat statistical independence.
• Test failover regularly. Schedule failover drills; untested redundancy fails when you need it most.
• Alert on lost redundancy, not only on outages, so you fix the safety net before the second failure lands.
• Layer HA and DR together. HA (FHRP, LB, clustering) for common component failures with near-zero RTO; DR (replication, standby sites) for catastrophes with bounded RTO/RPO.

References

• RFC 5798 — Virtual Router Redundancy Protocol (VRRP) Version 3
• RFC 2281 — Cisco Hot Standby Router Protocol (HSRP)
• Cisco — First Hop Redundancy Protocols (HSRP/VRRP/GLBP) Configuration Guide
• Cisco — GLBP Overview and Configuration
• HAProxy Documentation
• NGINX — HTTP Load Balancing Guide
• AWS — Elastic Load Balancing (ALB/NLB) Documentation
• Google SRE Book — Availability, MTBF/MTTR, and the “nines”