Web Server, Proxy & Load BalancerWeb Servers, Proxies & Load Balancers

Part of the DevOps Roadmap knowledge base.

Overview

Between the internet and your application code sits a layer of web servers, proxies, and load balancers. This layer terminates TLS, serves static files, compresses and caches responses, routes requests to the right backend, absorbs traffic spikes, and shields applications from malformed or malicious traffic. Nearly every production HTTP request passes through at least one of these components — often several in sequence (CDN → cloud load balancer → Nginx ingress → app).

For DevOps engineers this layer is daily work: writing Nginx configs, setting up TLS termination, tuning cache headers, defining health checks, and choosing between L4 and L7 load balancing. It’s also the first place to look during incidents — 502/504 errors, connection resets, and TLS failures all surface here, and a proxy that logs upstream timing turns “the site is slow” into “backend #3 is slow.”

The concepts are portable. Whether the concrete tool is Nginx, HAProxy, Envoy, Traefik, an AWS ALB, or a Kubernetes ingress controller, the same ideas apply: reverse proxying, upstream pools, balancing algorithms, health checks, connection handling, and caching. Learn the model once and every tool becomes a matter of syntax.

Fundamentals

What a web server actually does

At its core a web server (a) accepts TCP connections, (b) parses HTTP requests, and (c) produces responses — either from static files on disk or by delegating to application code. The delegation mechanism matters:

• Static files — read from disk and stream, with Content-Type, caching, and range support. Web servers are extremely fast at this.
• Reverse proxy to an app — forward the request to an upstream over HTTP (proxy_pass) — the dominant pattern for modern apps (Node, Go, Python/Gunicorn, Java).
• FastCGI / uWSGI — a binary protocol to language runtimes (classic PHP-FPM, Python/uWSGI) — lower overhead than HTTP proxying for those stacks.
• Embedded module — run the language inside the server process (Apache mod_php) — simple but couples server and app lifecycle.

The other axis is the concurrency model, which determines how many connections a server handles under load.

Web servers compared

Nginx dominates the reverse-proxy/ingress niche due to its low memory footprint and high concurrency; Apache remains common where .htaccess per-directory flexibility matters (shared hosting, legacy PHP); Caddy wins on simplicity and is the fastest way to get correct, auto-renewing HTTPS; IIS is the default in Windows shops. Envoy and HAProxy deserve mention too: Envoy is the modern L7 proxy powering service meshes (Istio) and API gateways with rich observability and dynamic (xDS) config; HAProxy is a battle-tested high-performance load balancer strong at both L4 and L7.

Forward proxy vs reverse proxy

• A forward proxy sits in front of clients: the destination server sees the proxy, not the individual client. Uses: egress control and filtering, corporate policy enforcement, anonymity, caching outbound requests, and centralizing outbound IPs (e.g. Squid).
• A reverse proxy sits in front of servers: the client sees the proxy, not the backends. Uses: TLS termination, load balancing, caching, compression, rate limiting, request routing, and hiding internal topology (e.g. Nginx, HAProxy, Envoy, Traefik).

Memory aid: forward proxy hides the client; reverse proxy hides the server. An API gateway is a specialized reverse proxy adding auth, rate limiting, request transformation, and per-route policy for microservices.

Caching layers and CDNs

Caching is the highest-leverage performance tool at this layer — the fastest request is the one that never reaches your origin.

• Browser / client cache — controlled by Cache-Control (max-age, no-cache, private/public, immutable), ETag/If-None-Match, Last-Modified/If-Modified-Since, and Expires.
• Reverse-proxy cache — Nginx proxy_cache or Varnish (a dedicated HTTP accelerator with the powerful VCL configuration language) placed in front of slow origins.
• CDN — a globally distributed reverse-proxy cache (Cloudflare, CloudFront, Fastly, Akamai — Fastly is built on Varnish). Static assets and cacheable API responses are served from edge POPs near users, cutting latency and origin load and absorbing spikes and DDoS.

Key cache concepts:

• Cache key — usually the URL plus a selected set of headers; the Vary header declares which request headers (e.g. Accept-Encoding) split the cache into variants.
• TTL / freshness — how long an entry is served before revalidation.
• stale-while-revalidate / stale-if-error — serve a slightly stale copy instantly while refreshing in the background, or when the origin is down (huge availability win).
• Invalidation / purging — the hard problem: explicit purge (by URL or surrogate/cache tag), or cache-busting via versioned/hashed asset URLs (app.9f2a.js).
• Cacheability — only cache safe, idempotent responses; never cache authenticated per-user content without private and a user-scoped key.

Key Concepts

Load balancing algorithms

Health checks make balancing safe: passive (mark a backend down after N failed real requests, e.g. Nginx max_fails/fail_timeout) and active (a dedicated probe hitting /healthz on a schedule). Without health checks, a balancer happily routes traffic to a dead node. Distinguish liveness (is the process alive?) from readiness (can it serve traffic right now?) so an overloaded or warming-up node drains instead of flapping.

Session affinity (“sticky sessions”) pins a client to one backend (via cookie or IP hash) so in-memory session state stays consistent — but it defeats even load distribution and complicates scaling. Prefer stateless backends with shared session storage (Redis, JWT) and avoid affinity where you can.

L4 vs L7 load balancing

Rule of thumb: L7 for HTTP microservices (rich routing, retries, observability, TLS termination); L4 when you carry non-HTTP protocols, need extreme throughput with minimal CPU, want end-to-end TLS not to be broken (passthrough), or need to preserve the client source IP without header tricks. Many stacks use both: an L4 NLB in front to spread raw connections, then L7 proxies behind it for routing.

The proxy tier as a security boundary

The proxy tier is the natural enforcement point:

• Network isolation — cloud security groups allow only the load balancer to reach app instances (reference the LB’s security group, not CIDRs). Backends should never be directly reachable from the internet.
• WAF — ModSecurity (with the OWASP Core Rule Set), AWS WAF, or Cloudflare filters L7 attacks (SQLi, XSS, path traversal) in front of or on the proxy.
• Rate limiting — limit_req/limit_conn in Nginx blunts brute force, scraping, and application-layer DDoS far more cheaply than scaling out.
• TLS and security headers — centralized certificate management, HSTS, and header hardening at one choke point.
• Request hygiene — size limits (client_max_body_size), timeouts, and header normalization stop many malformed-request attacks.

Connection handling and timeouts

Two subtleties bite operators constantly:

• Keepalive to upstreams — without a keepalive directive and proxy_http_version 1.1, Nginx opens a fresh TCP (and possibly TLS) connection per request to the backend, adding latency and exhausting ports under load. Always configure an upstream keepalive pool.
• Timeouts — connect, send, and read timeouts must be set explicitly. Defaults are often too generous; one slow backend with unbounded proxy_read_timeout can pin every worker connection and take the whole proxy down (head-of-line at the proxy).

Sample Nginx configurations

Reverse proxy with an upstream pool and keepalive:

upstream app_backend {
    least_conn;
    server 10.0.1.10:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.12:8080 backup;              # only used if the others are down
    keepalive 32;                              # reuse upstream connections
}

server {
    listen 80;
    server_name app.example.com;

    location / {
        proxy_pass http://app_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection        "";  # required for upstream keepalive
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_connect_timeout 5s;
        proxy_send_timeout    30s;
        proxy_read_timeout    60s;
        proxy_next_upstream error timeout http_502 http_503;   # retry the next backend
    }
}

TLS termination (with HTTP→HTTPS redirect and modern settings):

server {
    listen 80;
    server_name app.example.com;
    return 301 https://$host$request_uri;      # force HTTPS
}

server {
    listen 443 ssl;
    http2 on;
    server_name app.example.com;

    ssl_certificate     /etc/letsencrypt/live/app.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/app.example.com/privkey.pem;
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers off;             # let TLS 1.3 clients choose
    ssl_session_cache   shared:SSL:10m;
    ssl_session_timeout 1d;
    ssl_stapling on;                           # OCSP stapling
    ssl_stapling_verify on;

    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Content-Type-Options "nosniff" always;
    server_tokens off;                         # hide Nginx version

    client_max_body_size 10m;

    location / {
        proxy_pass http://app_backend;
        proxy_set_header Host              $host;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    }
}

Proxy caching with stale-on-error and stampede protection:

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=api_cache:10m
                 max_size=1g inactive=60m use_temp_path=off;

server {
    location /api/ {
        proxy_cache api_cache;
        proxy_cache_key "$scheme$request_method$host$request_uri";
        proxy_cache_valid 200 302 10m;
        proxy_cache_valid 404      1m;
        proxy_cache_use_stale error timeout updating http_502 http_503 http_504;
        proxy_cache_background_update on;        # refresh in the background
        proxy_cache_lock on;                     # collapse concurrent misses (anti-stampede)
        add_header X-Cache-Status $upstream_cache_status;   # HIT / MISS / STALE
        proxy_pass http://app_backend;
    }
}

Serving static files with long-lived, immutable caching:

location /static/ {
    root /var/www/app;
    expires 1y;
    add_header Cache-Control "public, immutable";
    access_log off;
    gzip_static on;                              # serve pre-compressed .gz if present
}

Rate limiting an endpoint:

limit_req_zone $binary_remote_addr zone=login:10m rate=5r/s;

location /login {
    limit_req zone=login burst=10 nodelay;
    proxy_pass http://app_backend;
}

Operational commands:

nginx -t                     # validate config BEFORE applying — never reload blind
nginx -s reload              # graceful reload: new workers pick up config, old ones drain
nginx -T                     # dump the full effective config (all includes)
curl -I -H "Host: app.example.com" http://127.0.0.1/   # test routing locally without DNS

Caddy equivalent (automatic HTTPS in three lines):

app.example.com {
    reverse_proxy 10.0.1.10:8080 10.0.1.11:8080 {
        lb_policy least_conn
        health_uri /healthz
    }
}

Caddy provisions and renews the certificate automatically — no certbot, no cron — which is why it’s a favourite for small services and internal tools.

Best Practices

1. Always run nginx -t and use graceful reloads — config validation plus nginx -s reload means zero-downtime changes; a syntax error should never reach live traffic. In CI, run nginx -t against the rendered config in a container.
2. Terminate TLS at the edge, but consider re-encryption — TLS termination centralizes certificate management; for sensitive, regulated, or zero-trust environments, re-encrypt proxy→backend (or use a service mesh with mTLS). Never send credentials over plaintext internal hops in untrusted networks.
3. Forward the real client identity — set X-Forwarded-For, X-Forwarded-Proto, and Host, and configure the app to trust only your proxy (Nginx real_ip / set_real_ip_from). Otherwise logs, rate limits, geolocation, and auth decisions all use the proxy’s IP and are wrong (and spoofable).
4. Health checks must reflect real readiness — probe an endpoint that verifies critical dependencies, not just “process is up”; separate liveness from readiness so overloaded or starting nodes drain rather than flap.
5. Set explicit timeouts everywhere — connect/send/read on proxy and upstream. Defaults are often unbounded or too generous, and one slow backend can exhaust proxy worker connections and cascade into a full outage.
6. Configure upstream keepalive — keepalive in the upstream block plus proxy_http_version 1.1 and Connection "" avoids a new TCP/TLS handshake per request; a frequent, invisible source of latency and port exhaustion.
7. Serve stale on backend failure — proxy_cache_use_stale (Nginx) or grace mode (Varnish) keeps users served during origin outages; pair with proxy_cache_lock/proxy_cache_background_update to prevent cache stampedes when a popular entry expires.
8. Never expose backends directly — security groups should accept traffic only from the load balancer/proxy tier (reference its security group); verify with an external port scan and periodic audits.
9. Rate limit and size-limit at the edge — limit_req/limit_conn per IP or token and client_max_body_size protect backends from abuse and resource-exhaustion attacks far more cheaply than scaling out.
10. Enable HTTP/2 (and HTTP/3 where supported) toward clients — multiplexing meaningfully improves page load; the proxy→backend hop can stay HTTP/1.1 with keepalive, which is simpler and usually sufficient.
11. Cache aggressively but correctly — long-lived immutable caching for hashed static assets, short TTLs with revalidation for API responses, correct Vary, and never cache per-user authenticated content publicly. Emit X-Cache-Status so you can see hit rates.
12. Hide server details and send security headers — server_tokens off;, plus HSTS, X-Content-Type-Options: nosniff, X-Frame-Options/frame-ancestors, and a Content-Security-Policy appropriate to the app.
13. Compress responses — gzip or Brotli for text/JSON/HTML/CSS/JS (not for already-compressed images/video); prefer pre-compressed static assets (gzip_static). Big bandwidth and latency wins for little cost.
14. Log in structured form with cache/upstream fields — include $upstream_addr, $upstream_status, $upstream_response_time, and $upstream_cache_status; 5xx and latency triage becomes trivial when you can see which backend served the request and whether the cache was hit.
15. Keep proxy configs in version control and template them — configs are code: review them in PRs, lint in CI (nginx -t in a container), render per-environment with a templating tool, deploy via configuration management or a Kubernetes ingress controller, and never hand-edit on live servers.
16. Tune worker and connection limits for the workload — worker_processes auto;, sufficient worker_connections, and OS-level file-descriptor limits; the defaults assume a small box and silently cap throughput on larger ones.

References

• roadmap.sh DevOps roadmap: https://roadmap.sh/devops
• Nginx documentation: https://nginx.org/en/docs/
• Nginx admin guide (load balancing, caching): https://docs.nginx.com/nginx/admin-guide/
• Nginx ngx_http_proxy_module reference: https://nginx.org/en/docs/http/ngx_http_proxy_module.html
• Apache HTTP Server documentation: https://httpd.apache.org/docs/
• Caddy documentation: https://caddyserver.com/docs/
• HAProxy documentation: https://docs.haproxy.org/
• Varnish Cache documentation: https://varnish-cache.org/docs/
• Envoy Proxy documentation: https://www.envoyproxy.io/docs
• Traefik documentation: https://doc.traefik.io/traefik/
• Cloudflare Learning — What is a reverse proxy: https://www.cloudflare.com/learning/cdn/glossary/reverse-proxy/
• Cloudflare Learning — What is a CDN: https://www.cloudflare.com/learning/cdn/what-is-a-cdn/
• Mozilla SSL Configuration Generator: https://ssl-config.mozilla.org/
• MDN — HTTP caching: https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching
• OWASP ModSecurity Core Rule Set: https://coreruleset.org/
• RFC 9111 — HTTP Caching: https://www.rfc-editor.org/rfc/rfc9111