ContainersContainers

Part of the DevOps Roadmap knowledge base.

Overview

Containers are a lightweight virtualization technique that packages an application together with its dependencies (libraries, runtime, configuration) into a single, portable unit that runs consistently on any host with a compatible kernel. Unlike virtual machines, containers do not ship a guest operating system — they share the host’s kernel and isolate processes using native Linux features. The result: startup in milliseconds instead of minutes, images measured in megabytes instead of gigabytes, and far higher density per host.

For DevOps, containers solve the classic “works on my machine” problem. The same image that a developer builds locally is what CI tests and what production runs, eliminating environment drift. Containers are also the foundational building block for everything downstream in the roadmap: container orchestration (Kubernetes), immutable infrastructure, GitOps, and modern CI/CD pipelines all assume containerized workloads.

Docker popularized the format, but the ecosystem is now standardized around the OCI (Open Container Initiative) specifications for images and runtimes, so images built with Docker run under Podman, containerd, CRI-O, and any other OCI-compliant runtime.

A mental model

Think of an image as a class and a container as an instance of that class. The image is a read-only, versioned template; a container is a running (or stopped) process created from it, with its own writable layer, its own PID, and its own isolated view of the system. You build images once and run many containers from them. This is why “rebuild the image, redeploy” is the correct fix for a bad container, never “SSH in and patch it” — you would be mutating an instance whose class no longer describes it.

Three ideas underpin everything else on this page:

• Isolation is a kernel feature, not a boundary of hardware. Containers are ordinary processes that the kernel has been told to show a restricted view of the world.
• Images are immutable and content-addressed. Identical layers are stored once and shared, which is what makes containers cheap.
• Containers are cattle, not pets. They are disposable; state lives in volumes and external systems, never inside the container.

Fundamentals

Containers vs Virtual Machines

They are complementary: most production clusters run containers inside VMs (e.g., EC2 nodes in an EKS cluster). The VM gives you a hard security boundary between tenants and the freedom to pick a kernel; the container gives you fast, dense packing of workloads on top.

When containers are the wrong tool: you need a different kernel than the host (a Windows container cannot run on a Linux kernel and vice versa), you require hard multi-tenant isolation for untrusted code (use a VM, or a sandboxed runtime like gVisor / Kata Containers / Firecracker microVMs), or the workload is a monolith with deep host coupling that gains nothing from packaging.

Linux underpinnings

Containers are ordinary Linux processes wrapped in three kernel mechanisms:

• Namespaces — give a process its own isolated view of a system resource. Key namespaces: pid (process tree — PID 1 inside is not PID 1 on the host), net (network interfaces, routing tables, iptables), mnt (mount points), uts (hostname and domain name), ipc (shared memory, semaphores), user (UID/GID mapping — lets root inside map to an unprivileged host UID), cgroup (view of the cgroup hierarchy), and time.
• cgroups (control groups) — limit and account for resource usage: CPU shares/quotas, memory limits, block I/O throttling, PID counts. This is what enforces --memory=512m or Kubernetes resource limits. cgroups v2 (unified hierarchy) is now the default on modern distros and is what Kubernetes prefers.
• Union filesystems (OverlayFS is the common driver) — stack read-only image layers with a thin writable layer on top via a copy-on-write mechanism. Layers are content-addressed by digest and shared between containers, which is why 50 containers from the same image cost almost no extra disk. Writing to a file from a lower layer copies it up into the writable layer first (copy-up).

Two more primitives round out container security:

• Capabilities — Linux splits root’s powers into ~40 discrete capabilities (CAP_NET_BIND_SERVICE, CAP_SYS_ADMIN, …). Containers drop most by default; you can drop the rest with --cap-drop=ALL and add back only what you need.
• seccomp — filters which syscalls a process may make. Docker ships a default seccomp profile that blocks dangerous syscalls; combined with AppArmor/SELinux (MAC) it hardens the container further.

# See it yourself: a container is just a process on the host
docker run -d --name web nginx
docker inspect web --format '{{.State.Pid}}'   # host PID
sudo ls /proc/<pid>/ns/                          # its namespaces (symlinks with inode ids)
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/memory.max   # cgroup v2 memory limit
docker top web                                   # processes as seen from the host

Docker architecture

docker CLI ──REST──> dockerd (daemon) ──gRPC──> containerd ──> containerd-shim ──> runc ──> container process

• docker CLI — client that talks to the daemon over a Unix socket (/var/run/docker.sock) or TCP.
• dockerd — API server; handles builds, networking, volumes, and image management. Historically a large monolith; much of its work now delegates downward.
• containerd — the actual container lifecycle manager: pulls images, manages snapshots, and supervises containers. Also used directly by Kubernetes via the CRI, which is why “Docker” was removed as a Kubernetes runtime (dockershim) — the cluster talks to containerd underneath either way.
• containerd-shim — a small process that stays alive per container so the container keeps running even if containerd or dockerd restarts. It also reaps the process and reports exit status.
• runc — the OCI reference runtime; sets up namespaces/cgroups and executes the process, then exits. Being short-lived is deliberate — the shim, not runc, is the long-lived parent.
• OCI specs — image-spec (layer + manifest format), runtime-spec (how to run a filesystem bundle + config.json), and distribution-spec (registry API). Any compliant tool can interoperate. Alternative runtimes plug in at the runc layer: gVisor (runsc) intercepts syscalls in userspace for stronger isolation, and Kata Containers wraps each container in a lightweight VM.

The daemon model has a downside: dockerd runs as root and is a single point of failure. That is precisely the gap daemonless tools like Podman fill.

Images, layers, and registries

An image is a manifest pointing to an ordered list of read-only tarball layers plus a config object (entrypoint, env, working dir, user, exposed ports). Each Dockerfile instruction that changes the filesystem creates a layer. Layers are cached and reused: if a layer’s inputs haven’t changed, rebuilds skip it and reuse the cached digest.

• Content addressing — every layer and the config are identified by a SHA-256 digest. Two images sharing a base layer store that layer once.
• Tags vs digests — a tag (nginx:1.27) is a mutable pointer that can be moved; a digest (nginx@sha256:...) is immutable. Pin digests for reproducibility.
• Manifest lists (multi-arch) — one tag can point to a manifest list that resolves to the right image for linux/amd64, linux/arm64, etc. docker buildx builds these.
• Registries — Docker Hub, GHCR, Amazon ECR, Google Artifact Registry, Azure ACR, and self-hosted Harbor store and distribute images over the distribution-spec API. Registries handle auth, retention, and often built-in vulnerability scanning.

Key Concepts

Anatomy of a Dockerfile

The most common instructions, and what they do:

ENTRYPOINT vs CMD: use ENTRYPOINT ["/app"] for the binary and CMD ["--serve"] for default, overridable arguments. Always prefer the exec form (["/bin/sh", "-c", ...] JSON array) over the shell form — the shell form wraps your process in /bin/sh -c, which becomes PID 1 and does not forward signals, so docker stop will hang until the timeout and hard-kill your app.

Dockerfile best practices in action

# ---- build stage ----
FROM golang:1.22-alpine AS build
WORKDIR /src
# Copy dependency manifests first: this layer only invalidates when deps change
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o /out/app ./cmd/app

# ---- runtime stage ----
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/app /app
USER nonroot:nonroot
EXPOSE 8080
HEALTHCHECK --interval=30s --timeout=3s CMD ["/app", "healthcheck"]
ENTRYPOINT ["/app"]

Why this works:

• Multi-stage build — compilers and source never reach production; the final image is a few MB. Each FROM starts a fresh stage, and only what you COPY --from survives.
• Layer-cache-friendly ordering — least-changing instructions first (deps before source). A source edit invalidates only the final layers, not go mod download.
• Small base image — alpine, distroless, or scratch shrink attack surface and pull time. Distroless ships no shell or package manager, so there is nothing for an attacker to pivot with.
• Non-root user — a container escape from root is far more dangerous than from an unprivileged UID.

A worked cache pitfall: putting COPY . . before RUN npm install means every source change busts the dependency-install layer, forcing a full reinstall on every build. Copy package.json/package-lock.json first, install, then copy the rest. The same principle applies to go.mod, requirements.txt, pom.xml, and Cargo.toml.

BuildKit is the modern builder (default in current Docker). It builds layers in parallel, supports build-time secret mounts (RUN --mount=type=secret), cache mounts (RUN --mount=type=cache,target=/root/.cache), and cross-platform builds via buildx.

docker-compose

Compose describes multi-container applications declaratively — the standard tool for local development environments and simple single-host deployments:

services:
  api:
    build: .
    ports: ["8080:8080"]
    environment:
      DATABASE_URL: postgres://app:secret@db:5432/app
    depends_on:
      db:
        condition: service_healthy
    restart: unless-stopped
  db:
    image: postgres:16-alpine
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app"]
      interval: 5s
      timeout: 3s
      retries: 5
volumes:
  pgdata:

docker compose up -d starts everything on a shared user-defined bridge network where services resolve each other by name (db). depends_on with condition: service_healthy waits for the DB’s health check before starting the API — plain depends_on only waits for the container to start, not for the app to be ready. Compose is excellent for dev and CI; for production at scale, graduate to Kubernetes.

Networking and volumes

The default bridge and a user-defined bridge differ in one important way: only user-defined bridges provide automatic DNS-based service discovery by container name. Compose and docker network create give you user-defined bridges; the legacy default bridge requires --link (deprecated).

Storage:

• Volumes (docker volume create, -v data:/path) — managed by Docker under /var/lib/docker/volumes, survive container removal; the right choice for databases and any persistent state. Can be backed by drivers (NFS, cloud block storage).
• Bind mounts (-v $(pwd):/app) — map a specific host directory into the container; great for live-reload in development but couple you to the host’s filesystem layout.
• tmpfs — in-memory, never written to disk; for secrets and scratch data.

Container filesystems are ephemeral by design — anything not in a volume is lost when the container is removed. The writable layer is also slow (copy-on-write overhead) and not meant for heavy I/O; put databases and hot paths on volumes.

Alternatives to Docker

Common pitfalls

• PID 1 and zombie reaping — your app as PID 1 must reap child processes and handle SIGTERM, or you leak zombies and ignore graceful shutdown. Use --init (Docker) or tini as a tiny init, or handle signals in your app.
• Latest tag drift — image:latest resolves to different content over time; builds and rollbacks become non-reproducible.
• Secrets baked into layers — deleting a secret in a later layer does not remove it; it is still recoverable from the earlier layer. Never COPY a secret, then RUN rm.
• Fat images — installing build tools in the runtime image bloats it and widens the attack surface; multi-stage fixes this.
• Root by default — most base images run as root; a compromised process then has root inside the container and, if the container is privileged or the socket is mounted, potentially on the host.
• Mounting the Docker socket (-v /var/run/docker.sock) — this is effectively root on the host. Avoid; use rootless builders or a socket proxy with a restricted API.
• Ignoring .dockerignore — a bloated build context slows builds and can leak .git/.env into images.

Best Practices

1. Use multi-stage builds — keep compilers, dev dependencies, and source code out of runtime images; smaller images pull faster, start faster, and expose less attack surface.
2. Pin base images by tag and ideally digest — python:3.12-slim@sha256:... instead of python:latest; unpinned tags make builds non-reproducible and rollbacks meaningless.
3. Run as a non-root user — add USER in the Dockerfile; combine with a read-only root filesystem (--read-only) and --cap-drop=ALL so a compromise has minimal privilege.
4. Order Dockerfile instructions for cache efficiency — copy dependency manifests and install deps before copying the full source tree; put the most frequently changing content last.
5. One process/concern per container — enables independent scaling, restarts, and log streams; use Compose or Kubernetes Pods for co-located helpers instead of stuffing multiple daemons into one image.
6. Add a .dockerignore — exclude .git, node_modules, secrets, and build artifacts from the build context; speeds builds and prevents accidental leaks into layers.
7. Never bake secrets into images — layers are permanent and recoverable; use runtime env vars, mounted secrets, or BuildKit --mount=type=secret for build-time credentials that leave no trace.
8. Scan images for vulnerabilities in CI — run Trivy/Grype on every build, fail on high/critical CVEs, and rebuild regularly so patched base layers actually reach production.
9. Generate an SBOM and sign images — produce a software bill of materials (docker sbom, Syft) and sign with cosign so consumers can verify provenance (supply-chain security).
10. Define HEALTHCHECK and resource limits — health checks let orchestrators restart broken containers; --memory/--cpus (or K8s limits) prevent one workload from starving its neighbors.
11. Handle signals and use a proper init — trap SIGTERM for graceful shutdown and use --init/tini so PID 1 reaps zombies; otherwise deploys hang and processes leak.
12. Use the exec form of ENTRYPOINT/CMD — JSON-array form runs your binary directly as PID 1, so signals reach it; the shell form buries it under /bin/sh -c.
13. Log to stdout/stderr — never to files inside the container; let the platform (Docker logging driver, Fluent Bit) collect and ship logs.
14. Treat containers as immutable — never docker exec to patch production; rebuild the image and redeploy so the running state always matches a committed artifact.
15. Keep state in volumes, not the container — databases and uploads belong in named volumes or external services; the writable layer is ephemeral and slow.
16. Prefer small, purpose-built base images — distroless/alpine/scratch reduce CVEs and pull time; be mindful that Alpine uses musl libc, which can trip up glibc-dependent binaries.
17. Build multi-arch images when targeting mixed fleets — docker buildx produces amd64+arm64 manifests so the same tag runs on Graviton and Intel nodes alike.

References

• Docker documentation
• Dockerfile reference
• Dockerfile best practices (official)
• BuildKit
• Open Container Initiative specifications
• containerd
• runc
• Podman · Buildah
• Docker Compose reference
• Trivy image scanner · Grype
• Sigstore cosign (image signing)
• gVisor · Kata Containers
• Linux namespaces (man7) · cgroups (man7)
• roadmap.sh — DevOps: Containers