StorageStorage

Part of the Kubernetes Roadmap knowledge base.

Overview

Containers are ephemeral by design: when a Pod dies, its writable container layer dies with it. That is fine for stateless services, but real systems need to keep data — databases, message queues, uploaded files, caches, build artifacts. Kubernetes storage is the set of abstractions that let Pods consume storage that outlives them, that can be attached to whichever node a Pod lands on, and that is provisioned on demand from cloud disks, network filesystems, or local media. The system is designed so application authors ask for storage by requirement (“I need 20Gi of fast, read-write-once storage”) without knowing which backend fulfills it.

The heart of the design is a claim-and-provision separation of concerns. A workload declares what it needs with a PersistentVolumeClaim (PVC); a StorageClass describes how to make it, and a CSI driver does the actual provisioning of a PersistentVolume (PV) — a cloud disk, an NFS export, whatever. This decoupling is what lets the same manifest run unchanged on AWS, GKE, on-prem, or a laptop: the workload asks for a class of storage, and the cluster’s configured drivers satisfy it. Understanding the PVC → StorageClass → PV → CSI chain, plus access modes and reclaim policies, is what separates “my data survived a node failure” from “my database is gone.”

A mental model

Storage in Kubernetes is a request-fulfillment marketplace. The PVC is a purchase order (“20Gi, ReadWriteOnce, class=fast”) written by the app team. The StorageClass is a catalog entry (“fast” = SSD provisioned by the AWS EBS CSI driver, with these parameters). The PV is the delivered goods — an actual volume bound to exactly one PVC. The CSI driver is the supplier that manufactures and attaches it. The app never names a specific disk; it names a requirement, and the cluster fulfills it. This indirection is the whole point: portable, self-service, dynamically provisioned storage. Static provisioning (an admin pre-creating PVs by hand) still exists, but dynamic provisioning through StorageClasses is the norm.

Fundamentals

Ephemeral vs persistent storage

The first decision is lifetime. Kubernetes offers several tiers, from most to least ephemeral:

The rule of thumb: if losing the data when the Pod is deleted is acceptable, use an ephemeral volume; if not, you need a PersistentVolume. Do not attempt to persist real data with hostPath (it ties data to one node and is a security risk) — use a PVC.

Volumes: the in-Pod abstraction

A Volume is a directory, backed by some medium, mounted into one or more of a Pod’s containers at a chosen mountPath. Volumes are declared in the Pod spec.volumes and mounted via containers[].volumeMounts. Common volume types:

• emptyDir — an empty directory created when the Pod is assigned to a node and deleted when the Pod is removed. Shared by all containers in the Pod; ideal for scratch space or passing data between an init container and the main container. Can be backed by disk or RAM (medium: Memory).
• configMap / secret — mount configuration or sensitive data as files. Updates to the source object propagate to the mounted files (eventually), unlike env vars which are frozen at start.
• hostPath — mounts a file or directory from the node’s filesystem into the Pod. Dangerous (ties the Pod to one node, can escape isolation); legitimate only for node-level system agents (log collectors, CSI/CNI plugins) in DaemonSets — never for app data.
• persistentVolumeClaim — mounts a PV that was claimed by a PVC. This is the durable, portable way to attach real storage, covered below.
• projected — combines several sources (Secrets, ConfigMaps, the ServiceAccount token, downward API) into one directory.

apiVersion: v1
kind: Pod
metadata: { name: demo }
spec:
  containers:
    - name: app
      image: nginx
      volumeMounts:
        - { name: cache, mountPath: /var/cache }      # ephemeral scratch
        - { name: config, mountPath: /etc/app, readOnly: true }
        - { name: data, mountPath: /var/lib/app }      # durable
  volumes:
    - name: cache
      emptyDir: {}
    - name: config
      configMap: { name: app-config }
    - name: data
      persistentVolumeClaim: { claimName: app-data }   # → PV → real disk

The PV / PVC / StorageClass triangle

Three objects cooperate to deliver durable storage:

• PersistentVolume (PV) — a cluster-scoped piece of storage (a cloud disk, NFS share, etc.). It has a capacity, access modes, a reclaim policy, and a reference to the backing driver. A PV is a resource, like a Node.
• PersistentVolumeClaim (PVC) — a namespaced request for storage: “I want XGi with these access modes and this StorageClass.” Pods reference PVCs, not PVs.
• StorageClass — a template describing a kind of storage: which provisioner (CSI driver), what parameters (disk type, IOPS, filesystem, replication), reclaim policy, and binding mode. It’s what makes dynamic provisioning possible.

The binding lifecycle: a PVC is created → the control plane finds or (dynamically) provisions a matching PV → they bind one-to-one → a Pod mounting the PVC gets the PV attached to its node. A bound PVC/PV pair is exclusive; the PV cannot be bound to another claim.

Static vs dynamic provisioning

• Static: an administrator pre-creates PVs by hand (e.g., pointing at existing NFS exports). A PVC then binds to a pre-existing PV whose size/access modes/class satisfy it. Tedious and rarely used for cloud disks.
• Dynamic (the norm): the PVC names a StorageClass; when no matching PV exists, the class’s provisioner creates one on demand. No admin action, no pre-provisioning. A cluster typically has a default StorageClass (marked with the storageclass.kubernetes.io/is-default-class: "true" annotation) used when a PVC omits storageClassName.

Key Concepts

Access modes

Access modes describe how many nodes can mount a volume and in what mode. They are a property of both the PV and the PVC request, and are enforced at attach time:

The crucial nuance: RWO is per-node, not per-Pod. Most block storage (AWS EBS, GCE PD, Azure Disk) is RWO — it can attach to only one node at a time, so a Deployment with 2 replicas landing on different nodes will leave one Pod stuck ContainerCreating waiting for a volume it can’t attach. For genuine multi-node shared read-write you need RWX, which requires a file/network backend (NFS, CephFS, AWS EFS, Azure Files, GlusterFS) — block volumes cannot do RWX. Choosing a workload topology without matching the access mode to the backend is the single most common storage mistake.

Volume modes: Filesystem vs Block

A PV’s volumeMode is either Filesystem (default) — the volume is formatted with a filesystem and mounted as a directory — or Block — the raw block device is exposed to the container at a device path (/dev/xvda) with no filesystem. Raw block is for databases and storage software that manage their own on-disk format and want to skip the filesystem layer for performance.

Reclaim policies

When a PVC is deleted, the PV’s persistentVolumeReclaimPolicy decides the fate of the underlying data:

• Delete (default for dynamic provisioning) — the PV and the backing cloud volume are deleted. Convenient, but a deleted PVC destroys data irrecoverably. Dangerous for anything valuable.
• Retain — the PV and its data are kept after the PVC is deleted; the PV moves to Released and an admin must manually reclaim or clean it. This is the safe choice for important data.
• Recycle — deprecated; ignore it.

The practical guidance: production StorageClasses for stateful data should use reclaimPolicy: Retain (or take snapshots), so an accidental kubectl delete pvc doesn’t wipe your database. The default Delete is a footgun for real data.

Worked example: StorageClass + PVC

# StorageClass: SSD on AWS, safe reclaim, expandable, bind late
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: { name: fast }
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "5000"
  fsType: ext4
reclaimPolicy: Retain               # keep data if the PVC is deleted
allowVolumeExpansion: true          # PVCs of this class can be grown
volumeBindingMode: WaitForFirstConsumer   # provision in the Pod's zone/node
---
# PVC: request 20Gi, RWO, from the "fast" class
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: app-data, namespace: prod }
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: fast
  resources:
    requests: { storage: 20Gi }

volumeBindingMode: WaitForFirstConsumer is important and underused: with the default Immediate, a PV is provisioned as soon as the PVC is created — possibly in a zone where the Pod can never be scheduled (a classic cross-AZ failure on cloud block storage). WaitForFirstConsumer delays provisioning until a Pod actually consumes the PVC, so the volume is created in the right zone/node. Use it for zonal block storage.

The Container Storage Interface (CSI)

CSI is the standard plugin interface between Kubernetes and storage systems. Historically, storage drivers (“in-tree volume plugins”) were compiled into Kubernetes itself; every new backend meant changing core code. CSI extracted all of that into out-of-tree drivers — independent components (typically a controller Deployment + a node DaemonSet) that implement provisioning, attach/detach, mount/unmount, snapshots, and expansion. Kubernetes has removed the old in-tree plugins in favor of CSI.

Practically: to use AWS EBS you install the aws-ebs-csi-driver; for GCE PD the pd.csi.storage.gke.io driver; for on-prem, ceph-csi, nfs-csi, democratic-csi, etc. The CSI driver registers itself, the StorageClass’s provisioner names it, and it also unlocks advanced features — volume snapshots, online volume expansion, topology awareness, and cloning — that the driver implements. When “PVCs are stuck Pending,” a missing or unhealthy CSI driver is a prime suspect.

StatefulSets and volumeClaimTemplates

A Deployment shares one PVC across all replicas (or, more often, can’t — because RWO block storage won’t attach to multiple nodes). Stateful workloads need one dedicated, stable volume per replica. That is exactly what a StatefulSet’s volumeClaimTemplates provides: for each Pod, the StatefulSet controller creates a separate PVC (named <template>-<statefulset>-<ordinal>, e.g., data-postgres-0), and that PVC follows the Pod across rescheduling.

apiVersion: apps/v1
kind: StatefulSet
metadata: { name: postgres }
spec:
  serviceName: postgres            # headless Service for stable Pod DNS
  replicas: 3
  selector: { matchLabels: { app: postgres } }
  template:
    metadata: { labels: { app: postgres } }
    spec:
      containers:
        - name: postgres
          image: postgres:16
          ports: [{ containerPort: 5432 }]
          volumeMounts:
            - { name: data, mountPath: /var/lib/postgresql/data }
  volumeClaimTemplates:            # one PVC PER replica, created automatically
    - metadata: { name: data }
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: fast
        resources: { requests: { storage: 50Gi } }

This gives each replica (postgres-0, postgres-1, postgres-2) its own 50Gi PVC that persists across restarts and reschedules, and — paired with a headless Service — a stable DNS name. Note a deliberate safety feature: deleting a StatefulSet does not delete its PVCs, so scaling down or removing the set won’t destroy data unless you delete the PVCs explicitly.

Volume snapshots

CSI drivers that support it let you take point-in-time snapshots of a PVC and restore new PVCs from them — the basis of backups and clone-for-test workflows. It uses three objects: a VolumeSnapshotClass (like a StorageClass, for snapshots), a VolumeSnapshot (the request to snapshot a PVC), and a VolumeSnapshotContent (the actual snapshot resource).

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata: { name: pg-snap-2026-07-19, namespace: prod }
spec:
  volumeSnapshotClassName: csi-snapclass
  source:
    persistentVolumeClaimName: data-postgres-0
---
# Restore: a new PVC whose dataSource is the snapshot
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: data-postgres-restore, namespace: prod }
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: fast
  resources: { requests: { storage: 50Gi } }
  dataSource:
    name: pg-snap-2026-07-19
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

Snapshots are crash-consistent, not application-consistent — for databases, quiesce/flush before snapshotting (or use the DB’s own backup tooling). They are a building block, not a complete backup solution; tools like Velero orchestrate snapshotting plus off-cluster storage.

Expanding volumes

Running out of space no longer means recreating the volume. If the StorageClass sets allowVolumeExpansion: true and the CSI driver supports it, you grow a PVC by editing its spec.resources.requests.storage to a larger value. The CSI driver expands the backing volume and (for filesystem mode) the filesystem, often online with no downtime. You can only grow, never shrink. If expansion needs a Pod restart the PVC shows a FileSystemResizePending condition.

kubectl patch pvc app-data -n prod -p '{"spec":{"resources":{"requests":{"storage":"40Gi"}}}}'
kubectl get pvc app-data -n prod -w    # watch capacity update

Best Practices

1. Use PVCs for anything that must survive a Pod — never hostPath or the container filesystem. hostPath pins data to one node and is a security hole; the container layer is destroyed on restart. Durable data belongs on a PersistentVolume.
2. Set reclaimPolicy: Retain (or snapshot regularly) for valuable data. The dynamic-provisioning default Delete destroys the backing volume when the PVC is deleted; one accidental kubectl delete pvc can wipe a database. Retain keeps the data for manual recovery.
3. Match access mode to backend and topology. RWO block storage attaches to one node only — multi-node read-write requires RWX on a file backend (EFS/NFS/CephFS). Requesting RWX from a block-only class silently strands Pods.
4. Prefer WaitForFirstConsumer binding for zonal block storage. Immediate binding can provision a volume in a zone the Pod can’t reach; delaying until scheduling ensures the volume lands where the Pod does.
5. Use StatefulSets with volumeClaimTemplates for stateful apps. Each replica gets its own stable PVC that follows it across reschedules — the correct primitive for databases and clustered stores, unlike sharing one PVC across a Deployment.
6. Set allowVolumeExpansion: true on production StorageClasses. It lets you grow volumes online without recreation; retrofitting expandability onto an existing class after you’re out of space is far more painful.
7. Keep secrets out of regular volumes and enable encryption at rest. Mount credentials via Secret volumes (or an external secrets operator), and turn on encryption for both etcd Secrets and the storage backend (CSI encryption / cloud KMS).
8. Snapshot before risky operations and test restores. Take VolumeSnapshots before upgrades/migrations; a backup you have never restored is not a backup. Automate with Velero and rehearse recovery.
9. Right-size requests and set ResourceQuotas on storage. Over-requesting wastes money on cloud disks; use requests.storage quotas per namespace to cap consumption and prevent runaway provisioning.
10. Choose the StorageClass consciously; define an explicit default. Different classes (SSD vs HDD, replicated vs single, IOPS tiers) have very different cost/performance. Don’t rely on whichever default happens to exist — set one intentionally and pick per workload.
11. Understand volume vs data lifecycle for StatefulSets. Deleting a StatefulSet leaves its PVCs behind by design; scale-down PVCs are also retained. Clean them up deliberately, and never assume kubectl delete statefulset frees storage.
12. Ensure the CSI driver is installed, healthy, and version-matched. Pending PVCs and stuck attaches usually trace to a missing/broken CSI controller or node driver; monitor it like any critical system component and keep it compatible with your Kubernetes version.
13. Use emptyDir (with size limits) for scratch, not persistent needs. It’s fast and simple for caches and inter-container sharing, but it dies with the Pod; set sizeLimit so a runaway process can’t fill the node’s disk.
14. Monitor volume capacity and set alerts. A full PVC can crash or corrupt a database with no automatic remediation. Track PV usage (via kubelet_volume_stats_* metrics) and alert before volumes fill.
15. For databases, prefer local NVMe or high-IOPS classes and app-consistent backups. Network block storage adds latency; pick an IOPS/throughput tier that fits, and rely on the database’s own consistent backup rather than raw crash-consistent snapshots alone.

References

• Kubernetes — Volumes
• Kubernetes — Persistent Volumes (PV, PVC, reclaim, modes)
• Kubernetes — Storage Classes
• Kubernetes — Dynamic Volume Provisioning
• Kubernetes — Volume Snapshots
• Kubernetes — Expanding Persistent Volumes
• Kubernetes — Ephemeral Volumes
• Kubernetes — StatefulSets
• Kubernetes CSI documentation
• AWS EBS CSI driver · GCP PD CSI · Ceph CSI
• Velero — backup and restore
• roadmap.sh — Kubernetes