Cluster AdministrationCluster Administration

Part of the Kubernetes Roadmap knowledge base.

Overview

Building an application on Kubernetes and operating the cluster underneath it are two very different jobs. Cluster administration is the day-2 discipline: bootstrapping a cluster, keeping the control plane highly available, protecting the state store (etcd), moving through the relentless upgrade cadence without downtime, draining nodes for maintenance, rotating certificates before they expire, and planning capacity so the scheduler never runs out of room. These are the tasks that decide whether a platform is reliable or a 3 a.m. pager magnet.

The central tension of cluster administration is that Kubernetes is a distributed system that stores all of its truth in etcd and serves it through a single API server. Everything an operator does — HA, backups, upgrades, cert rotation — ultimately protects those two things. If the API server is unreachable, you fly blind; if etcd is lost without a backup, the cluster is gone. Most administration best practices are, at bottom, ways of never being one failure away from either outcome.

A key decision shapes how much of this you actually own: managed vs self-managed. With a managed offering (EKS, GKE, AKS) the provider runs, patches, and backs up the control plane and etcd — you own worker nodes, upgrades of those nodes, and workloads. With a self-managed cluster (kubeadm, kOps, Cluster API, or bare kubelets) you own everything, including etcd backups and control-plane HA. This page covers the full self-managed surface because understanding it makes you a better operator even on managed platforms — you know what the provider is doing for you and where the seams are.

Fundamentals

The control plane, and why HA matters

The control plane is the set of components that make cluster-wide decisions: kube-apiserver (the only front door and the only thing that talks to etcd), etcd (the Raft-backed key-value store holding all cluster state), kube-scheduler (Pod placement), and kube-controller-manager (the built-in reconciliation loops). On a single-control-plane cluster, losing that one node means no new scheduling, no scaling, no self-healing, and no kubectl — existing Pods keep running, but the cluster is frozen and blind.

High availability removes that single point of failure by running the control plane in triplicate (or five) behind a load balancer:

• kube-apiserver is stateless and active-active — every replica can serve every request; the load balancer spreads traffic and drops unhealthy backends.
• kube-scheduler and kube-controller-manager run on every control-plane node but use leader election so only one instance acts at a time (avoiding two schedulers placing the same Pod twice).
• etcd must maintain quorum: a cluster of N members tolerates the loss of (N-1)/2. So 3 members tolerate 1 failure, 5 tolerate 2. Always run an odd number (3 or 5) — even counts add cost without adding fault tolerance and risk split-brain.

etcd: the crown jewels

etcd holds the entire desired and observed state of the cluster. It is a strongly consistent (linearizable) store using the Raft consensus algorithm. Two operational truths follow:

1. etcd is the single source of truth. Lose it with no backup and the cluster is unrecoverable — you rebuild from scratch and re-apply manifests (which is exactly why GitOps, storing manifests in Git, is a resilience strategy).
2. etcd is latency- and I/O-sensitive. It fsyncs every write; slow disks cause apply request took too long warnings, leader elections, and cascading control-plane instability. Give it fast SSD/NVMe, keep it on dedicated disks, and keep write latency (etcd_disk_wal_fsync_duration_seconds) low.

The scheduler and capacity

The scheduler places Pods onto nodes using resource requests (not limits or actual usage). A node’s allocatable capacity is its total capacity minus reservations for the kubelet, OS, and eviction thresholds. When the sum of requested resources leaves no node with room, Pods go Pending — a capacity signal, not a bug. Capacity planning is the art of keeping enough headroom that node failures and rollouts don’t cause Pending storms, without paying for idle machines.

Certificates everywhere

Kubernetes is a mesh of mutually authenticated TLS: apiserver↔etcd, apiserver↔kubelet, controller-manager/scheduler↔apiserver, and kubelet client certs. kubeadm generates a cluster CA and issues these certs with a 1-year default lifetime. They renew automatically on kubeadm upgrade, but a cluster left un-upgraded for a year will have its control plane certs expire — a classic, avoidable outage. The kubelet’s own serving/client certs can auto-rotate via the RotateKubeletServerCertificate/RotateKubeletClientCertificate features and the CSR API.

Key Concepts

Bootstrapping a cluster

Managed control planes (EKS/GKE/AKS) skip bootstrapping the control plane entirely — the provider runs and backs up apiserver + etcd. You still choose node groups, CNI (sometimes), and upgrade timing. Trade-off: less operational burden and a hardened, patched control plane vs less control (limited apiserver flags, provider-set version support windows, and cost per cluster).

A minimal kubeadm bootstrap of the first control-plane node for an HA cluster:

# On the first control-plane node — point at the load balancer VIP so
# additional control-plane nodes can join behind the same endpoint.
kubeadm init \
  --control-plane-endpoint "k8s-api.internal:6443" \
  --upload-certs \
  --pod-network-cidr "10.244.0.0/16" \
  --kubernetes-version v1.30.2

# kubeadm prints two join commands: one for control-plane nodes
# (includes --control-plane --certificate-key) and one for workers.
# Install a CNI (e.g. Cilium/Calico) before nodes report Ready:
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml

etcd backup and restore

This is the single most important administration skill for self-managed clusters. Take a consistent snapshot with etcdctl snapshot save, store it off-cluster, and — crucially — rehearse the restore.

# Back up etcd (run on a control-plane node, or anywhere with the certs).
# The endpoints/certs match a kubeadm-managed etcd.
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/backups/etcd-$(date +%Y%m%d-%H%M%S).db

# Verify the snapshot is intact (checks the hash and prints revision/size).
ETCDCTL_API=3 etcdctl snapshot status /var/backups/etcd-20260718.db -w table

Restore (disaster recovery) — restore the snapshot into a new data directory, then point etcd at it:

# 1. Stop the kubelet and etcd (kubeadm runs etcd as a static Pod;
#    move its manifest out of /etc/kubernetes/manifests to stop it).
# 2. Restore into a fresh data dir:
ETCDCTL_API=3 etcdctl snapshot restore /var/backups/etcd-20260718.db \
  --data-dir=/var/lib/etcd-restored

# 3. Point etcd's static Pod manifest at /var/lib/etcd-restored (hostPath),
#    restart the kubelet, and etcd comes back with the snapshot's state.

For multi-member etcd, restore on one member with the correct --initial-cluster / --initial-advertise-peer-urls, then let peers re-sync. Higher-level tools — Velero (with etcd snapshot or resource-level backup) or provider snapshots — automate this. The only backup that counts is one you have restored; schedule regular restore drills into a scratch cluster.

Cluster upgrades

Kubernetes ships a minor release roughly three times a year and supports the latest three minors (~14 months of patches each). Falling behind means unsupported versions and painful multi-hop upgrades — so upgrade continuously.

Version skew policy (the rules that make rolling upgrades safe):

The golden rules: upgrade the control plane before the nodes, never skip a minor version (1.29 → 1.30 → 1.31, not 1.29 → 1.31), and never run a kubelet newer than the apiserver.

The kubeadm upgrade flow (one minor version at a time):

# ---- Control-plane node(s), one at a time ----
# 1. Upgrade the kubeadm binary to the target version (via your package manager).
apt-get update && apt-get install -y kubeadm=1.30.2-*

# 2. Plan: kubeadm shows the component versions it will move to and any
#    manual steps / deprecated APIs to address.
kubeadm upgrade plan

# 3. Apply on the FIRST control-plane node (upgrades apiserver, controller-
#    manager, scheduler static Pods, renews control-plane certs, bumps etcd):
kubeadm upgrade apply v1.30.2

# 3b. On OTHER control-plane nodes, use the node variant instead:
#     kubeadm upgrade node

# 4. Now upgrade the kubelet + kubectl on this node, then restart kubelet:
apt-get install -y kubelet=1.30.2-* kubectl=1.30.2-*
systemctl daemon-reload && systemctl restart kubelet

# ---- Worker nodes, one at a time (see draining below) ----
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
#   ssh to node: install kubeadm 1.30.2, run `kubeadm upgrade node`,
#   install kubelet 1.30.2, restart kubelet
kubectl uncordon <node>

Before any upgrade, check for removed APIs with kubectl deprecation warnings or a tool like pluto/kubent, and test the whole flow in a staging cluster first.

Node management: cordon, drain, uncordon

Maintenance (kernel patch, kubelet upgrade, hardware work) requires safely evacuating a node’s workloads:

# Mark the node unschedulable — no NEW Pods land here, existing ones stay.
kubectl cordon node-7

# Evict the Pods gracefully, respecting PodDisruptionBudgets and grace periods.
#   --ignore-daemonsets: DaemonSet Pods can't be evicted (they belong to every node)
#   --delete-emptydir-data: acknowledge that emptyDir data will be lost
#   --grace-period / --timeout: bound how long to wait
kubectl drain node-7 --ignore-daemonsets --delete-emptydir-data --timeout=300s

# ... do the maintenance ...

# Return the node to the scheduler's pool.
kubectl uncordon node-7

• cordon = “stop scheduling here” (a scheduling gate). drain = cordon plus evict existing Pods. uncordon = re-enable scheduling.
• Drain respects PodDisruptionBudgets (PDBs) — if evicting a Pod would violate a workload’s minAvailable, the eviction blocks until it’s safe. This is why every important workload needs a PDB and multiple replicas, or drains hang or cause outages.
• Taints for maintenance: instead of cordoning, you can apply a NoSchedule/NoExecute taint (e.g. node.kubernetes.io/maintenance) — NoExecute also evicts running Pods that don’t tolerate it. Cloud autoscalers and cluster-autoscaler use taints for node lifecycle.

Certificate management and rotation

# See when every control-plane certificate expires (run on a control-plane node).
kubeadm certs check-expiration

# Manually renew all control-plane certs (also happens automatically on upgrade).
kubeadm certs renew all
systemctl restart kubelet   # and restart the static-Pod control-plane components

# The kubelet's own client cert auto-rotates via the CSR API; pending CSRs:
kubectl get csr
kubectl certificate approve <csr-name>   # if not auto-approved

Rotate the cluster CA only when compromised — it’s disruptive (every cert must be re-issued and every kubeconfig refreshed). For workloads that need certs, use cert-manager rather than hand-rolling PKI.

API server flags and feature gates

The apiserver is configured by flags (in its static-Pod manifest for kubeadm, at /etc/kubernetes/manifests/kube-apiserver.yaml). Common knobs: --audit-policy-file (audit logging), --encryption-provider-config (encrypt Secrets at rest with KMS), --enable-admission-plugins, and --feature-gates. Feature gates toggle features by maturity — Alpha (off by default, may break/change), Beta (increasingly on by default), GA (locked on). Managed platforms expose a limited, curated subset of these; self-managed clusters get the full surface but own the risk.

Namespaces and multi-tenancy basics

Namespaces partition a cluster for isolation, quotas, and RBAC scoping. Soft multi-tenancy (trusted teams sharing a cluster) is well supported; hard multi-tenancy (untrusted tenants) needs stronger isolation and is often better served by separate clusters or virtual-cluster tools (vCluster).

apiVersion: v1
kind: ResourceQuota
metadata: { name: team-a-quota, namespace: team-a }
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi
    pods: "100"
---
apiVersion: v1
kind: LimitRange          # default requests/limits for Pods with none set
metadata: { name: defaults, namespace: team-a }
spec:
  limits:
    - type: Container
      default: { cpu: 500m, memory: 512Mi }
      defaultRequest: { cpu: 100m, memory: 128Mi }

Pair per-namespace ResourceQuota (caps total consumption), LimitRange (sane per-Pod defaults), RBAC (who can do what, where), NetworkPolicy (default-deny cross-namespace traffic), and Pod Security Admission (baseline/restricted) for a workable soft-multi-tenant setup.

Best Practices

1. Run an HA control plane with 3 (or 5) etcd members — an odd number for quorum, behind a load-balanced API endpoint. A single control-plane node is fine for learning, never for production; one machine failure should never freeze the cluster.
2. Back up etcd on a schedule and rehearse the restore — a snapshot you’ve never restored is a guess, not a backup. Store snapshots off-cluster and run periodic restore drills into a scratch cluster so disaster recovery is muscle memory, not improvisation.
3. Give etcd fast, dedicated, low-latency storage — etcd fsyncs every write; slow disks cause leader elections and control-plane instability. Monitor etcd_disk_wal_fsync_duration_seconds and disk saturation, and keep etcd off shared/noisy volumes.
4. Upgrade continuously, one minor at a time — Kubernetes supports only the latest three minors and never skips versions. Staying current avoids painful multi-hop upgrades and unsupported clusters; test each upgrade in staging first.
5. Respect the version skew policy — upgrade the control plane before the nodes, keep kubelets no more than the allowed minors behind apiserver, and never run a kubelet newer than the apiserver.
6. Scan for removed APIs before every upgrade — use pluto/kubent and heed kubectl deprecation warnings; a manifest using a removed API version will silently fail to apply after the upgrade.
7. Always drain before node maintenance, and uncordon after — never yank a node from under running Pods. Draining evicts gracefully and honors grace periods and PodDisruptionBudgets.
8. Give every important workload a PodDisruptionBudget and ≥2 replicas — without a PDB, a drain can take a service to zero; with one but a single replica, drains hang. Both are required for safe voluntary disruptions.
9. Monitor certificate expiry and let upgrades renew certs — run kubeadm certs check-expiration (or alert on it); a cluster idle for a year expires its control-plane certs and goes down. Enable kubelet cert auto-rotation.
10. Encrypt Secrets at rest with a KMS provider — etcd stores Secrets base64-encoded, i.e. effectively in the clear. Configure --encryption-provider-config with an external KMS and restrict who can read Secrets via RBAC.
11. Plan capacity with headroom for failures and rollouts — size so that losing a node (or a zone) doesn’t cause Pending storms, and reserve room for surge during rolling updates. Right-size requests from real usage data rather than guessing.
12. Reserve resources for the kubelet and OS — set --kube-reserved/--system-reserved and eviction thresholds so node-level daemons never get starved by Pods, which otherwise triggers node instability and cascading evictions.
13. Isolate tenants with namespaces + ResourceQuota + LimitRange + RBAC + NetworkPolicy — soft multi-tenancy needs all five; for untrusted tenants prefer separate clusters or virtual clusters over trying to make one cluster perfectly airtight.
14. Keep all manifests in Git (GitOps) — the fastest disaster recovery after losing etcd is re-applying a Git-tracked source of truth. Declarative, version-controlled cluster config also makes audits and rollbacks trivial.
15. Enable audit logging and centralize control-plane logs/metrics — you cannot administer what you cannot see. Ship apiserver audit logs and control-plane metrics off-cluster so a control-plane outage doesn’t also take your forensics with it.
16. Prefer managed control planes unless you have a reason not to — EKS/GKE/AKS remove the hardest, riskiest operational work (control-plane HA, etcd backups, patching). Self-manage only when you need capabilities managed platforms don’t offer, and know you’re taking on etcd’s care and feeding.

References

• Kubernetes documentation
• Administer a Cluster (tasks)
• Creating a cluster with kubeadm
• Options for Highly Available topology
• Operating etcd clusters for Kubernetes
• Backing up an etcd cluster (etcd docs)
• Upgrading kubeadm clusters
• Version skew policy
• Safely Drain a Node
• Taints and Tolerations
• Certificate Management with kubeadm
• Feature Gates
• Encrypting Secret Data at Rest
• Resource Quotas · Limit Ranges
• Velero · Cluster API · kind · minikube
• Amazon EKS · Google GKE · Azure AKS
• roadmap.sh — Kubernetes