Scheduling and Resource ManagementScheduling and Resource Management

Part of the Kubernetes Roadmap knowledge base.

Overview

When you create a Pod, it does not run immediately — it is born in the Pending state with no node assigned. The kube-scheduler watches for these unscheduled Pods and decides, for each one, which node it should land on. That single decision has enormous downstream consequences: it determines whether the Pod gets the CPU and memory it needs, whether replicas survive a zone failure, whether latency-sensitive workloads sit next to noisy neighbors, and whether the cluster packs efficiently or wastes capacity. Scheduling is where declarative intent meets physical reality.

The scheduler is not a black box you must accept. Kubernetes gives you a rich vocabulary to steer placement: nodeSelector and node affinity to attract Pods to node types, pod affinity/anti-affinity to co-locate or separate Pods, taints and tolerations to reserve nodes, topology spread constraints to distribute across failure domains, and priority and preemption to decide who wins when capacity is scarce. Meanwhile, requests drive scheduling (what the scheduler reserves) and limits drive runtime enforcement (throttling and OOM-kills), and node-pressure eviction decides who gets removed when a node runs hot.

This page explains how the scheduler actually works (filtering then scoring), each placement mechanism with worked YAML, how requests/limits/QoS tie into eviction and OOM behavior, how DaemonSets bypass the normal flow, and how the descheduler fixes drift over time. The goal is to move from “the scheduler put my Pod somewhere” to “I told the scheduler exactly what my workload needs.”

A mental model

The scheduler answers one question per Pod: “Given every node, which are feasible, and among those, which is best?” Feasibility is a hard yes/no (filtering); best is a ranked score (scoring). Everything you configure is either a hard constraint that removes infeasible nodes (requiredDuringScheduling…, taints without tolerations, insufficient requests) or a soft preference that nudges the score (preferredDuringScheduling…, spread weights). Once you internalize “hard = filter, soft = score,” every affinity and constraint knob falls into place.

Fundamentals

The kube-scheduler: filtering and scoring

For each Pending Pod, the scheduler runs a two-phase cycle over the nodes, implemented as a pipeline of plugins (the scheduling framework):

1. Filtering (predicates) — eliminate nodes that cannot run the Pod. A node is filtered out if it lacks enough allocatable CPU/memory for the Pod’s requests, has a taint the Pod doesn’t tolerate, fails a nodeSelector/required node affinity, has no free ports the Pod needs, can’t attach the required volume (zone mismatch), etc. The survivors are feasible nodes. If none survive, the Pod stays Pending with a FailedScheduling event.
2. Scoring (priorities) — rank the feasible nodes 0–100 by weighted plugins: NodeResourcesFit (bin-packing vs. spreading), InterPodAffinity, PodTopologySpread, TaintToleration, ImageLocality (prefer nodes that already cached the image), NodeAffinity preferred terms, and more. Weights are summed; the highest-scoring node wins (ties broken randomly).

Then binding writes the chosen node into the Pod’s spec.nodeName, and the kubelet on that node takes over. Crucially, the scheduler only decides placement; the kubelet runs and enforces. Scheduling is based on requests, never on live usage — a node 90% actually utilized but with unreserved requests still looks empty to the scheduler.

Requests drive scheduling, limits drive enforcement

This division is the backbone of resource management:

• Requests are a reservation. The scheduler sums the requests of all Pods on a node and only places a new Pod if its requests still fit in allocatable capacity. Requests never change based on real usage. Set them too low and you overcommit the node (evictions, OOM); too high and you waste capacity and can’t schedule.
• Limits are a runtime ceiling enforced by the kubelet/kernel via cgroups. CPU over limit → throttled (the container is slowed, not killed — CPU is compressible). Memory over limit → OOM-killed and restarted (memory is incompressible). Limits play no part in scheduling.

A node can therefore be scheduled to 100% of requests while actually using far less, or be overcommitted (sum of limits > capacity) and risk OOM under load. Understanding which knob does what prevents both mystery Pending Pods (requests too high) and mystery OOMKilled restarts (limit too low or heap misconfigured).

Node-pressure eviction and OOM

Two different mechanisms remove workloads under memory pressure, and people conflate them:

• Container OOM-kill (kernel): a single container exceeds its memory limit → the kernel OOM-killer terminates that container → the kubelet restarts it (with CrashLoopBackoff if it repeats). This is per-container and driven by limits.
• Node-pressure eviction (kubelet): the node crosses an eviction threshold (low memory.available, nodefs, imagefs) → the kubelet proactively evicts whole Pods to reclaim resources, ranking victims by QoS class then by usage over requests. BestEffort go first, then Burstable most-over-request, Guaranteed last. Evicted Pods are deleted and rescheduled elsewhere.

The takeaway: requests + QoS determine who survives node pressure; limits determine who gets OOM-killed individually. Guaranteed Pods (requests == limits for CPU and memory) are the most protected.

Key Concepts

nodeSelector — the simplest steering

A hard requirement that a node carry specific labels. The Pod is filtered onto matching nodes only:

spec:
  nodeSelector:
    disktype: ssd
    kubernetes.io/arch: arm64

Simple and effective, but all-or-nothing (no soft preference, no OR logic). Node affinity is its more expressive successor.

Node affinity and anti-affinity

Node affinity attracts Pods to nodes by label, with hard and soft variants and richer operators (In, NotIn, Exists, Gt, Lt):

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:     # HARD — filter
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values: ["us-east-1a", "us-east-1b"]
    preferredDuringSchedulingIgnoredDuringExecution:    # SOFT — score
      - weight: 80
        preference:
          matchExpressions:
            - key: node.kubernetes.io/instance-type
              operator: In
              values: ["m6i.large"]

Note IgnoredDuringExecution: affinity is evaluated only at scheduling time. If a node’s labels later change so the Pod no longer matches, the running Pod is not evicted (there is no RequiredDuringExecution yet). “Node anti-affinity” is expressed with NotIn/DoesNotExist operators — there is no separate anti-affinity field for nodes.

Pod affinity and anti-affinity

These schedule Pods relative to other Pods, using a topologyKey that defines the domain of co-location (same node, same zone, …):

affinity:
  podAffinity:                    # attract: run near Pods with these labels
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels: { app: cache }
        topologyKey: kubernetes.io/hostname          # same node as the cache
  podAntiAffinity:                # repel: spread replicas apart
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels: { app: web }
          topologyKey: topology.kubernetes.io/zone   # one web Pod per zone, ideally

Pod affinity co-locates (a web tier next to its cache to cut latency). Pod anti-affinity separates (spread replicas across nodes/zones so one failure doesn’t take them all). The topologyKey is the crux — kubernetes.io/hostname means “per node,” topology.kubernetes.io/zone means “per zone.” Caveat: required pod affinity/anti-affinity is computationally expensive (the scheduler compares against all Pods) and can significantly slow scheduling in large clusters — prefer topology spread constraints for the common “spread my replicas” case.

Taints and tolerations

Taints and tolerations are the inverse of affinity: instead of a Pod choosing nodes, a node repels Pods unless they explicitly tolerate the taint. Used to reserve nodes for specific workloads.

# Taint a node: repel everything without a matching toleration
kubectl taint nodes gpu-node-1 dedicated=gpu:NoSchedule

# A Pod that tolerates it (and thus may be scheduled there):
tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"

The three effects:

Kubernetes uses taints internally: control-plane nodes carry node-role.kubernetes.io/control-plane:NoSchedule; the node controller adds node.kubernetes.io/not-ready:NoExecute and unreachable:NoExecute on failure, which is how Pods get evicted from a dead node. A toleration only permits scheduling; it does not attract — pair taints/tolerations with node affinity if you also want to force the Pod onto those nodes (otherwise a tolerating Pod may land on any untainted node).

Topology spread constraints

The modern, preferred way to distribute Pods evenly across failure domains — cleaner and cheaper than pod anti-affinity for the “spread my replicas” use case:

topologySpreadConstraints:
  - maxSkew: 1                                   # max imbalance between domains
    topologyKey: topology.kubernetes.io/zone     # the failure domain
    whenUnsatisfiable: DoNotSchedule             # or ScheduleAnyway (soft)
    labelSelector:
      matchLabels: { app: web }
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname          # also spread per node
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels: { app: web }

maxSkew is the maximum allowed difference in Pod count between any two topology domains. whenUnsatisfiable: DoNotSchedule makes it a hard constraint (Pod stays Pending rather than violate the skew); ScheduleAnyway makes it a soft scoring preference. Combine a zone-level and a node-level constraint to spread across both. Cluster-level defaults can be set in the scheduler config so every Pod spreads without per-workload YAML.

Pod priority and preemption

A PriorityClass assigns an integer priority to Pods. When a high-priority Pod is Pending and no node fits, the scheduler may preempt — evict lower-priority Pods to make room:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: { name: high-priority }
value: 1000000
globalDefault: false
preemptionPolicy: PreemptLowerPriority     # or Never (priority for ordering only)
description: "Critical user-facing services"
---
# In the Pod spec:
spec:
  priorityClassName: high-priority

Higher value = more important. When capacity is tight, a Pending high-priority Pod triggers preemption: the scheduler finds a node where evicting lower-priority Pods would let it fit, deletes those victims (respecting their graceful termination and, where possible, PodDisruptionBudgets), and schedules the important one. preemptionPolicy: Never lets a Pod jump the scheduling queue without evicting anyone. Kubernetes ships two built-in classes for critical add-ons: system-cluster-critical and system-node-critical. Pitfall: priority interacts with QoS at eviction time — under node pressure the kubelet considers priority alongside QoS, so a high-priority BestEffort Pod is less protected than you might assume; set requests too.

DaemonSet scheduling

A DaemonSet runs one Pod per (matching) node — log shippers, CNI agents, node exporters, CSI drivers. Historically the DaemonSet controller placed Pods directly; modern Kubernetes schedules DaemonSet Pods through the default scheduler (the NodeAffinity the controller injects pins each Pod to its target node). Key differences from normal scheduling:

• DaemonSet Pods respect nodeSelector/affinity to target a subset of nodes (e.g., only GPU nodes).
• They must tolerate node taints to land on tainted nodes — that’s why node-critical agents carry broad tolerations (including NoExecute) so they run on control-plane and not-ready nodes too.
• They typically set priorityClassName: system-node-critical so they aren’t preempted or evicted, since the node depends on them.

The descheduler

The scheduler makes a good decision at placement time, but clusters drift: nodes are added, Pods die and reschedule, utilization skews, affinity rules that couldn’t be honored at 3 a.m. now can. The scheduler never revisits a running Pod. The descheduler (a separately installed component) periodically evicts Pods that violate policies so they get rescheduled onto better nodes. Common strategies: LowNodeUtilization (rebalance off hot nodes), RemoveDuplicates (spread replicas of the same set off one node), RemovePodsViolatingInterPodAntiAffinity, RemovePodsViolatingTopologySpreadConstraint, and RemovePodsViolatingNodeTaints. It only evicts; the scheduler then re-places. Respect PodDisruptionBudgets and pair it with cluster autoscaling for real rebalancing.

Worked example: affinity + taints/tolerations + topology spread

A production web Deployment: keep it off spot/GPU nodes, spread across zones and nodes, and prefer to sit near its cache.

apiVersion: apps/v1
kind: Deployment
metadata: { name: web }
spec:
  replicas: 6
  selector: { matchLabels: { app: web } }
  template:
    metadata: { labels: { app: web } }
    spec:
      priorityClassName: high-priority
      tolerations:
        - key: "workload"                 # allowed onto nodes tainted workload=web
          operator: "Equal"
          value: "web"
          effect: "NoSchedule"
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node-type         # only general-purpose nodes
                    operator: In
                    values: ["general"]
        podAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 50
              podAffinityTerm:
                labelSelector: { matchLabels: { app: cache } }
                topologyKey: kubernetes.io/hostname   # prefer same node as cache
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule            # hard: balance across zones
          labelSelector: { matchLabels: { app: web } }
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway           # soft: also spread per node
          labelSelector: { matchLabels: { app: web } }
      containers:
        - name: web
          image: ghcr.io/example/web:1.4.2
          resources:
            requests: { cpu: "250m", memory: "256Mi" }   # drives scheduling
            limits:   { cpu: "1",    memory: "512Mi" }    # drives throttle/OOM

Reading it top to bottom: priorityClassName protects it under contention; the toleration lets it onto reserved nodes; node affinity restricts it to general-purpose nodes; pod affinity nudges it toward cache nodes; the two spread constraints balance it across zones (hard) and nodes (soft); and requests/limits set its scheduling footprint and QoS (Burstable here).

Best Practices

1. Set resource requests on every Pod, based on real usage — requests are the only signal the scheduler uses for placement; without them scheduling is blind, packing causes evictions, and the Pod drops to BestEffort. Right-size with VPA recommendations or metrics, not guesses.
2. Understand that requests schedule and limits enforce — diagnose Pending as “requests too high / node too full” and OOMKilled as “memory limit too low or heap misconfigured”; conflating them wastes hours.
3. Prefer topology spread constraints over pod anti-affinity for spreading — spread constraints are cheaper for the scheduler and express “balance my replicas across zones/nodes” directly; reserve required pod anti-affinity for true hard-separation needs.
4. Always spread replicas across failure domains — a service with all replicas on one node or zone dies with it; use topology.kubernetes.io/zone spread (plus per-node) so a single failure never takes the whole service down.
5. Use preferred (soft) affinity unless you truly need required — hard constraints leave Pods Pending when unsatisfiable; soft preferences degrade gracefully. Reserve requiredDuringScheduling for genuine hard dependencies (GPU, licensing, data locality).
6. Pair taints with node affinity when dedicating nodes — a taint only repels others; a toleration only permits, it doesn’t attract. To truly pin a workload onto reserved nodes, taint the nodes AND add matching node affinity, or the tolerating Pod may land elsewhere.
7. Reserve NoExecute taints for eviction intent — NoExecute evicts running Pods, not just blocks new ones; use it deliberately (node draining, dedicating a live node) and set tolerationSeconds on Pods that may briefly tolerate a not-ready node.
8. Use PriorityClasses sparingly and deliberately — too many priority tiers make preemption unpredictable; define a small set (critical, normal, batch), set globalDefault carefully, and remember system-critical add-ons already have their own classes.
9. Give critical DaemonSets system-node-critical priority and broad tolerations — node agents (CNI, CSI, logging) must run on every node including tainted and not-ready ones, and must never be preempted, or the node itself degrades.
10. Make latency-critical workloads Guaranteed QoS — setting requests == limits for CPU and memory gives the strongest eviction protection (evicted last) and, with the static CPU manager, exclusive cores; use it for the workloads you cannot afford to lose under pressure.
11. Avoid unbounded overcommit of memory limits — summing limits far above node capacity invites OOM-kills under load; keep memory requests realistic and limits close to them for important workloads, since memory is incompressible.
12. Combine PodDisruptionBudgets with priority, preemption, and the descheduler — a PDB stops preemption, node drains, and the descheduler from taking too many replicas down at once; without it, voluntary disruptions can drop a service to zero.
13. Run a descheduler to correct drift — the scheduler never revisits placed Pods; a descheduler rebalances off hot nodes and re-honors affinity/spread that couldn’t be satisfied earlier, especially after autoscaling events.
14. Watch for IgnoredDuringExecution surprises — affinity and nodeSelector are only checked at scheduling time; relabeling a node does not move already-running Pods, so don’t rely on label changes to reschedule — cordon/drain or use the descheduler instead.
15. Don’t rely on the default scheduler for special cases you can express declaratively — use affinity, taints, spread, and priority to encode intent rather than manual node assignment (nodeName), which bypasses the scheduler and its safety checks entirely.
16. Label nodes consistently with well-known keys — use topology.kubernetes.io/zone, kubernetes.io/hostname, node.kubernetes.io/instance-type, and kubernetes.io/arch so affinity, spread, and cost tooling all work off the same, portable taxonomy.

References

• Kubernetes Scheduler
• Scheduling Framework
• Assigning Pods to Nodes (nodeSelector & affinity)
• Taints and Tolerations
• Pod Topology Spread Constraints
• Pod Priority and Preemption
• Node-pressure Eviction
• Pod Quality of Service Classes
• DaemonSet
• Pod Disruption Budgets
• Descheduler for Kubernetes
• roadmap.sh — Kubernetes