Configuration and SecretsConfiguration and Secrets

Part of the Kubernetes Roadmap knowledge base.

Overview

An application is more than its image. The same container needs different database URLs, feature flags, log levels, API keys, and resource envelopes depending on where it runs. The twelve-factor principle is to keep configuration out of the image and inject it at runtime — so one immutable artifact promotes cleanly from dev to staging to prod, differing only by the config attached to it. Kubernetes provides two first-class objects for this: ConfigMap for non-sensitive configuration and Secret for sensitive data, both injectable as environment variables or mounted files.

Alongside injection, Kubernetes governs how much of the node a workload may consume through resource requests and limits. Requests are a reservation the scheduler honors; limits are a ceiling the kubelet and kernel enforce. Together with QoS classes, LimitRange, and ResourceQuota, they decide where a Pod lands, whether it gets throttled, and who gets evicted first when a node runs hot. Configuration and resource management are two halves of the same question: what does this workload need to run correctly and safely?

This page covers both: how to model configuration and secrets, why Secrets are not encryption by default, how to turn on real encryption at rest, how the downward API exposes Pod metadata to the app, and how requests/limits/QoS/quotas shape scheduling and eviction. The recurring theme is that these objects are cheap to create and easy to misuse — the details (immutability, base64 vs. encryption, env-vs-volume update semantics, QoS tiers) are where production incidents hide.

A mental model

Think of a Pod as being assembled from three inputs at admission time: the image (the code), the config (ConfigMaps, Secrets, downward API — the environment), and the resource contract (requests and limits — the budget). Change the image and you change behavior; change the config and you change environment; change the contract and you change where it runs and how it dies under pressure. Keeping these three concerns separate — never baking config or secrets into images, never leaving the resource contract blank — is what makes Kubernetes workloads portable, reproducible, and predictable.

Fundamentals

ConfigMap vs. Secret at a glance

The API surface is nearly identical — a Secret is essentially “a ConfigMap the platform treats with more care.” The care is what matters: RBAC scoping, encryption at rest, and not printing them into logs.

ConfigMap as environment variables

Two injection styles. Individual keys with valueFrom, or the whole ConfigMap with envFrom:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  LOG_LEVEL: "info"
  FEATURE_X_ENABLED: "true"
  DATABASE_HOST: "postgres.default.svc.cluster.local"
---
# In the Pod spec:
containers:
  - name: app
    image: ghcr.io/example/app:1.4.2
    env:
      - name: LOG_LEVEL                 # rename/select individual keys
        valueFrom:
          configMapKeyRef: { name: app-config, key: LOG_LEVEL }
    envFrom:
      - configMapRef: { name: app-config }   # inject every key as-is
        prefix: APP_                          # optional: APP_LOG_LEVEL, ...

Use valueFrom when you want to rename a key or pull one value; use envFrom to inject the whole map. Critical semantics: environment variables are resolved once, when the container starts. Editing the ConfigMap afterward does not change the running container’s env — you must trigger a new Pod (rollout) for it to take effect. This surprises people constantly.

ConfigMap as a mounted volume

Mount the ConfigMap as files — ideal for config files (nginx.conf, application.yaml) an app reads from disk:

volumes:
  - name: config
    configMap:
      name: app-config
      items:                    # optional: project specific keys to specific paths
        - key: nginx.conf
          path: nginx.conf
containers:
  - name: app
    volumeMounts:
      - name: config
        mountPath: /etc/app
        readOnly: true

Each key becomes a file under mountPath whose contents are the value. Volume-mounted ConfigMaps DO update in place — the kubelet syncs changes (with a periodic delay, up to ~1 minute plus cache TTL). But the file changing is not the app noticing: your process must watch the file or support a reload signal. subPath mounts (mounting a single key into an existing directory) are the exception — they do not receive updates, a common gotcha.

Immutable ConfigMaps and Secrets

Setting immutable: true freezes the object: it can no longer be updated, only deleted and recreated. This has two benefits — it protects against accidental changes that would ripple through every consuming Pod, and it improves cluster performance at scale because the kubelet stops watching immutable objects for changes, reducing apiserver load. Combine immutability with content-hashed names (app-config-a1b2c3, as Kustomize’s configMapGenerator does) so a config change produces a new object and a natural rollout.

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config-a1b2c3
immutable: true
data:
  LOG_LEVEL: "info"

Secret types

Secrets carry a type that hints at their contents and, for built-in types, enforces required keys:

Base64 is encoding, NOT encryption

This is the single most misunderstood fact about Secrets. The data field is base64-encoded, not encrypted — anyone who can kubectl get secret -o yaml or read etcd can trivially decode it:

kubectl create secret generic db --from-literal=password='s3cr3t'
kubectl get secret db -o jsonpath='{.data.password}' | base64 -d   # -> s3cr3t

Base64 exists only so binary data survives YAML/JSON transport. It provides zero confidentiality. Real protection comes from three orthogonal controls: (1) RBAC so few subjects can read Secrets, (2) encryption at rest so etcd on disk is ciphertext, and (3) not exposing Secrets in logs, env dumps, or Git. (Use stringData when authoring to write plaintext that the API server base64-encodes for you.)

Encryption at rest via EncryptionConfiguration

By default, Secrets sit in etcd base64-only. Turn on encryption at rest by passing an EncryptionConfiguration to the API server’s --encryption-provider-config flag:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets", "configmaps"]
    providers:
      - kms:                 # BEST: envelope encryption via external KMS (KMS v2)
          apiVersion: v2
          name: myKmsPlugin
          endpoint: unix:///var/run/kmsplugin/socket.sock
      - aescbc:              # or a local AES key (key material lives on the control plane)
          keys:
            - name: key1
              secret: <base64-32-byte-key>
      - identity: {}         # fallback: no encryption (needed for reads during migration)

Provider order matters: the first provider encrypts new writes; all listed providers are tried for decryption. The identity provider means plaintext — keep it last (or absent) once migrated. Because encryption happens on write, existing Secrets stay in their old form until rewritten: after enabling, run kubectl get secrets -A -o json | kubectl replace -f - to force re-encryption. Prefer the KMS v2 provider (envelope encryption backed by a cloud KMS or HSM) over static local keys — it keeps root key material off the control plane and supports rotation.

env vs. envFrom

env also lets you reference other env vars with $(VAR) interpolation and pull from the downward API — capabilities envFrom lacks.

The downward API

The downward API exposes Pod and container metadata to the containers themselves, without hardcoding or querying the API server. Two mechanisms: fieldRef (Pod fields like name, namespace, IP, node) and resourceFieldRef (the container’s own requests/limits).

env:
  - name: POD_NAME
    valueFrom: { fieldRef: { fieldPath: metadata.name } }
  - name: POD_NAMESPACE
    valueFrom: { fieldRef: { fieldPath: metadata.namespace } }
  - name: NODE_NAME
    valueFrom: { fieldRef: { fieldPath: spec.nodeName } }
  - name: POD_IP
    valueFrom: { fieldRef: { fieldPath: status.podIP } }
  - name: MEMORY_LIMIT             # feed the app its own limit (JVM/Go heap sizing)
    valueFrom:
      resourceFieldRef: { containerName: app, resource: limits.memory, divisor: 1Mi }
volumes:
  - name: podinfo                  # or expose labels/annotations as files
    downwardAPI:
      items:
        - path: labels
          fieldRef: { fieldPath: metadata.labels }

Common uses: telling a language runtime its real memory limit (so a JVM/Go program sizes its heap correctly instead of seeing the whole node), injecting POD_NAME/NODE_NAME into logs and traces, and per-Pod identity for clustering. Labels and annotations are only available via the volume form, not fieldRef env vars.

Key Concepts

Resource requests and limits

Every container can declare requests (guaranteed minimum) and limits (hard ceiling) for CPU and memory:

resources:
  requests:
    cpu: "250m"        # 0.25 of a vCPU core, guaranteed & used for scheduling
    memory: "256Mi"    # 256 mebibytes, reserved
  limits:
    cpu: "1"           # throttled above 1 core
    memory: "512Mi"    # OOM-killed above 512Mi

CPU units: 1 = one vCPU/core; 500m = 500 millicores = half a core. CPU is compressible — exceeding the limit throttles the container (it slows down) but does not kill it. Memory units: Mi/Gi are binary (1 Mi = 1024²), M/G are decimal (1 M = 10⁶) — mixing them is a classic sizing bug. Memory is incompressible — exceeding the limit gets the container OOM-killed by the kernel and restarted.

The two knobs do different jobs: requests drive scheduling (the scheduler sums requests, not limits or actual usage, to decide if a Pod fits a node), while limits drive runtime enforcement (throttling for CPU, OOM-kill for memory). A Pod can request little and burst up to its limit, but it is only guaranteed its requests.

QoS classes

Kubernetes derives a Quality of Service class from how requests and limits are set. It cannot be set directly; it determines eviction priority when a node is under memory pressure.

Guaranteed Pods are the safest and get exclusive CPU (with the static CPU manager policy) — reserve it for latency-critical, well-characterized workloads. BestEffort is for truly disposable batch work. Most real workloads are Burstable: set requests to the steady-state need and limits to a safe burst ceiling. Under memory pressure, the kubelet ranks eviction candidates by QoS and then by usage relative to requests — a Burstable Pod far over its request is evicted before one near it.

LimitRange

A LimitRange sets per-namespace defaults and bounds for containers/Pods, so Pods that omit resources still get sane values and nobody can request absurd amounts:

apiVersion: v1
kind: LimitRange
metadata: { name: defaults, namespace: team-a }
spec:
  limits:
    - type: Container
      default:              # applied as limits if the container omits them
        cpu: "500m"
        memory: "256Mi"
      defaultRequest:       # applied as requests if omitted
        cpu: "100m"
        memory: "128Mi"
      max: { cpu: "2", memory: "2Gi" }     # reject anything above
      min: { cpu: "50m", memory: "64Mi" }  # reject anything below

This is how you eliminate BestEffort Pods in a namespace: a defaultRequest guarantees every container gets requests. LimitRange operates at admission — violating Pods are rejected outright.

ResourceQuota

A ResourceQuota caps aggregate consumption per namespace — total CPU/memory requested, object counts, storage. It is the primary multi-tenancy guardrail:

apiVersion: v1
kind: ResourceQuota
metadata: { name: team-a-quota, namespace: team-a }
spec:
  hard:
    requests.cpu: "10"
    requests.memory: "20Gi"
    limits.cpu: "20"
    limits.memory: "40Gi"
    pods: "50"
    persistentvolumeclaims: "10"
    count/secrets: "25"

A subtle but important rule: once a ResourceQuota on CPU/memory exists in a namespace, every Pod must specify the corresponding requests/limits or it is rejected. This is why quotas and LimitRanges are usually deployed together — the LimitRange supplies defaults so developers aren’t forced to annotate every Pod.

External secrets: a pointer

Native Secrets solve storage and injection but not lifecycle: rotation, central auditing, and keeping plaintext out of Git for GitOps. Three common approaches sit on top:

• External Secrets Operator (ESO) — a controller that syncs from an external store (AWS Secrets Manager, Vault, GCP Secret Manager, Azure Key Vault) into native Kubernetes Secrets via an ExternalSecret CRD. The source of truth stays external; ESO reconciles and rotates.
• Sealed Secrets (Bitnami) — encrypt a Secret client-side with the controller’s public key into a SealedSecret CRD that is safe to commit to Git. Only the in-cluster controller can decrypt it. Great for GitOps where you want the secret in the repo but encrypted.
• SOPS (Mozilla) + age/KMS — encrypt YAML values in place; decrypt at apply time (often via Flux’s kustomize-controller SOPS integration or Helm plugins). Fine-grained, works with any KMS or age key.

Rule of thumb: ESO when a real secret manager is the source of truth; Sealed Secrets or SOPS when Git must be the source of truth but must never hold plaintext.

Configuration reloading

Because env-injected config is frozen at Pod start and volume-mounted config updates the file but not the process, changing config needs a strategy to take effect:

1. Roll the Deployment — the cleanest, most auditable approach. Change the config, then trigger a rollout so new Pods pick it up. Reloader (Stakater) automates this: annotate a Deployment and it restarts Pods when a referenced ConfigMap/Secret changes.
2. Content-hash the name — name ConfigMaps/Secrets by a hash of their contents (Kustomize’s configMapGenerator does this). A content change yields a new object name, which changes the Pod template, which triggers a rollout automatically. Pairs perfectly with immutable: true.
3. In-app reload — the app watches the mounted file (inotify) or accepts a SIGHUP/reload endpoint (nginx, Prometheus). No restart, but the app must support it and subPath mounts won’t update.

Worked example: Deployment consuming ConfigMap + Secret + resources

apiVersion: v1
kind: ConfigMap
metadata: { name: web-config }
data:
  LOG_LEVEL: "info"
  UPSTREAM_URL: "http://api.default.svc.cluster.local"
---
apiVersion: v1
kind: Secret
metadata: { name: web-secret }
type: Opaque
stringData:                       # authored as plaintext; API server base64-encodes
  DB_PASSWORD: "s3cr3t-rotate-me"
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: web }
spec:
  replicas: 3
  selector: { matchLabels: { app: web } }
  template:
    metadata: { labels: { app: web } }
    spec:
      containers:
        - name: web
          image: ghcr.io/example/web:1.4.2
          envFrom:
            - configMapRef: { name: web-config }     # bulk non-sensitive config
          env:
            - name: DB_PASSWORD                        # single sensitive value
              valueFrom:
                secretKeyRef: { name: web-secret, key: DB_PASSWORD }
            - name: POD_NAME
              valueFrom: { fieldRef: { fieldPath: metadata.name } }
          resources:
            requests: { cpu: "250m", memory: "256Mi" }
            limits:   { cpu: "1",    memory: "512Mi" }   # Burstable QoS
          volumeMounts:
            - name: tls
              mountPath: /etc/tls
              readOnly: true
      volumes:
        - name: tls
          secret:
            secretName: web-tls          # a kubernetes.io/tls Secret mounted as files

This Pod is Burstable (requests < limits), pulls bulk config via envFrom, one secret via secretKeyRef, its own name via the downward API, and TLS certs as mounted files. The password is a candidate for ESO/Sealed Secrets rather than plaintext stringData in real life.

Best Practices

1. Never bake config or secrets into the image — inject at runtime via ConfigMaps/Secrets so one immutable image promotes unchanged across environments; a rebuilt image per environment defeats reproducibility and auditing.
2. Always set resource requests, and set them from data — requests are what the scheduler uses to place Pods; without them the scheduler is blind, packing leads to evictions, and the Pod falls to BestEffort. Base them on observed usage (VPA recommendations, metrics), not guesses.
3. Set memory limits equal to requests for critical workloads — memory is incompressible; making a latency-sensitive Pod Guaranteed protects it from eviction. For CPU, prefer requests-only or generous limits since throttling hurts tail latency.
4. Treat base64 as encoding, not security — never assume a Secret is protected because it “looks encrypted”; lock down RBAC, enable encryption at rest, and keep Secrets out of logs and Git.
5. Enable encryption at rest with a KMS v2 provider — envelope encryption backed by a cloud KMS/HSM keeps root keys off the control plane and supports rotation; re-encrypt existing Secrets after enabling, and keep identity last only during migration.
6. Scope Secret access tightly with RBAC — grant get/list on Secrets to as few ServiceAccounts and users as possible; a list secrets grant in a namespace effectively hands over every credential in it.
7. Prefer mounted-volume Secrets over env vars for sensitive data — env vars leak into child processes, crash dumps, /proc, and logging that dumps the environment; mounted files are narrower and update in place.
8. Use an external secrets workflow for lifecycle — External Secrets Operator when a secret manager is the source of truth, Sealed Secrets or SOPS when Git must be, so plaintext never lands in a repo and rotation is centralized.
9. Make ConfigMaps/Secrets immutable and content-hash their names — immutability prevents accidental fleet-wide changes and cuts apiserver watch load; hashed names turn a config change into an automatic, traceable rollout.
10. Automate config-change rollouts — remember env vars are frozen at Pod start; use Reloader, hashed names, or an explicit kubectl rollout restart so config changes actually reach running Pods.
11. Deploy LimitRange and ResourceQuota together per namespace — the quota caps a tenant’s blast radius while the LimitRange supplies defaults so developers aren’t forced to annotate every Pod and no BestEffort Pods slip through.
12. Watch your memory units — Mi/Gi (binary) vs M/G (decimal) differ by ~5–7%; standardize on Mi/Gi to avoid subtly under-provisioned limits that cause surprise OOM-kills.
13. Feed the app its own limits via the downward API — pass resourceFieldRef memory/CPU limits so runtimes (JVM, Go, Node) size heaps and pools to the container, not the whole node — a top cause of container OOM-kills.
14. Avoid subPath mounts for config you expect to reload — subPath-mounted ConfigMap/Secret files do not receive live updates; mount the whole volume if the app watches files for changes.
15. Keep Secrets and ConfigMaps small and single-purpose — the ~1 MiB object limit is a ceiling, not a target; large blobs bloat etcd and every Pod’s env, and one giant shared map couples unrelated apps.
16. Rotate secrets and design for it — assume every credential will be rotated; use external managers with rotation, reference secrets by name so a new version flows through, and never treat a committed secret as safe even if encrypted.

References

• ConfigMaps
• Secrets
• Encrypting Confidential Data at Rest
• Using a KMS provider for data encryption
• Distribute Credentials Securely Using Secrets
• Expose Pod Information to Containers Through the Downward API
• Managing Resources for Containers (requests & limits)
• Pod Quality of Service Classes
• Resource Quotas · Limit Ranges
• Node-pressure Eviction
• External Secrets Operator · Sealed Secrets · SOPS
• Reloader (Stakater)
• roadmap.sh — Kubernetes