Observability & Troubleshooting (Khả năng quan sát & Gỡ lỗi)Observability & Troubleshooting

Part of the Kubernetes Roadmap knowledge base.

Overview

Kubernetes is a control loop that constantly reconciles desired state (what your YAML says) against actual state (what the kubelet and scheduler observe). Observability and troubleshooting on Kubernetes is the discipline of making that reconciliation legible: understanding why a Pod is Pending, why it keeps restarting, and whether your workloads are actually healthy from a user’s perspective — not just “running”. The platform gives you primitives that no ordinary VM stack has: probes that let the kubelet act on health, Events that narrate every scheduling and lifecycle decision, per-container resource accounting, and a uniform kubectl surface for logs, exec, and live debugging.

The mental model that saves the most time: almost every Kubernetes problem is either a scheduling problem (the Pod can’t be placed — Pending, unschedulable), an image problem (the container image can’t be pulled — ImagePullBackOff), a startup/runtime problem (the process crashes or fails its probes — CrashLoopBackOff, readiness failing), or a resource problem (the container is throttled or killed — OOMKilled, evicted). Each class has a distinct signature in kubectl describe, kubectl logs, and the Events stream, and each has a well-worn diagnostic path. Learning to read those signatures turns “the app is down” from a panic into a five-minute triage.

On top of raw troubleshooting sits the observability stack: metrics-server for kubectl top and autoscaling, the Prometheus ecosystem (kube-state-metrics, node-exporter, Grafana, the Prometheus Operator) for cluster-wide monitoring, node-level log aggregation (Fluent Bit → Loki/Elasticsearch), and distributed tracing via OpenTelemetry for request-level insight. Kubernetes deliberately ships no long-term metrics or log storage in the box — it emits telemetry (container stdout/stderr, cAdvisor metrics, Events with a short TTL) and expects you to bolt on a stack to collect and retain it. Understanding that division — what the platform gives you versus what you must add — is the whole game.

The observability contract of a container on Kubernetes

A well-behaved workload does three things that make it observable: it writes logs to stdout/stderr (never to a file inside the container, which the node agent can’t reach), it exposes health endpoints the kubelet can probe (liveness/readiness/startup), and it declares resource requests and limits so the scheduler and the OOM killer make correct decisions. Get these three right and 80% of “why is my app broken on Kubernetes?” answers itself from kubectl alone.

Fundamentals

Health probes: liveness, readiness, startup

The kubelet actively probes containers to decide their fate. There are three probe types, each answering a different question, and confusing them causes the most self-inflicted outages on Kubernetes.

Effects you must internalize:

• A liveness failure is a hammer: it kills and restarts the container. If your liveness probe checks something that depends on a slow dependency (a database), a transient DB blip triggers a restart storm — you’ve turned a minor issue into an outage. Liveness should test only the process itself (can it respond at all?), never downstream dependencies.
• A readiness failure is graceful: the Pod stays running but is pulled out of the Service load-balancer rotation, so no user traffic reaches it. This is exactly what you want during a slow dependency or a warmup — the Pod waits, doesn’t get killed, and rejoins when healthy. Readiness may check dependencies (should I receive traffic if my DB is unreachable?).
• A startup probe exists for slow-booting apps (legacy JVMs, apps loading large models). Without it, you’re forced to set a huge initialDelaySeconds on liveness, which also delays detecting a real hang. The startup probe lets you say “give this app up to 5 minutes to boot, but once booted, check liveness every 10 seconds” — decoupling boot tolerance from steady-state sensitivity.

Each probe supports three mechanisms: httpGet (2xx/3xx = pass), tcpSocket (can we open the port?), and exec (a command with exit code 0 = pass). grpc probes are also supported natively. Tunables: initialDelaySeconds, periodSeconds, timeoutSeconds, successThreshold, failureThreshold.

# A container with all three probes done correctly.
containers:
  - name: api
    image: myorg/api:1.4.2
    ports:
      - containerPort: 8080
    # Startup: allow up to 30 * 10s = 5 min to boot, then hand off.
    startupProbe:
      httpGet: { path: /healthz, port: 8080 }
      periodSeconds: 10
      failureThreshold: 30
    # Liveness: is the process wedged? Cheap, dependency-free endpoint.
    livenessProbe:
      httpGet: { path: /healthz, port: 8080 }
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3          # 3 * 10s = 30s of failure before restart
    # Readiness: can I serve? May reflect dependency health.
    readinessProbe:
      httpGet: { path: /ready, port: 8080 }
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 3

metrics-server and kubectl top

metrics-server is a lightweight, cluster-wide aggregator of resource metrics (CPU and memory) scraped from each kubelet’s Summary API (backed by cAdvisor). It is not a monitoring system — it keeps only the latest sample in memory, no history — but it powers two critical things: the kubectl top command and the Horizontal Pod Autoscaler (and VPA). Most clusters need it installed; managed offerings often include it.

kubectl top nodes                 # CPU/memory usage per node vs capacity
kubectl top pods -A               # usage per pod across all namespaces
kubectl top pods --containers     # break down by container within pods
kubectl top pod mypod --sort-by=memory

If kubectl top returns “Metrics API not available”, metrics-server is missing or unhealthy (a very common cause on kubeadm/self-managed clusters is the --kubelet-insecure-tls flag being needed for self-signed kubelet certs).

The monitoring stack (Prometheus ecosystem)

For real, historical, alertable monitoring, the de-facto standard is the Prometheus stack, usually installed as a bundle via the kube-prometheus-stack Helm chart:

• Prometheus — pull-based metrics server that scrapes /metrics endpoints discovered via Kubernetes service discovery. Stores time series locally; remote-writes to Thanos/Mimir/Cortex for long-term, HA storage.
• kube-state-metrics (KSM) — exposes the state of Kubernetes objects as metrics: kube_pod_status_phase, kube_deployment_status_replicas_available, kube_pod_container_status_restarts_total, etc. This is how you alert on “a Deployment has fewer ready replicas than desired” or “a Pod restarted 5 times in 10 minutes”. KSM describes objects; it is distinct from metrics-server (which measures resource usage).
• node-exporter — a DaemonSet exposing host-level metrics (CPU, memory, disk, filesystem, network) per node.
• cAdvisor — built into the kubelet; exposes per-container resource metrics (container_cpu_usage_seconds_total, container_memory_working_set_bytes). Prometheus scrapes it via the kubelet.
• Grafana — dashboards over Prometheus (and Loki/Tempo). The kube-prometheus-stack ships curated cluster/node/workload dashboards out of the box.
• Prometheus Operator — a controller that manages Prometheus, Alertmanager, and scrape config through CRDs: ServiceMonitor and PodMonitor (declare what to scrape via label selectors, instead of editing a giant static config), PrometheusRule (alerting/recording rules), and Alertmanager (routing). This turns monitoring config into version-controlled Kubernetes objects.

# ServiceMonitor: the Operator scrapes any Service matching these labels,
# on the port named "metrics", at /metrics — no central config edits.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: api
  labels: { release: kube-prometheus-stack }
spec:
  selector:
    matchLabels: { app: api }
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

Logging on Kubernetes

Kubernetes has no native cluster-wide log storage. The model is:

1. Containers write to stdout/stderr. The container runtime captures those streams and the kubelet writes them to files on the node (/var/log/pods/...), rotated by the kubelet.
2. kubectl logs reads those node-local files — which means logs vanish when the Pod is deleted or the node dies, and are lost across container restarts unless you ask for the previous instance.
3. For durable, searchable, cluster-wide logs you run a node-level agent as a DaemonSet that tails every container’s log files and ships them to a backend.

kubectl logs mypod                       # current container logs
kubectl logs mypod -c sidecar            # a specific container in a multi-container pod
kubectl logs mypod --previous            # logs from the PREVIOUS (crashed) container — vital for CrashLoopBackOff
kubectl logs -f deploy/api               # follow logs of one pod of a Deployment
kubectl logs -l app=api --all-containers --tail=100   # across all pods matching a label

Node-level agents and backends:

• Fluent Bit — lightweight C log processor/forwarder, the usual DaemonSet choice; low footprint.
• Fluentd — heavier, plugin-rich Ruby collector (CNCF, same family).
• Vector — a modern high-performance alternative.
• Backends: Loki (label-indexed, cheap, pairs with Grafana and LogQL), the ELK/Elastic or OpenSearch stack (full-text search, powerful but heavier), or a managed service (CloudWatch, Cloud Logging).

The anti-pattern is writing logs to a file inside the container: the node agent tails stdout/stderr, so a log file on the container filesystem is invisible to the collector and lost on restart. Twelve-Factor logging (stdout as an event stream) exists precisely for this.

Kubernetes Events

Events are short-lived (default TTL ~1 hour) records the control plane emits to narrate what’s happening to objects: scheduling decisions, image pulls, probe failures, volume mounts, OOM kills, evictions. They are the single richest, most under-used troubleshooting signal.

kubectl get events --sort-by=.lastTimestamp          # chronological, current namespace
kubectl get events -A --sort-by=.lastTimestamp       # all namespaces
kubectl describe pod mypod                            # Events for THIS pod at the bottom — start here
kubectl get events --field-selector involvedObject.name=mypod,type=Warning

Because Events expire, chasing an intermittent issue means shipping them to your logging backend (Fluent Bit/kube-events exporters can do this) so you retain the history.

Distributed tracing (pointer)

Metrics tell you a service is slow; traces tell you where in a multi-service request the time went. On Kubernetes the standard is OpenTelemetry (OTel): instrument apps with OTel SDKs (or auto-instrumentation), run the OTel Collector as a DaemonSet/Deployment to receive, batch, sample and export spans, and store them in Jaeger or Grafana Tempo. The load-bearing detail is context propagation — the W3C traceparent header must survive every hop (including through Services, ingress, and message queues) or the trace fragments. Tracing is covered in depth in the DevOps observability page; on Kubernetes the key is deploying the Collector and wiring the sidecar/DaemonSet correctly.

Key Concepts

The troubleshooting workflow

A repeatable triage order beats guessing. Start wide, then narrow:

# 1. What's the high-level state? Look for non-Running / non-Ready / restart counts.
kubectl get pods -o wide
kubectl get pods -A | grep -vE 'Running|Completed'

# 2. Why? describe shows Events, probe results, the last state, exit codes, and reasons.
kubectl describe pod <pod>

# 3. What did the app say? Logs — and PREVIOUS logs if it crashed.
kubectl logs <pod>
kubectl logs <pod> --previous

# 4. Poke inside a running container (only if it stays up).
kubectl exec -it <pod> -- sh
kubectl exec <pod> -- env            # verify config/env vars, DNS, mounted secrets

# 5. If the container has no shell / is crashing, attach an ephemeral debug container.
kubectl debug -it <pod> --image=busybox:1.36 --target=<container>

# 6. Cluster-level context.
kubectl get events --sort-by=.lastTimestamp
kubectl top pod <pod> --containers   # is it near its memory/CPU limit?

kubectl debug and ephemeral containers deserve emphasis: modern hardened images are often distroless (no shell, no curl, no package manager), so kubectl exec ... -- sh fails. Ephemeral containers let you attach a fully-tooled debug container that shares the target Pod’s namespaces (process, network) without restarting or modifying the Pod — you can inspect the crashing process’s network and files live. kubectl debug node/<node> gives you a privileged container on a node for node-level debugging.

Common failure signatures

Exit code cheatsheet: 0 = clean exit; 1/2 = app error; 137 = SIGKILL (usually OOMKilled or failed liveness kill); 143 = SIGTERM (graceful shutdown); 139 = SIGSEGV.

QoS classes and the OOM killer

A container’s memory limit is enforced by the kernel cgroup: exceed it and the process is OOMKilled (137). The request influences scheduling and the Pod’s QoS class, which decides eviction order under node pressure:

• Guaranteed — every container has requests == limits for both CPU and memory. Last to be evicted.
• Burstable — at least one request set, but not equal to limits. Evicted after BestEffort.
• BestEffort — no requests/limits at all. First to be evicted under pressure.

This is why “no limits = fast” is a trap: BestEffort Pods are the first casualties when a node runs hot. Setting sensible requests both keeps the scheduler honest and protects your Pod during pressure.

Reading kubectl describe like an X-ray

The three most diagnostic sections of kubectl describe pod: State / Last State (current status, and how the previous container instance died — exit code and reason, e.g. OOMKilled), Conditions (Ready, PodScheduled, ContainersReady — a False here localizes the failure), and the Events list at the bottom (the chronological narrative — pull errors, probe failures, scheduling messages). Ninety percent of first-line triage is here before you ever look at application logs.

Best Practices

1. Define liveness, readiness, and startup probes deliberately — they are not interchangeable. Readiness gates traffic; liveness restarts; startup buys boot time. Mixing them up (e.g. a liveness probe that checks a database) turns transient dependency blips into restart storms.
2. Keep liveness probes cheap and dependency-free. Liveness should answer “is this process wedged?” — never call downstream services from it, or a dependency outage becomes a cascading restart loop across your fleet.
3. Let readiness reflect the ability to serve, including warmup and dependencies. A Pod that isn’t ready is simply removed from the load balancer, exactly the graceful behavior you want during warmup or a degraded dependency.
4. Use a startup probe for slow-booting apps instead of a huge liveness initialDelaySeconds. It decouples boot tolerance from steady-state hang detection, so you can be patient at startup and sensitive afterward.
5. Always set resource requests and (memory) limits. Requests drive scheduling and QoS; a memory limit prevents one leaky Pod from taking down a node. No requests means BestEffort — first to be evicted under pressure.
6. Log to stdout/stderr as structured JSON, never to a file in the container. The node agent only sees stdout/stderr; file logs are invisible to the collector and lost on restart. Include a trace_id for cross-signal correlation.
7. Install metrics-server and a real monitoring stack. metrics-server enables kubectl top and autoscaling; kube-prometheus-stack gives historical, alertable metrics. Neither is optional for a production cluster.
8. Alert on Kubernetes-object state via kube-state-metrics. Alert on “replicas available < desired”, “restarts spiking”, “Pods stuck Pending”, and node pressure — symptoms of workload health, not just raw CPU.
9. Manage scrape config with the Prometheus Operator’s CRDs. ServiceMonitor/PodMonitor/PrometheusRule make monitoring config declarative, version-controlled, and reviewable instead of a fragile central file.
10. Ship Events and logs off-node before they expire. Events have a ~1h TTL and container logs die with the Pod; forward both to a durable backend (Loki/Elastic) so post-mortems have the history.
11. Learn the describe → logs --previous → exec/debug triage order. Start with describe (Events, exit codes, conditions), read the previous logs for crash loops, and only then poke inside. This order resolves most incidents fast.
12. Master ephemeral debug containers for distroless images. Hardened images have no shell; kubectl debug --target attaches a fully-tooled container sharing the Pod’s namespaces without restarting it — the only way to inspect a crashing distroless Pod live.
13. Read exit codes as first-class evidence. 137 = OOMKilled or liveness kill, 143 = graceful SIGTERM, 1/2 = app error. The code plus the “Last State” reason usually names the failure class immediately.
14. Set graceful shutdown (terminationGracePeriodSeconds + SIGTERM handling) and readiness on shutdown. During a rolling update, an app that ignores SIGTERM drops in-flight requests; failing readiness first drains it from the Service before the kill.
15. Instrument requests with OpenTelemetry and propagate traceparent. Metrics say a service is slow; traces say where. A dropped propagation header silently fragments traces — verify it survives every hop, including ingress and queues.
16. Right-size probes and thresholds; test failure modes. Too-tight timeoutSeconds/failureThreshold cause flapping and false restarts. Run game-days: kill a dependency, exhaust memory, and verify your probes, alerts, and dashboards behave as intended.

References

• roadmap.sh — Kubernetes Roadmap
• Kubernetes — Configure Liveness, Readiness and Startup Probes
• Kubernetes — Pod Lifecycle (probes, restart policy, conditions)
• Kubernetes — Troubleshooting Applications
• Kubernetes — Debug Running Pods & Ephemeral Containers
• Kubernetes — Debug a Pod with kubectl debug
• Kubernetes — Logging Architecture
• Kubernetes — Resource Management & QoS Classes
• Kubernetes — Tools for Monitoring Resources (Metrics Pipelines)
• metrics-server (GitHub)
• Kubernetes — kubectl top reference
• Prometheus documentation · kube-state-metrics · node_exporter
• Prometheus Operator documentation · kube-prometheus-stack Helm chart
• Grafana documentation · Grafana Loki
• Fluent Bit for Kubernetes · Fluentd
• OpenTelemetry — Kubernetes · Jaeger · Grafana Tempo
• Google SRE Book — Monitoring Distributed Systems