Terminal & ScriptingTerminal & Scripting

Part of the DevOps Roadmap knowledge base.

Overview

The terminal is the primary interface of DevOps work. GUIs and web consoles exist, but servers are administered over SSH, CI jobs run shell steps, containers have no desktop, and every serious infrastructure tool ships as a CLI first. Fluency in the shell — navigating, composing pipelines, editing files with vim, and reading system state with monitoring commands — is the difference between resolving an incident in five minutes and in an hour.

Shell scripting is the automation layer that predates and underlies everything else: bootstrap scripts, CI pipeline steps, Docker entrypoints, cron jobs, and installer scripts (curl … | bash) are almost all Bash. Even when your “real” automation lives in Python or Ansible, shell one-liners are how you explore, verify, and debug. The Unix philosophy — small tools that do one thing well, composed with pipes — is a design lesson that applies to all of DevOps, not just the command line.

This topic covers three skill clusters: writing robust Bash/Zsh scripts, mastering the classic text-processing and monitoring toolset (grep, sed, awk, ps, top, vmstat, sar), and using networking commands (curl, dig, ss, tcpdump) to debug connectivity from first principles. The through-line is composability: once you internalize that every command reads stdin and writes stdout, you can build an unbounded number of ad-hoc tools by connecting a handful of primitives.

Fundamentals

Shell basics: Bash and Zsh

Bash is the default scripting shell on most Linux systems; Zsh (default on macOS since Catalina) is largely Bash-compatible interactively but scripts should target Bash (or POSIX sh) for portability. Note that /bin/sh on Debian/Ubuntu is dash, not Bash — so #!/bin/sh scripts must avoid “bashisms” like arrays and [[ ]]. Core mechanics to internalize:

• Streams and file descriptors: every process has stdin (fd 0), stdout (fd 1), and stderr (fd 2). Understanding these three is the key to redirection.
• Pipes and redirection: cmd1 | cmd2 (stdout → stdin), > file (overwrite stdout), >> file (append), 2>&1 (redirect stderr to wherever stdout currently points — order matters), &> file (both), < file (stdin), <<< (here-string), <<EOF (heredoc), 2>/dev/null (discard errors).
• Exit codes: 0 = success; 1–255 = failure; $? holds the last exit code; && runs the next command only on success, || only on failure. 130 = terminated by Ctrl-C (128+SIGINT).
• Quoting: double quotes expand variables and command substitution ("$var"), single quotes are literal ('$var' stays $var). Unquoted variables undergoing word-splitting and globbing are the number-one source of shell bugs.
• Expansion order (worth knowing to debug surprises): brace → tilde → parameter/variable → command substitution → arithmetic → word splitting → filename globbing. Word splitting happening after variable expansion is exactly why unquoted variables break on spaces.
• Substitution and parameter expansion: command substitution $(cmd), arithmetic $((x + 1)), defaults ${var:-default}, assign-default ${var:=default}, error-if-unset ${var:?message}, strip suffix ${file%.txt}, strip prefix ${path##*/} (basename), length ${#var}, substitution ${var/old/new}.

A robust script skeleton:

#!/usr/bin/env bash
set -Eeuo pipefail            # -e exit on error, -u error on unset var,
                             # -o pipefail catch failures anywhere in a pipe,
                             # -E make ERR trap fire inside functions
IFS=$'\n\t'                  # safer word-splitting (no bare spaces)
trap 'echo "error on line $LINENO (exit $?)" >&2' ERR
trap 'rm -rf "$tmp"' EXIT     # always clean up, even on error

tmp="$(mktemp -d)"
readonly LOG_DIR="${LOG_DIR:-/var/log/myapp}"

usage() { echo "usage: $0 <env>" >&2; exit 2; }
[[ $# -eq 1 ]] || usage

log() { printf '%s %s\n' "$(date -Is)" "$*" >&2; }

main() {
  local env="$1"
  for f in "$LOG_DIR"/*.log; do
    [[ -e "$f" ]] || continue    # guard against no-match glob
    gzip -k -- "$f"              # -- stops option parsing (filenames with dashes)
  done
  log "done for $env"
}
main "$@"

Every line here earns its place: set -Eeuo pipefail turns silent failure into a loud crash, the EXIT trap guarantees cleanup, mktemp -d avoids predictable temp paths, [[ -e "$f" ]] || continue handles the “glob matched nothing” case, and -- prevents filenames from being parsed as options.

Text editors: vim survival and beyond

You will land on a server where vim (or vi) is the only editor. Minimum survival kit:

Beyond survival, learn operators + motions/text objects, because they compose multiplicatively rather than additively: d (delete), c (change), y (yank) combine with iw (inner word), i" (inside quotes), ip (inner paragraph), it (inside tag). So ciw changes a word, di" deletes inside quotes, ci( changes inside parentheses, dap deletes a paragraph. Learn . (repeat last change) and counts (3dd). vimtutor (30 minutes at the terminal) is the fastest structured start; nano is a fine fallback if it is installed and you are not editing over a flaky link.

The Unix pipeline mindset

# Top 10 client IPs by request count in an access log
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10

Each tool does one job; | composes them into a data-processing assembly line: awk extracts the field, sort groups identical lines together so uniq -c can count them, the second sort -rn orders by count descending, head takes the top 10. Learning ~15 core tools deeply beats memorizing hundreds of flags — and the same five-stage shape (extract → sort → count → sort → limit) solves a huge fraction of real log-analysis questions.

Key Concepts

Text manipulation toolbox

Choosing among grep/sed/awk is a common source of confusion. A rule of thumb: grep to find/filter lines, sed for simple line-oriented substitution/deletion, awk when you need fields, arithmetic, conditionals, or accumulation. If your sed command has grown more than one ;-separated clause and a hold buffer, it should probably be awk (or a real script).

awk deserves real investment — it is a complete language for tabular data:

# Sum column 5 (bytes) grouped by column 1 (IP), print totals sorted
awk '{bytes[$1] += $5} END {for (ip in bytes) print bytes[ip], ip}' access.log | sort -rn

jq deserves equal attention — nearly every modern API and CLI emits JSON, and parsing it with grep/sed is fragile:

curl -s https://api.github.com/repos/kubernetes/kubernetes |
  jq '{name: .name, stars: .stargazers_count, license: .license.spdx_id}'

# select + map: names of pods that are not Running
kubectl get pods -o json |
  jq -r '.items[] | select(.status.phase != "Running") | .metadata.name'

Regular expressions: the shared substrate

grep, sed, awk, and most tools share regular expressions, but in dialects: BRE (basic — grep, sed by default, where + ? { } ( ) must be backslash-escaped), ERE (extended — grep -E, awk, sed -E, where those metacharacters work unescaped), and PCRE (grep -P, with lookaround and \d). When a pattern “doesn’t work,” the dialect is the usual culprit. Prefer -E for readability. Anchors (^ $), classes ([[:digit:]], \w), quantifiers (* + ? {n,m}), and groups/backreferences are the vocabulary worth memorizing.

Process and performance monitoring

Reading vmstat 1 output (the columns that matter):

• r (run queue) — runnable threads waiting for CPU. Persistently above the CPU count → CPU saturation.
• b — processes blocked on I/O. Consistently nonzero → I/O wait.
• si/so — swap-in / swap-out per second. Nonzero → the machine is swapping; latency will suffer.
• us / sy / id / wa / st — CPU time in user / kernel / idle / iowait / stolen (by the hypervisor). High wa → disk-bound (confirm with iostat); high sy → syscall or context-switch storm; high st → a noisy neighbor is stealing your CPU.

Load average nuance: it counts both running and uninterruptible-sleep (D-state, usually disk) tasks on Linux, so a high load with low CPU usually means I/O, not compute. Divide load by core count for a per-core figure.

A practical 60-second incident triage, based on Brendan Gregg’s checklist:

uptime                     # 1. load trend — rising, falling, steady?
dmesg -T | tail            # 2. OOM kills, disk errors, kernel warnings
vmstat 1 5                 # 3. CPU vs memory vs swap vs iowait at a glance
free -h                    # 4. memory headroom (watch "available")
iostat -xz 1 3             # 5. is a disk saturated? (%util near 100, high await)
ss -s                      # 6. socket/connection state summary
top                        # 7. now zoom into the guilty process

Networking commands

The most valuable curl skill is the -w timing breakdown, which tells you where latency lives (DNS, connect, TLS, first byte):

curl -sS -o /dev/null \
  -w 'dns:%{time_namelookup} conn:%{time_connect} tls:%{time_appconnect} ttfb:%{time_starttransfer} total:%{time_total}\n' \
  https://api.example.com/

A layered connectivity debug flow — always work up the stack, since each layer depends on the one below:

dig +short svc.example.com          # 1. Does DNS resolve, and to what?
ping -c3 <ip>                        # 2. Is the host reachable? (ICMP may be blocked — not conclusive)
nc -zv <ip> 5432                     # 3. Is the TCP port open / listening?
curl -v https://svc.example.com/     # 4. Does the app + TLS layer answer correctly?
sudo tcpdump -nn host <ip> and port 5432   # 5. What is actually on the wire? (last resort, ground truth)

Interpreting the outcome: DNS fails → resolver/records; ping works but nc refused → service down or firewall on that port; nc works but curl hangs → TLS or application-layer problem; everything “works” but the app misbehaves → tcpdump to see the real bytes.

Bash control flow worth memorizing

# Retry with exponential backoff — a pattern you will reuse forever
for i in {1..5}; do
  curl -fsS https://example.com/healthz && break
  echo "attempt $i failed; retrying in $((2 ** i))s" >&2
  sleep $((2 ** i))
done

# Safe iteration over find results (handles spaces/newlines in names)
find /data -name '*.csv' -print0 | while IFS= read -r -d '' f; do
  process "$f"
done

# Read a file line by line without losing the last unterminated line
while IFS= read -r line || [[ -n "$line" ]]; do
  echo "got: $line"
done < input.txt

# Case dispatch (the shape of most service-control scripts)
case "${1:-}" in
  start)   systemctl start myapp ;;
  stop)    systemctl stop myapp ;;
  restart) systemctl restart myapp ;;
  *)       echo "usage: $0 {start|stop|restart}" >&2; exit 2 ;;
esac

# Arrays and associative arrays (Bash 4+)
declare -A limits=([cpu]=2 [mem]=512Mi)
for k in "${!limits[@]}"; do echo "$k -> ${limits[$k]}"; done

Interactive shell efficiency

The interactive shell rewards a small set of habits: Ctrl-r for reverse history search, !! (last command), !$ (last argument), Ctrl-a/Ctrl-e (line start/end), Ctrl-w/Ctrl-u (delete word/line), cd - (previous directory), and Alt-. (insert last argument). Persistent sessions matter over SSH: tmux (or screen) keeps your work alive across disconnects and lets you split panes for monitoring while you work. Learn to configure history (HISTSIZE, HISTCONTROL=ignoredups:erasedups), aliases, and a prompt that shows the current directory, git branch, and Kubernetes context so you never run the right command in the wrong place.

Best Practices

1. Start every Bash script with set -Eeuo pipefail. Silent failure mid-script is how half-finished deployments happen; make errors loud and fatal. Add IFS=$'\n\t' to tame word-splitting.
2. Quote every variable expansion ("$var", "${arr[@]}"). Unquoted expansions undergo word-splitting and globbing and break on spaces, tabs, and * — the classic and most frequent shell bug class.
3. Run ShellCheck on all scripts (in your editor and in CI). It statically catches quoting, portability, and logic errors instantly and teaches you the rules as it goes.
4. Prefer $(...) over backticks and [[ ... ]] over [ ... ] in Bash — command substitution nests cleanly, and [[ ]] avoids word-splitting pitfalls and supports pattern/regex matching.
5. Keep scripts idempotent. Use mkdir -p, ln -sf, rm -f, and explicit state checks before mutations; re-running after a partial failure must be safe and produce the same end state.
6. Guard destructive commands. Support --dry-run, echo the command before running it, and never let rm -rf "$dir/" run with a possibly-empty variable — use ${dir:?dir is required} so an unset value aborts instead of deleting /.
7. Graduate to Python past ~100–200 lines or when you need data structures, real error handling, arithmetic on floats, or tests. Shell is glue and orchestration, not an application language.
8. Use mktemp for temp files and trap ... EXIT for cleanup. Predictable temp paths (/tmp/myapp.tmp) are both a race-condition bug and a security issue (symlink attacks); guaranteed cleanup prevents disk leaks.
9. Learn jq/yq properly. Parsing JSON/YAML with grep/sed/cut is fragile and breaks on formatting changes; one jq expression replaces a brittle pipeline and survives reformatting.
10. Master the pipeline five-step (extract | sort | uniq -c | sort -rn | head). It answers a large fraction of “what’s the top N …?” questions over logs without any scripting at all.
11. Understand vmstat/iostat/top columns, not just the commands. Knowing that high wa means iowait and high st means a noisy neighbor is what turns raw numbers into a diagnosis.
12. Debug connectivity in layers, bottom-up (DNS → host → port → app → packets). Jumping straight to tcpdump wastes time when dig would have shown the problem in one line.
13. Capture evidence before you conclude. During incidents, redirect outputs to files (vmstat 1 60 > vmstat.txt, tcpdump -w cap.pcap, dmesg -T > dmesg.txt) so the transient state survives for analysis after mitigation.
14. Prefer curl -w timing over guessing. When something is “slow,” the DNS/connect/TLS/TTFB breakdown tells you which layer to fix instead of speculating.
15. Use persistent sessions (tmux) for anything long-running over SSH. A dropped connection should never kill a migration or a tail -f you are watching.
16. Comment the why, not the what, and include a usage() function. Six months later, the intent behind a clever pipeline and the meaning of each flag are documented only if you wrote them down.

References

• DevOps Roadmap — roadmap.sh/devops
• GNU Bash Reference Manual
• Bash Guide (mywiki.wooledge.org / greg’s wiki) and its essential BashFAQ / BashPitfalls
• Google Shell Style Guide
• ShellCheck — shell script static analysis
• explainshell — annotate any command
• jq Manual and jq play (online)
• GNU sed manual
• GNU awk (gawk) manual — and The AWK Programming Language (Aho, Kernighan, Weinberger)
• GNU grep manual
• curl documentation and Everything curl (free book)
• regex101 — build & debug regular expressions
• Vim documentation — and run vimtutor locally
• tmux wiki / manual
• Brendan Gregg — Linux Performance and the USE Method
• Book: The Linux Command Line — William Shotts (free PDF)
• Book: Classic Shell Scripting — Robbins & Beebe (O’Reilly)
• Book: Wicked Cool Shell Scripts — Dave Taylor & Brandon Perry (No Starch Press)