Công cụ & Xử lý sự cốTools & Troubleshooting

Part of the Network Engineer Roadmap knowledge base.

Overview

Troubleshooting is the single most valuable skill a network engineer has. Networks fail in messy, ambiguous ways — a user says “the internet is slow” or “I can’t reach the app,” and it is your job to turn that vague symptom into a precise root cause. The difference between a junior and a senior engineer is rarely knowledge of one more command; it is method. A senior engineer forms a hypothesis, tests it cheaply, and eliminates whole classes of problems with each step. A junior engineer runs ping a dozen times and hopes.

This note is deliberately hands-on. It pairs a structured methodology (how to think) with the standard toolset (what to run) and a set of symptom → cause → tool tables (what to reach for when). The tools split into a few families:

Everything here builds on the layering model in ./03-network-models-and-layers.md and the protocols in ./06-core-protocols.md. Troubleshooting and monitoring are two sides of one coin — monitoring (./14-monitoring-and-observability.md) tells you that something is wrong; troubleshooting tells you why.

Fundamentals

A structured methodology

The goal of a method is to make progress guaranteed rather than lucky. Every good approach walks the OSI/TCP-IP stack in some order, because a failure at a lower layer makes everything above it fail too. There is no point debugging HTTP if the cable is unplugged.

The universal loop is: isolate → hypothesize → test → fix → verify.

1. Isolate. Narrow the scope. Is it one host or all hosts? One site or the LAN? One application or all traffic? One direction or both? Reproduce it if you can. “Works from my laptop but not the server” is a hundred times more useful than “it’s broken.”
2. Hypothesize. State a falsifiable guess: “DNS is resolving to a stale IP,” “the default gateway is unreachable,” “there’s an MTU mismatch on the tunnel.” A hypothesis you can’t test with one command is too vague.
3. Test. Run the cheapest command that confirms or kills the hypothesis. A single ping 8.8.8.8 distinguishes “no connectivity” from “DNS broken” instantly.
4. Fix. Make one change at a time. Changing three things and seeing it work teaches you nothing and may have masked the real fault.
5. Verify. Confirm from the user’s perspective, not just yours. Then confirm you didn’t break something else. Document the root cause.

Three ways to walk the layers

A concrete bottom-up walk:

# L1/L2 — is the link up and do we have an address?
ip link show                 # state UP? Any carrier?
ip addr show                 # do we have an IPv4/IPv6 address?

# L3 — can we reach the gateway, then the internet by IP?
ip route show                # is there a default route?
ping -c 3 192.168.1.1        # the default gateway (LAN reachability)
ping -c 3 8.8.8.8            # a public IP (routing + internet, no DNS)

# L7-ish — is name resolution working?
dig +short example.com       # does DNS return an answer?
ping -c 3 example.com        # end-to-end by name

# L7 — does the actual service respond?
curl -I https://example.com  # HTTP status line and headers

If every step passes until dig, you have proven Layers 1–3 are healthy and the fault is DNS. That is the power of walking the stack: each passing test permanently rules out a layer.

”Is it DNS? It’s always DNS.”

The running joke exists because DNS sits between a working network and a working application, so DNS faults masquerade as everything else. The instant test that separates DNS from connectivity:

ping -c 3 8.8.8.8            # works?  -> IP connectivity is FINE
ping -c 3 google.com        # fails?  -> the problem is name resolution (DNS)

If pinging an IP works but pinging a name fails, stop looking at routing and cables — go straight to dig.

Reading exit codes and being systematic

Most tools return 0 on success and non-zero on failure, so you can script checks. But more importantly: change one variable at a time, keep a scratch log of what you tried and what you saw, and always test both directions of a connection — asymmetric routing and one-way firewall rules are classic traps.

Key Concepts

Connectivity tools

ping — is the host reachable (ICMP)?

ping sends ICMP Echo Request packets and times the Echo Replies. It answers two questions at once: reachability (do replies come back?) and latency (how long do they take?).

ping -c 4 8.8.8.8

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=12.3 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=11.9 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=118 time=12.1 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=118 time=12.0 ms

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 11.9/12.0/12.3/0.15 ms

How to read it:

• time= is the round-trip time (RTT). Stable low numbers = healthy. Wildly varying numbers = jitter, often a congested or wireless link.
• packet loss should be 0% on a wired LAN. Anything above 1–2% end-to-end is worth investigating.
• ttl= hints at how many hops away the host is (start value minus hops). A Linux host starts at 64, Windows at 128; a reply ttl=118 suggests ~10 hops from a Windows-ish stack.
• Request timeout / Destination Host Unreachable — no reply means either the host is down, a firewall drops ICMP, or there is no route. ICMP being blocked is common, so a failed ping alone does not prove the host is down.

Useful flags:

ping -c 4 -i 0.2 8.8.8.8         # send every 0.2s (faster sampling)
ping -c 4 -s 1472 -M do 8.8.8.8  # 1472-byte payload, don't fragment (MTU test, see below)
ping -c 4 -I eth0 8.8.8.8        # force a source interface
ping6 -c 4 2001:4860:4860::8888  # ICMPv6 to a v6 address

On Windows: ping -n 4 8.8.8.8 (-n = count, -t = ping until Ctrl-C, -l = size, -f = don’t fragment).

traceroute / tracert — where does the packet go, and where does it die?

traceroute reveals the path hop by hop by sending packets with increasing TTL values; each router that decrements TTL to 0 sends back a “Time Exceeded” message, revealing itself.

traceroute -n google.com

traceroute to google.com (142.250.x.x), 30 hops max, 60 byte packets
 1  192.168.1.1      0.5 ms  0.4 ms  0.4 ms
 2  10.20.0.1        8.1 ms  8.0 ms  8.2 ms
 3  100.64.0.1      11.0 ms 10.9 ms 11.1 ms
 4  * * *
 5  142.250.x.x     12.4 ms 12.3 ms 12.5 ms

How to read it:

• Each line is one hop (router) with three RTT probes.
• * * * means no reply at that hop. Often harmless — many routers rate-limit or don’t reply to the probe but still forward traffic. Only worry if * * * continues to the destination.
• A sudden, sustained jump in latency at one hop and every hop after it points to congestion at that link. A jump at one hop only (that recovers) is usually just that router deprioritizing ICMP — not a real problem.
• -n skips reverse-DNS so it runs faster and shows raw IPs.

Linux uses UDP probes by default; add -I for ICMP or -T for TCP (traceroute -T -p 443 example.com), which is invaluable when firewalls block UDP/ICMP but allow the app’s TCP port. Windows tracert uses ICMP by default.

mtr — traceroute + ping, continuously

mtr combines traceroute and ping: it repeatedly probes every hop and shows a live-updating table of loss and latency per hop. It is the best tool for diagnosing intermittent loss and which hop introduces latency.

mtr -rwzbc 100 google.com     # report mode: 100 cycles, wide, show ASN

HOST: myhost                       Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. AS??? 192.168.1.1               0.0%   100    0.4   0.5   0.3   1.2   0.1
 2. AS1234 10.20.0.1                0.0%   100    8.1   8.3   7.9  10.2   0.4
 3. AS5678 100.64.0.1               2.0%   100   11.0  11.4  10.8  25.1   1.9
 4. AS15169 142.250.x.x             0.0%   100   12.4  12.6  12.1  14.0   0.3

Key insight: loss that appears at one hop and continues to the final hop is real loss; loss that appears at a middle hop but is 0% at the destination is just that router deprioritizing ICMP and can be ignored. In the example above, hop 3 shows 2% but hop 4 shows 0% — so the “loss” is cosmetic, the path is actually clean.

DNS tools

See ./06-core-protocols.md for how DNS works; here we focus on the diagnostic commands.

dig — the professional’s DNS tool

dig example.com A

;; QUESTION SECTION:
;example.com.                   IN      A

;; ANSWER SECTION:
example.com.            3600    IN      A       93.184.216.34

;; Query time: 14 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)

Read the ANSWER SECTION for the record and its TTL (here 3600s). SERVER tells you which resolver answered — critical when you suspect the wrong resolver is in use.

dig +short example.com               # just the answer, script-friendly
dig +trace example.com               # walk from root -> TLD -> authoritative (find where resolution breaks)
dig @1.1.1.1 example.com             # query a SPECIFIC resolver (bypass local one)
dig @8.8.8.8 example.com +short      # cross-check: does another resolver agree?
dig example.com MX                   # mail records
dig example.com NS                   # nameservers
dig example.com TXT                  # SPF/DKIM/verification records
dig -x 8.8.8.8                       # reverse lookup (PTR)
dig example.com +norecurse          # ask if the server has it cached (authoritative check)

The single most useful DNS diagnostic: query two resolvers and compare. If dig @1.1.1.1 and dig @8.8.8.8 disagree, you have a stale cache or a propagation issue. dig +trace then shows exactly which nameserver in the delegation chain returns the wrong answer.

nslookup — everywhere, including Windows

nslookup example.com                 # basic A lookup
nslookup example.com 8.8.8.8         # use a specific server
nslookup -type=MX example.com        # specify record type

nslookup ships on Windows and macOS/Linux by default. It is less scriptable than dig and its output format is clunkier, but it is always available.

host — quick and terse

host example.com                     # A, AAAA, and MX in one line each
host -t NS example.com               # nameservers
host 93.184.216.34                   # reverse lookup

Interface & socket tools — what does this host think?

ip (Linux, modern) / ifconfig (legacy) / ipconfig (Windows)

ip addr show                # all interfaces, addresses, state (replaces ifconfig -a)
ip -br addr                 # brief one-line-per-interface summary
ip link show               # L2 state, MAC, MTU, UP/DOWN
ip route show              # the routing table; look for a 'default via' line
ip neigh show              # the ARP/neighbor table (replaces arp -a)
ip -s link show eth0       # interface statistics: RX/TX errors, drops

The ip -br addr brief output is ideal for a quick glance:

lo               UNKNOWN        127.0.0.1/8 ::1/128
eth0             UP             192.168.1.42/24 fe80::.../64

Check for: an interface stuck DOWN, a missing IP (DHCP failed), a 169.254.x.x address (APIPA — DHCP failed, no server answered), or the absence of a default route.

On Windows:

ipconfig /all           # full config: IP, mask, gateway, DNS, MAC, DHCP lease
ipconfig /release       # drop the DHCP lease
ipconfig /renew         # request a new lease
ipconfig /flushdns      # clear the resolver cache (fixes stale DNS on the client)
ipconfig /displaydns    # show cached DNS entries

ipconfig /flushdns is the Windows equivalent fix when a client keeps resolving a name to an old IP.

ss / netstat — sockets and listening ports

ss is the modern, faster replacement for netstat.

ss -tulpn        # TCP+UDP listening sockets with the owning process and port
ss -tan          # all TCP sockets with state (no name resolution)
ss -tn state established           # only established connections
ss -tn dst 10.0.0.5                 # connections to a specific host
ss -s                              # summary counts by state

Netid State   Recv-Q Send-Q Local Address:Port  Peer Address:Port  Process
tcp   LISTEN  0      128    0.0.0.0:22           0.0.0.0:*          users:(("sshd",pid=812,fd=3))
tcp   ESTAB   0      0      192.168.1.42:22      192.168.1.10:53344 users:(("sshd",pid=1099,fd=4))

Use this to answer “is my service actually listening?” (look for LISTEN on the expected port) and “who is connected?” Watch TCP states: many sockets stuck in SYN-SENT means the far end isn’t answering (firewall/down); many in TIME-WAIT is usually normal churn; many in CLOSE-WAIT means the local app isn’t closing sockets (an app bug).

Windows equivalent: netstat -ano (the -o shows the PID; cross-reference with Task Manager).

arp — MAC ↔ IP mapping on the local segment

ip neigh show          # modern Linux
arp -a                 # traditional, also on Windows/macOS

192.168.1.1 dev eth0 lladdr aa:bb:cc:dd:ee:ff REACHABLE
192.168.1.10 dev eth0 lladdr 11:22:33:44:55:66 STALE

Two devices showing the same MAC for different IPs, or a gateway IP whose MAC keeps changing, points to an IP conflict or ARP spoofing. INCOMPLETE next to a neighbor means ARP got no reply — the host is off or on a different VLAN.

Scanning & discovery

Authorized use only. Port scanning networks you do not own or lack written permission to test is illegal in many jurisdictions and against most acceptable-use policies. Scan only your own lab or systems you are explicitly authorized to assess.

nmap — host and service discovery

# Host discovery only (ping sweep) — who is alive on this subnet?
nmap -sn 192.168.1.0/24

Nmap scan report for 192.168.1.1
Host is up (0.0012s latency).
Nmap scan report for 192.168.1.42
Host is up (0.00030s latency).
Nmap done: 256 IP addresses (2 hosts up) scanned in 2.11 seconds

# Common TCP ports on one host
nmap 192.168.1.10

# Service + version detection on specific ports
nmap -sV -p 22,80,443 192.168.1.10

# Fast top-100 ports, no DNS, aggressive timing
nmap -F -n -T4 192.168.1.10

# OS detection (needs root/admin)
sudo nmap -O 192.168.1.10

# UDP scan (slow — many services don't reply)
sudo nmap -sU -p 53,123,161 192.168.1.10

Reading it: each open port shows PORT STATE SERVICE VERSION. open = something is listening; filtered = a firewall is dropping probes (no answer); closed = host is up but nothing listens there. -sV grabs banners to identify the exact software and version — invaluable for “what is running on that mystery box?”

Throughput: iperf3 and speedtest

speedtest measures your internet download/upload to a public server:

speedtest              # speedtest-cli or the official Ookla client

iperf3 measures throughput between two hosts you control — the real tool for testing a LAN, WAN link, or VPN tunnel. Run a server on one end and a client on the other:

# On the server (destination):
iperf3 -s

# On the client (source):
iperf3 -c 10.0.0.5                 # TCP throughput for 10 seconds
iperf3 -c 10.0.0.5 -t 30 -P 4      # 30s, 4 parallel streams (saturate the link)
iperf3 -c 10.0.0.5 -R             # reverse: measure download instead of upload
iperf3 -c 10.0.0.5 -u -b 100M      # UDP at 100 Mbit/s — reports jitter and loss

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec   0    sender
[  5]   0.00-10.00  sec  1.09 GBytes   940 Mbits/sec        receiver

943 Mbits/sec on a “gigabit” link is healthy (overhead accounts for the rest). A high Retr (retransmit) count signals loss or a bottleneck. UDP mode is the way to measure jitter and packet loss for VoIP/video planning.

Packet analysis

When higher-level tools disagree with reality, capture the actual packets. Both tcpdump and Wireshark use the same libpcap engine and the same capture-filter syntax (BPF); Wireshark adds a far richer display-filter language and a GUI.

Capture filters vs display filters

Rule of thumb: use a capture filter to keep the file small on a busy interface; use a display filter to explore what you captured. Never over-filter at capture time — you can’t recover what you didn’t record.

tcpdump — capture from the CLI

sudo tcpdump -i eth0 -n host 8.8.8.8            # traffic to/from one host, no DNS
sudo tcpdump -i eth0 -n port 53                  # all DNS
sudo tcpdump -i eth0 -n 'tcp port 443 and host 10.0.0.5'
sudo tcpdump -i any -n icmp                       # pings across all interfaces
sudo tcpdump -i eth0 -nn -c 100 -w capture.pcap   # write 100 packets to a file for Wireshark
sudo tcpdump -i eth0 -A -s0 port 80               # print packet payloads as ASCII (plaintext HTTP)
sudo tcpdump -r capture.pcap 'tcp port 443'       # read and filter an existing file

A TCP handshake in tcpdump output looks like this:

10:00:01.100 IP 10.0.0.10.53344 > 10.0.0.5.443: Flags [S],  seq 100, win 64240   # SYN
10:00:01.112 IP 10.0.0.5.443 > 10.0.0.10.53344: Flags [S.], seq 900, ack 101      # SYN-ACK
10:00:01.112 IP 10.0.0.10.53344 > 10.0.0.5.443: Flags [.],  ack 901               # ACK

[S] = SYN, [S.] = SYN-ACK, [.] = ACK, [P.] = PSH+ACK (data), [F.] = FIN, [R] = RST. Seeing a SYN with no SYN-ACK in reply means the server never answered — firewall or the service is down. A SYN answered by an [R] (RST) means the port is closed / actively refused.

Wireshark — the analysis GUI

Wireshark decodes hundreds of protocols and lets you follow a conversation visually. Essential display filters:

ip.addr == 10.0.0.5                     # any packet to/from this IP
tcp.port == 443                         # this TCP port
dns                                     # only DNS
http.request                            # HTTP requests only
tcp.flags.syn == 1 && tcp.flags.ack == 0   # SYNs (connection attempts)
tcp.analysis.retransmission             # Wireshark-flagged retransmits (loss!)
tcp.analysis.zero_window                # receiver told sender to stop (flow-control stall)
tcp.stream eq 0                         # isolate one whole TCP conversation
frame contains "password"               # find a byte string in payloads

Right-click a packet → Follow → TCP Stream to see an entire conversation reassembled. Statistics → Conversations ranks who is talking the most (find the bandwidth hog). Statistics → I/O Graph plots throughput over time. When you see tcp.analysis.retransmission piling up, you have real packet loss — correlate it with the mtr hop that showed loss.

Network simulators & labs

Practice and reproduce faults without risking production. Each tool trades fidelity for weight:

Rule of thumb: Packet Tracer to learn concepts cheaply, GNS3 when you need real firmware behavior on your own machine, EVE-NG for shared, large-scale, or multi-vendor labs.

Common issues: symptom → likely cause → tool

Diagnosing an MTU mismatch (worked example)

MTU problems are sneaky: small packets (ping, DNS, the start of a TCP session) work, but large transfers hang. Find the path MTU with a don’t-fragment ping and shrink until it passes:

ping -M do -s 1472 -c 2 example.com   # 1472 payload + 28 header = 1500 (standard Ethernet)
# If you get: "Frag needed and DF set (mtu = 1420)" -> the path MTU is 1420, not 1500
ping -M do -s 1392 -c 2 example.com   # 1392 + 28 = 1420, should now succeed

The largest -s value that succeeds, plus 28, is your path MTU. A common fix on VPN/PPPoE links is lowering the interface MTU or clamping TCP MSS.

Quick command cheat-sheet (Linux ↔ Windows)

Best Practices

• Have a method, not a habit. Pick bottom-up, top-down, or divide-and-conquer on purpose based on the symptom — don’t just reflexively ping. Walking the stack guarantees progress; guessing does not.
• Change one thing at a time, and log it. Keep a running note of every command, its output, and every change. If a fix works, you know why; if it doesn’t, you can cleanly revert.
• Reproduce and isolate before you theorize. “One host or all hosts? One direction or both? One app or everything?” answered up front eliminates most of the search space instantly.
• Prove connectivity by IP before blaming anything else. ping 8.8.8.8 vs ping name is the fastest first cut: it separates the network from DNS in two seconds.
• Test both directions. Firewalls and routing are often asymmetric. A capture on each end (SYN out, SYN-ACK back?) beats guessing.
• Prefer dig and query multiple resolvers. When DNS is suspect, compare two resolvers and use +trace to find exactly where the chain breaks.
• Capture with a filter, analyze with a filter. Use a narrow BPF capture filter on busy links to keep files small; use Wireshark display filters to explore — but never filter so tightly at capture time that you lose the evidence.
• Distinguish cosmetic ICMP loss from real loss. In mtr/traceroute, mid-path * * * or loss that clears by the final hop is usually router ICMP deprioritization, not a fault. Real loss persists to the destination.
• Scan only what you’re authorized to. nmap against networks you don’t own or lack written permission for can be illegal. Keep it to your lab and approved targets.
• Lab your changes first. Reproduce complex topologies in GNS3/EVE-NG/Packet Tracer before touching production; verify a config change in the lab, then deploy.
• Verify from the user’s seat, then verify you broke nothing. Confirm the original symptom is gone and that adjacent services still work before you close the ticket.
• Feed findings back into monitoring. Every incident should leave behind a new alert or dashboard so the next occurrence is caught by ./14-monitoring-and-observability.md, not by a user.

References

• ping(8) man page
• traceroute(8) man page
• dig(1) man page
• ss(8) man page
• tcpdump manual & BPF filters
• Wireshark User’s Guide
• Nmap Reference Guide
• iperf3 documentation
• BIND / dig — ISC documentation