Giám sát & ObservabilityMonitoring & Observability

Part of the Network Engineer Roadmap knowledge base.

Overview

You cannot operate what you cannot see. Network monitoring and observability turn a black box of cables, switches, routers, firewalls, and links into a system whose health, performance, and behavior you can reason about in near real time. Monitoring answers the questions you already know to ask (“is the link up? what is the CPU on the core switch? how much bandwidth is that VLAN using?”). Observability goes further: it aims to let you answer questions you did not anticipate ahead of time, by exposing rich, high-cardinality telemetry that you can slice and correlate after the fact.

Why invest in this at all?

• Detect and diagnose faults fast. Mean Time To Detect (MTTD) and Mean Time To Repair (MTTR) dominate the user-perceived reliability of a network. Good telemetry cuts both.
• Understand performance and capacity. Latency creep, buffer bloat, microbursts, and link saturation are invisible until measured. Trend data drives capacity planning and budget justification.
• Security and forensics. Flow records and logs reveal exfiltration, scanning, DDoS, and lateral movement, and provide the audit trail after an incident.
• Prove SLAs / SLOs. Contracts with providers and internal service owners need objective, historical evidence.
• Feed automation. Closed-loop remediation and intent-based networking depend on a trustworthy stream of state.

There are two classic disciplines that observability spans:

This note maps the general “three pillars” of observability onto networking, then works through the core telemetry technologies — SNMP, flow export (NetFlow/sFlow/IPFIX), syslog, and streaming telemetry (gNMI) — and the stacks that store and visualize them (Prometheus/Grafana, and commercial platforms like Datadog and Dynatrace). It closes with alerting, baselining, SLIs/SLOs, and capacity planning. For hands-on packet-level diagnosis see ./15-tools-and-troubleshooting.md; for the protocols you are watching see ./06-core-protocols.md; and for the queueing/marking behavior these metrics reflect see ./13-traffic-management-and-qos.md.

Fundamentals

The three pillars, adapted to networking

The software-observability world talks about metrics, logs, and traces. Networking has direct analogs plus one extra pillar of its own:

Flows are the pillar unique to networking. A single flow record summarizes an entire conversation (5-tuple + byte/packet counts + timing), which is far cheaper than capturing every packet but far richer than a bandwidth gauge.

Push vs pull

• Pull (polling): the collector periodically asks devices for state. SNMP GET and Prometheus scraping are pull models. Simple, but resolution is limited by poll interval (often 30–300 s) and the collector must know every target.
• Push (streaming/events): the device sends data when it changes or on a fine cadence. SNMP traps, syslog, NetFlow export, and gNMI dial-out are push models. Lower latency, higher resolution, better for scale — but needs the device to be configured to talk to the right collector.

Modern designs blend both: pull metrics for steady-state health, push events/telemetry for change and high-resolution data.

Cadence, cardinality, and retention

Three knobs shape every monitoring system:

• Cadence / resolution — how often you sample (1 s streaming vs 5 min SNMP). Higher resolution catches microbursts but costs storage and load.
• Cardinality — how many distinct time series (per-interface × per-device × per-metric). This explodes quickly; a chassis with 400 ports × 20 counters = 8000 series on one device.
• Retention — how long you keep raw vs rolled-up (downsampled) data. Typical: raw for days, 5-min rollups for weeks, hourly for a year+.

Key Concepts

SNMP (Simple Network Management Protocol)

SNMP is the decades-old workhorse for polling device state. Despite its age it remains the lowest-common-denominator supported by virtually every managed device.

Architecture

• Manager (NMS): the polling/collecting station (e.g., LibreNMS, Zabbix, PRTG, or an SNMP exporter feeding Prometheus).
• Agent: software on the managed device that exposes data and can emit notifications.
• MIB (Management Information Base): a hierarchical, tree-structured schema of manageable objects. Vendors publish MIBs (e.g., IF-MIB, BGP4-MIB, CISCO-PROCESS-MIB).
• OID (Object Identifier): a dotted-decimal address of one object in the MIB tree. Example: 1.3.6.1.2.1.2.2.1.10 = ifInOctets (bytes received on an interface).

Operations

Polling works by the manager issuing GET/GETBULK against a list of OIDs on a schedule (e.g., every 60 s). To compute a rate (bps) you poll a counter twice and divide the delta by the interval. Watch for counter wrap: 32-bit counters (ifInOctets) wrap fast on 10G+ links — always use the 64-bit high-capacity counters (ifHCInOctets, OID 1.3.6.1.2.1.31.1.1.1.6).

Versions & security

Prefer SNMPv3 authPriv wherever the device and NMS support it; fall back to v2c only on isolated management networks.

Example — enabling SNMP on Cisco IOS (v2c read-only, then v3):

! SNMPv2c read-only, restricted by ACL to the NMS
access-list 99 permit host 10.0.0.10
snmp-server community R3adOnly-x8f2 RO 99
snmp-server location DC1-Row4-Rack7
snmp-server contact netops@example.com
snmp-server host 10.0.0.10 version 2c R3adOnly-x8f2

! SNMPv3 — group, view, user (authPriv, SHA + AES-128)
snmp-server view ALL iso included
snmp-server group MONgrp v3 priv read ALL
snmp-server user monitor MONgrp v3 auth sha S3cret-Auth-Pass priv aes 128 S3cret-Priv-Pass
snmp-server enable traps
snmp-server host 10.0.0.10 version 3 priv monitor

Query it from a Linux collector:

# v2c walk of the interface table
snmpwalk -v2c -c 'R3adOnly-x8f2' 10.0.0.1 IF-MIB::ifTable

# v3 authPriv GET of a single counter
snmpget -v3 -l authPriv -u monitor \
  -a SHA -A 'S3cret-Auth-Pass' \
  -x AES -X 'S3cret-Priv-Pass' \
  10.0.0.1 IF-MIB::ifHCInOctets.2

Flow-based monitoring — NetFlow, sFlow, IPFIX

Where SNMP tells you how much traffic crossed an interface, flow monitoring tells you what that traffic was: source/destination IPs, ports, protocol, ToS/DSCP, AS numbers, byte/packet counts, and timing. This is the foundation of traffic analysis, billing, peering decisions, and security detection (DDoS, scanning, exfiltration).

How flow export works (NetFlow/IPFIX)

1. The router/switch maintains a flow cache, keyed by the flow definition (classically the 5-tuple: src IP, dst IP, src port, dst port, protocol; plus ingress interface, ToS).
2. Packets update the matching cache entry’s counters.
3. A flow is expired and exported when it ends (TCP FIN/RST), goes idle (idle timeout), lives too long (active timeout, e.g., 60 s), or the cache fills.
4. Expired flows are packed into UDP datagrams and sent to a flow collector (e.g., nfdump/nfsen, Elastiflow, ntopng, Kentik).

Sampling. On high-speed links, tracking every flow is expensive, so devices sample 1-in-N packets (e.g., 1:1000). Sampling is essential at scale but means you must scale counts back up and lose visibility into very small flows.

Templates. NetFlow v9 and IPFIX are template-based: the exporter first sends a template describing the fields, then data records referencing it. This makes them extensible (unlike fixed NetFlow v5). IPFIX also supports variable-length fields and enterprise-specific elements (e.g., URLs, application IDs).

sFlow works differently: instead of a flow cache it does packet sampling — it grabs the first N bytes (header) of every Nth packet and ships that sample immediately, plus periodic interface counters. This is stateless on the device (cheap, hardware-friendly, real-time) but statistical rather than a full accounting of every conversation.

Comparison:

Rule of thumb: IPFIX/NetFlow v9 when you want accurate accounting and rich fields; sFlow when you want cheap, real-time, hardware-based visibility across many high-speed ports.

Example — Flexible NetFlow (IPFIX-style) on Cisco IOS:

flow record RECORD-v4
 match ipv4 source address
 match ipv4 destination address
 match transport source-port
 match transport destination-port
 match ipv4 protocol
 collect counter bytes
 collect counter packets
 collect interface input

flow exporter EXP-1
 destination 10.0.0.20
 transport udp 2055
 export-protocol ipfix
 template data timeout 60

flow monitor MON-1
 record RECORD-v4
 exporter EXP-1
 cache timeout active 60

interface GigabitEthernet0/1
 ip flow monitor MON-1 input
 ip flow monitor MON-1 output

Syslog and log management

Syslog (RFC 5424, older RFC 3164) is the standard for device event logging: link flaps, OSPF/BGP adjacency changes, config changes, ACL/firewall denies, authentication events, hardware alarms. Logs are the narrative record that explains why a metric moved.

Severity levels (0 = worst). A useful mnemonic: Every Awesome Cisco Engineer Will Need Ice-cream Daily.

Facilities categorize the source (e.g., local0–local7, auth, kern). A message combines facility + severity into a priority.

Ship logs to a central collector — rsyslog/syslog-ng, an ELK/OpenSearch stack, Graylog, Loki, or a SIEM (Splunk, Elastic Security) for correlation and retention. Centralization matters because a device that crashes takes its local logs with it, and cross-device correlation (a flap here, an error there) needs everything in one place.

Example — Cisco IOS to a syslog server, timestamped and level-filtered:

service timestamps log datetime msec localtime show-timezone
logging host 10.0.0.30 transport udp port 514
logging trap informational      ! send severity 6 and worse
logging source-interface Loopback0
logging buffered 64000 debugging

Always sync clocks with NTP first — logs without accurate, consistent timestamps are nearly useless for correlation.

Streaming telemetry (model-driven, gNMI)

SNMP polling has real limits: coarse resolution (you can’t poll thousands of OIDs every second), high overhead (request/response per object), and rigid MIB structures. Model-driven streaming telemetry is the modern answer.

• The device pushes data continuously (subscribe once, stream forever) instead of being polled.
• Data is structured by YANG models (open standards like OpenConfig, or vendor models) — the same models used for config, so monitoring and configuration share one schema.
• Transport is typically gNMI (gRPC Network Management Interface) over HTTP/2 with TLS; encoding is Protobuf or JSON. NETCONF/RESTCONF can also carry telemetry.
• Modes: dial-out (device connects to collector) or dial-in (collector subscribes to device); sample (fixed interval, e.g., 1 s) or on-change (event-driven, e.g., a BGP state change).

Result: sub-second resolution, lower device CPU per data point, and a consistent data model. Collect with a telemetry pipeline — Telegraf (with the gNMI/cisco_telemetry inputs), Cisco’s Pipeline, or gnmic — landing into a TSDB (InfluxDB, Prometheus) for Grafana.

Example — OpenConfig gNMI subscription on Cisco IOS-XR (dial-in path), plus a gnmic subscribe:

! Device side (IOS-XR) — enable gRPC/gNMI
grpc
 port 57400
 tls-mutual
!
telemetry model-driven
 sensor-group IFCOUNTERS
  sensor-path openconfig-interfaces:interfaces/interface/state/counters

# Collector side — subscribe at 1s sample interval
gnmic -a 10.0.0.1:57400 --skip-verify -u admin -p '***' \
  subscribe --path "openconfig-interfaces:interfaces/interface/state/counters" \
  --mode stream --stream-mode sample --sample-interval 1s

Metrics & time-series stack — Prometheus + Grafana

The open-source de-facto standard for storing and visualizing metrics.

• Prometheus — a pull-based TSDB that scrapes HTTP /metrics endpoints, stores dimensional time series (metric name + labels), and provides PromQL for querying and alerting.
• Exporters bridge non-native sources into Prometheus:
  • snmp_exporter — Prometheus scrapes it over HTTP; it in turn SNMP-polls network devices using a generated snmp.yml (built from MIBs by the generator). This is how you get switch/router SNMP data into Prometheus.
  • node_exporter — host-level metrics (CPU, disk, NIC counters) for Linux servers/collectors.
  • Others: blackbox_exporter (ICMP/TCP/HTTP probes for reachability & latency), gnmi via Telegraf → remote_write.
• Grafana — dashboards over Prometheus (and InfluxDB, Loki, etc.): time-series panels, heatmaps, topology, threshold coloring, and its own alerting.
• Alertmanager — dedupes, groups, silences, and routes Prometheus alerts to email/Slack/PagerDuty.

Example — Prometheus scraping the SNMP exporter for two switches:

# prometheus.yml
scrape_configs:
  - job_name: 'snmp-switches'
    static_configs:
      - targets:
          - 10.0.0.1      # core-sw-1
          - 10.0.0.2      # core-sw-2
    metrics_path: /snmp
    params:
      module: [if_mib]          # module defined in snmp.yml
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9116   # the snmp_exporter host:port

  - job_name: 'node'
    static_configs:
      - targets: ['10.0.0.30:9100']    # collector host node_exporter

Example PromQL — interface throughput in bits/sec and an error rate:

# Ingress throughput (bps) per interface, 5-min rate
rate(ifHCInOctets[5m]) * 8

# Interfaces with a rising input-error rate
rate(ifInErrors[5m]) > 0

Commercial observability platforms

Open-source stacks are powerful but need assembly and operation. Commercial SaaS platforms trade cost for turnkey breadth, correlation, and support.

They fit best when you value time-to-value, cross-domain correlation (network ↔ application ↔ cloud), and don’t want to run the pipeline yourself; open source fits when you need control, custom models, or want to avoid per-host/per-flow pricing.

Packet analysis for deep inspection

Metrics, flows, and logs tell you that and roughly what; when you need to know exactly why at the byte level, capture packets. Wireshark/tshark and tcpdump decode protocol fields, expose retransmissions, malformed frames, TCP window issues, and TLS handshakes. This is the deepest and most expensive form of visibility — use it surgically, triggered by what your metrics/flows flagged. Covered in depth in ./15-tools-and-troubleshooting.md.

Alerting, baselining, SLIs/SLOs, and capacity planning

Baselining. Networks are seasonal (business hours, backups at night, month-end). A static threshold (“alert if >80% util”) produces false alarms and misses anomalies. Establish a baseline of normal per-interface/per-hour behavior, then alert on deviation from it (anomaly detection). Baselines also define what “healthy” means for SLOs.

Alerting principles.

• Alert on symptoms (user-facing impact: link down, loss, latency) more than causes; keep cause dashboards for diagnosis.
• Make every alert actionable — if there’s nothing to do, it shouldn’t page.
• Use severity tiers: page for urgent, ticket/notify for the rest.
• Add hysteresis / for-duration (for: 5m) to avoid flapping alerts.
• Reduce noise via grouping, deduplication, and dependency suppression (don’t page for 40 downstream devices when their upstream link is the real fault).

Example Prometheus alert:

groups:
  - name: network
    rules:
      - alert: InterfaceDown
        expr: ifOperStatus == 2 and ifAdminStatus == 1
        for: 2m
        labels: {severity: critical}
        annotations:
          summary: "Interface {{ $labels.ifName }} down on {{ $labels.instance }}"

      - alert: HighLinkUtilization
        expr: (rate(ifHCInOctets[5m])*8 / ifHighSpeed / 1e6) > 0.85
        for: 10m
        labels: {severity: warning}
        annotations:
          summary: "Link >85% utilized on {{ $labels.instance }} {{ $labels.ifName }}"

SLIs / SLOs for networks. Borrow SRE practice:

• SLI (indicator): a measured quantity — e.g., ”% of 1-min windows with packet loss < 0.1% on the WAN,” or “p95 round-trip latency between DC-A and DC-B.”
• SLO (objective): the target — e.g., “99.9% of the month, RTT ≤ 20 ms and loss ≤ 0.05%.”
• Error budget: the allowed shortfall (0.1% of the month), used to balance change velocity vs stability.

Typical network SLIs: availability (reachability via synthetic probes/BFD), latency, jitter, loss, and throughput — measured with active probes (IP SLA, TWAMP, blackbox_exporter, ThousandEyes) rather than only device counters, so you measure the user’s path.

Capacity planning. Use long-retention trend data to forecast when links, CPU, TCAM/FIB, or NAT tables will exhaust. Track the 95th-percentile utilization (the industry billing metric for transit), model growth, and provision before you cross ~70–80% sustained. Flow data additionally shows which applications or peers drive growth, informing peering and QoS decisions (see ./13-traffic-management-and-qos.md).

Tool / technique comparison

Best Practices

• Monitor the four golden signals for links: utilization, errors, discards/drops, and latency — plus availability. Errors and discards are early warning signs that precede outages.
• Use SNMPv3 authPriv on production; restrict SNMP with ACLs and a dedicated management VRF/network; never leave public/private community strings.
• Always use 64-bit (HC) interface counters on ≥1 Gbps links to avoid counter wrap; compute rates from deltas, not absolute values.
• Sync time everywhere with NTP before anything else — metrics, flows, logs, and traces are only correlatable with consistent timestamps.
• Centralize logs and flows; retain per compliance/forensic need; downsample metrics for long-term trends.
• Prefer streaming telemetry on capable platforms for high-resolution and on-change data; keep SNMP for coverage of everything else.
• Alert on symptoms, make alerts actionable, add for-durations, and suppress dependent alerts to fight fatigue.
• Baseline before you threshold; static thresholds on seasonal traffic create noise.
• Define network SLIs/SLOs with active probes so you measure the user’s experience, not just device counters, and track error budgets.
• Capacity-plan on 95th-percentile trends; act before ~70–80% sustained utilization; let flow data explain the growth.
• Secure the telemetry plane itself: encrypt (SNMPv3/TLS/gRPC), authenticate collectors, and treat flow/log data as sensitive (it reveals who talks to whom).
• Instrument the collector too (node_exporter, self-monitoring) — a blind monitoring system is worse than none.
• Keep dashboards purposeful: an overview per site/role for at-a-glance health, deep dashboards for diagnosis; avoid “wall of graphs” nobody reads.

References

• RFC 3416 — Version 2 of the Protocol Operations for SNMP
• RFC 3414 / RFC 3410 — SNMPv3 User-based Security Model & Introduction
• RFC 7011 — Specification of the IP Flow Information Export (IPFIX) Protocol
• RFC 3954 — Cisco Systems NetFlow Services Export Version 9
• RFC 3176 — InMon sFlow: Traffic Monitoring using Packet Sampling
• RFC 5424 — The Syslog Protocol
• Prometheus documentation and snmp_exporter
• Grafana documentation · OpenConfig / gNMI