Observability (Khả năng quan sát)Observability

Part of the Cloud knowledge base — AWS deep dive. Complements DevOps Roadmap.

Overview

Observability is the discipline of understanding what a running system is doing from the outside — through the metrics, logs, and traces it emits — so you can answer questions you did not anticipate when you built it. On AWS the center of gravity is CloudWatch, which collects metrics and logs from nearly every service, evaluates alarms, and renders dashboards. Around it sit specialized services: CloudTrail for the audit trail of who did what, AWS Config for the configuration history and compliance of your resources, X-Ray for distributed traces, and ADOT (AWS Distro for OpenTelemetry) as the vendor-neutral collector that ties it all together.

The important mental shift for a DevOps engineer is the difference between monitoring and observability. Monitoring answers pre-defined questions (“is CPU above 80%?”) with dashboards and alarms you built in advance. Observability is the property that lets you ask new questions of a system in production without shipping new code — “why are only Canadian users on the checkout path seeing 500s in the last nine minutes?” You get there by emitting high-cardinality, structured telemetry (dimensioned metrics, JSON logs, correlated traces) rather than scalar counters and free-text log lines.

The governing trade-off is signal vs. cost. CloudWatch bills for ingested log data, custom metrics, dashboards, and alarm evaluations, and a careless setup — DEBUG logs from every Lambda, a custom metric per user ID — produces bills that dwarf the compute they observe. The craft is capturing enough signal to debug and alert well while ruthlessly controlling volume, cardinality, and retention. This page covers the AWS building blocks and the practices that keep them useful and affordable.

Fundamentals

The three pillars on AWS

Metrics are cheap to store and fast to query but low-cardinality — good for alerting and trend dashboards. Logs are rich and high-cardinality but expensive to store and slow to scan — good for forensic detail. Traces stitch a single request across many services so you can see latency and failure propagate through a call graph. A mature setup uses all three and, crucially, correlates them (a trace ID in your logs, a log link on a metric alarm) so you can pivot from “the alarm fired” to “here is the exact request” in seconds.

CloudWatch metrics model

A CloudWatch metric is uniquely identified by namespace + name + a set of dimensions. AWS/EC2 → CPUUtilization → {InstanceId=i-0abc} is one time series; a different InstanceId is a different series. Each unique dimension combination is a separately-billed custom metric (for custom namespaces), which is why cardinality is the cost lever.

• Namespaces — a container for metrics. AWS services use AWS/<Service> (e.g. AWS/Lambda); your own use anything else (e.g. MyApp/Checkout).
• Dimensions — up to 30 name/value pairs that scope a metric. More dimensions = more series = finer slicing but higher cost.
• Resolution — standard metrics are 1-minute granularity; high-resolution custom metrics go down to 1 second (more expensive).
• Statistics — you never store raw points forever; you query aggregations (Average, Sum, p99, Maximum) over a period. Percentiles (p90, p99) matter far more than averages for latency.

Logs model

CloudWatch Logs organizes data as log group → log stream → log event. A log group is the unit of retention and access control (e.g. /aws/lambda/checkout); a stream is a sequence of events from one source (one Lambda instance, one container). Retention is set per group — the default is never expire, which is the single most common source of surprise CloudWatch bills. Emitting structured JSON logs (rather than free text) is the unlock for querying, because Logs Insights can then filter and aggregate on fields.

Key Concepts

CloudWatch alarms

An alarm watches one metric (or a metric math expression / anomaly detection band) and transitions between OK, ALARM, and INSUFFICIENT_DATA based on a threshold evaluated over N periods. Alarm actions can notify (SNS → email/Slack/PagerDuty), auto-scale (ASG policies), or self-heal (EC2 recover/reboot).

Key knobs:

• Period + Evaluation periods + Datapoints to alarm — “3 of 5 five-minute periods above threshold” avoids flapping on a single spike.
• Treat missing data — notBreaching, breaching, ignore, or missing; wrong choice here causes false alarms or silent blind spots.
• Composite alarms — combine child alarms with AND/OR so you page only when several conditions hold (e.g. high latency AND high error rate), cutting alert noise.
• Anomaly detection — CloudWatch trains a band of expected values; you alarm on deviation instead of a fixed threshold, which suits metrics with daily/weekly seasonality.

Worked example — a p99 latency alarm (CLI):

aws cloudwatch put-metric-alarm \
  --alarm-name "checkout-p99-latency-high" \
  --alarm-description "Checkout p99 latency > 500ms for 3 of 5 minutes" \
  --namespace "AWS/ApplicationELB" \
  --metric-name "TargetResponseTime" \
  --dimensions Name=LoadBalancer,Value=app/checkout-alb/50dc6c495c0c9188 \
  --extended-statistic "p99" \
  --period 60 \
  --evaluation-periods 5 \
  --datapoints-to-alarm 3 \
  --threshold 0.5 \
  --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:us-east-1:111122223333:oncall-pager

Two details make this good: it alarms on p99, not average (averages hide the tail of slow requests users actually feel), and it requires 3 of 5 breaching minutes with notBreaching on gaps, so a single blip does not page the on-call at 3 a.m.

Custom metrics: the EMF trick

You can publish custom metrics with PutMetricData, but the API costs per call and adds latency to your request path. The better pattern for Lambda/containers is the Embedded Metric Format (EMF): you write a specially-structured JSON log line, and CloudWatch extracts metrics from it asynchronously at no per-call API cost. You get the log event and the metric from a single stdout write.

{
  "_aws": {
    "Timestamp": 1699900000000,
    "CloudWatchMetrics": [{
      "Namespace": "MyApp/Checkout",
      "Dimensions": [["Service", "Currency"]],
      "Metrics": [{ "Name": "OrderValue", "Unit": "None" }]
    }]
  },
  "Service": "checkout",
  "Currency": "CAD",
  "OrderValue": 42.50,
  "traceId": "1-6564abcd-..."
}

Keep the Dimensions list low-cardinality — never put a user ID or request ID in a metric dimension, because every unique value becomes a billed time series. High-cardinality identifiers belong in log fields (queried by Logs Insights), not metric dimensions.

CloudWatch Logs Insights

Logs Insights is a purpose-built query language for CloudWatch Logs — think “SQL-ish for logs.” It parses JSON fields automatically and supports filter, stats, sort, parse, and limit. You scan a chosen time range across one or more log groups and pay per GB scanned, so narrow the time window and pre-filter before aggregating.

Worked example — top 10 slowest requests with errors:

fields @timestamp, @message, duration, statusCode, path
| filter statusCode >= 500
| stats count() as errors, avg(duration) as avgMs, pct(duration, 99) as p99Ms by path
| sort errors desc
| limit 10

And a classic “find the needle” query using parse on unstructured logs:

fields @timestamp, @message
| parse @message "user=* action=* latency=*ms" as user, action, latency
| filter action = "checkout" and latency > 1000
| sort latency desc
| limit 20

The lesson: structured logs make the first query trivial; unstructured logs force you into parse gymnastics. Invest in JSON logging early.

CloudWatch dashboards & EventBridge

Dashboards are JSON-defined collections of widgets (metric graphs, single values, log tables, alarm status). Define them in IaC (Terraform aws_cloudwatch_dashboard / CloudFormation) so they are versioned and reproducible, not hand-drawn in the console. A good service dashboard leads with the RED/USE signals (Rate, Errors, Duration / Utilization, Saturation, Errors) at the top and drills down below.

EventBridge is the event bus that reacts to changes across AWS. It receives events (an EC2 state change, a CodePipeline stage, a custom app event, or a cron schedule), matches them against rules, and routes to targets (Lambda, SQS, SNS, Step Functions). For observability it is how you turn a CloudTrail/Config/GuardDuty event into an automated response — e.g. “when GuardDuty finds a compromised instance, isolate its security group.” EventBridge Scheduler also replaces old CloudWatch Events cron rules for scheduled jobs.

CloudTrail: the audit trail

CloudTrail records API activity in your account — who called what, when, from where, and whether it succeeded. It is your forensic and compliance backbone; when something changes unexpectedly, CloudTrail tells you which principal did it.

Best practice is one organization trail (created in the management account, applied to every member account) delivering to a central, locked-down S3 bucket in a separate logging account, with log file validation enabled so tampering is detectable. Data events are powerful but noisy and costly — scope them to the specific buckets/functions that matter (e.g. a bucket holding PII), not everything.

AWS Config: configuration state & compliance

Where CloudTrail records actions, AWS Config records state — a continuous snapshot and change history of every resource’s configuration — and evaluates that state against rules.

• Config rules — desired-state checks, either AWS-managed (e.g. s3-bucket-public-read-prohibited, encrypted-volumes) or custom (Lambda / Guard). Resources are flagged COMPLIANT or NON_COMPLIANT.
• Remediation — a non-compliant resource can trigger an automatic SSM Automation document (e.g. re-enable encryption, remove a public ACL).
• Conformance packs — a bundle of Config rules + remediations deployed as a single unit, mapped to a framework (PCI-DSS, CIS, HIPAA). Deploy via StackSets across the org for consistent guardrails.
• Configuration timeline — for any resource, see exactly how it looked at any past moment and what changed it, which is invaluable during incidents.

X-Ray: distributed tracing

X-Ray follows a request as it flows through your services, producing a trace made of segments (one per service) and subsegments (a DB call, an HTTP call). It renders a service map — a visual graph of your architecture with latency and error rates on each edge — so you can spot the one downstream dependency adding 300 ms or throwing errors.

• Instrumentation — the X-Ray SDK or, increasingly, OpenTelemetry via ADOT. API Gateway, Lambda, and App Mesh can emit traces with minimal code.
• Sampling — you never trace 100% of high-volume traffic; a sampling rule keeps a fixed rate (e.g. 1 req/sec + 5% of the rest) to control cost while still catching anomalies.
• Annotations vs. metadata — annotations are indexed key/values you can filter traces by (e.g. customerTier=premium); metadata is unindexed extra context.

AWS Distro for OpenTelemetry (ADOT)

OpenTelemetry (OTel) is the vendor-neutral, CNCF standard for generating and collecting metrics, logs, and traces with one set of SDKs and one wire protocol (OTLP). ADOT is AWS’s supported, security-patched distribution of the OTel Collector plus SDKs, with exporters pre-wired for CloudWatch, X-Ray, and Amazon Managed Service for Prometheus (AMP) / Grafana (AMG).

Why it matters: instrument once with OTel, and you can send telemetry to CloudWatch/X-Ray and to Prometheus/Grafana/Datadog/Jaeger without re-instrumenting when you change backends. ADOT runs as a sidecar (ECS/EKS), a Lambda layer, or a standalone collector deployment, receiving OTLP and fanning out to your chosen backends. For any greenfield instrumentation in 2026, prefer OTel/ADOT over the proprietary X-Ray SDK to avoid lock-in.

Container Insights

Container Insights is CloudWatch’s turnkey observability for ECS, EKS, and Fargate: it collects cluster-, node-, pod-, and container-level metrics (CPU, memory, network, disk) plus performance logs, and ships curated dashboards. The enhanced observability version adds granular per-container and control-plane metrics. On EKS it is typically deployed via the CloudWatch Observability EKS add-on (which bundles the ADOT/Fluent Bit collectors). It answers “which pod is throttling CPU?” and “is this node memory-saturated?” without you building the pipeline by hand. Lambda Insights is the equivalent for functions (cold starts, memory, duration).

Best Practices

1. Set a retention policy on every log group. Never leave the default “never expire.” Rationale: unbounded log retention is the number-one surprise CloudWatch bill; match retention to actual investigative and compliance needs (e.g. 30 days hot, archive to S3/Glacier beyond).
2. Emit structured JSON logs, not free text. Rationale: Logs Insights can filter and aggregate on fields directly, turning multi-minute parse archaeology into one-line queries and enabling EMF metrics for free.
3. Alarm on percentiles (p95/p99), not averages. Rationale: averages mask the slow tail that real users experience; a healthy average can hide 1% of requests timing out.
4. Require multiple datapoints and set “treat missing data” deliberately. Use “M of N” evaluation and choose notBreaching/breaching per metric. Rationale: single-datapoint alarms flap and cause pager fatigue; the wrong missing-data policy causes false alarms or silent blind spots.
5. Keep metric dimensions low-cardinality; put IDs in log fields. Rationale: every unique dimension combination is a separately billed time series — a user-ID dimension can explode into millions of metrics and a huge bill.
6. Use EMF instead of PutMetricData on the request path. Rationale: EMF derives metrics from a single log write with no synchronous API call, removing latency and per-call cost from your hot path.
7. Route alarms through SNS to a real paging system, and use composite alarms. Rationale: email alone gets ignored; composite alarms (latency AND errors) fire only on genuine incidents, cutting noise while preserving coverage.
8. Define dashboards and alarms in IaC. Rationale: console-drawn dashboards are unversioned and vanish with the person who made them; Terraform/CloudFormation makes observability reproducible and reviewable.
9. Enable an organization CloudTrail to a locked, central S3 bucket in a separate account. Turn on log-file validation and block public access. Rationale: a tamper-evident, out-of-band audit log is essential for incident forensics and compliance, and centralizing it prevents a compromised account from deleting its own tracks.
10. Scope CloudTrail data events narrowly. Enable them only on sensitive buckets/functions. Rationale: data events are high-volume and billed per event; blanket enablement can cost more than the workload itself with little added value.
11. Deploy AWS Config with conformance packs and auto-remediation. Rationale: continuous compliance evaluation plus automatic remediation catches drift (public buckets, unencrypted volumes) in minutes instead of at the next audit.
12. Sample traces; don’t trace 100% of high traffic. Rationale: full-fidelity tracing at scale is expensive and rarely necessary; a sampling rule keeps costs bounded while still surfacing latency and error anomalies.
13. Correlate the three pillars with a trace/request ID. Put the X-Ray/trace ID in every log line and link logs from alarms. Rationale: correlation is what turns “an alarm fired” into “here is the exact failing request” in seconds instead of hours.
14. Standardize on OpenTelemetry/ADOT for new instrumentation. Rationale: instrument-once, export-anywhere avoids backend lock-in and lets you change or add observability vendors without touching application code.
15. Turn on Container Insights (and Lambda Insights) for managed compute. Rationale: curated cluster/pod/function dashboards catch resource saturation and cold-start regressions you would otherwise miss, with almost no setup.
16. Watch the observability bill itself. Track CloudWatch/CloudTrail/Config spend as a metric and alarm on anomalies. Rationale: telemetry cost can quietly exceed compute cost; treating it as a first-class budget line keeps a debug-log left on in prod from bankrupting the team.

References

• Amazon CloudWatch User Guide
• CloudWatch Logs Insights query syntax
• CloudWatch Embedded Metric Format (EMF)
• Creating CloudWatch alarms
• Amazon EventBridge User Guide
• AWS CloudTrail User Guide
• AWS Config Developer Guide
• AWS Config conformance packs
• AWS X-Ray Developer Guide
• AWS Distro for OpenTelemetry
• CloudWatch Container Insights
• AWS Well-Architected — Operational Excellence Pillar