ObservabilityObservability

Part of the Cloud knowledge base — GCP deep dive. Complements DevOps Roadmap.

Overview

Observability on Google Cloud is delivered through the Cloud Operations suite (formerly Stackdriver): a set of integrated, mostly-managed services covering the three pillars — metrics (Cloud Monitoring), logs (Cloud Logging), and traces (Cloud Trace) — plus higher-level tooling for exception aggregation (Error Reporting) and continuous CPU/heap profiling (Cloud Profiler). For teams standardizing on Prometheus, Google Cloud Managed Service for Prometheus (GMP) offers a globally-scalable, drop-in backend, and everything integrates with OpenTelemetry as the vendor-neutral instrumentation standard.

The suite is enabled by default for most resources: GCE VMs, GKE, Cloud Run, and App Engine emit platform metrics and logs automatically, and Google’s infrastructure logs (audit, VPC flow, load balancer) flow into Cloud Logging without any agent. You pay mostly for ingestion and retention volume, not for a running cluster.

Fundamentals

Cloud Monitoring — metrics

A metric is a monitored resource + a metric type producing a time series. Metric types are named like compute.googleapis.com/instance/cpu/utilization. Every time series carries resource labels (which VM, which zone) and metric labels (dimensions you slice by).

• Built-in metrics — emitted automatically by GCP services (CPU, disk, request count, Pub/Sub backlog, etc.), free of charge.
• Agent metrics — the Ops Agent on GCE VMs collects guest-level metrics (memory, disk, process, and third-party integrations like Nginx/PostgreSQL) and logs. The Ops Agent replaces the legacy Monitoring + Logging agents.
• Custom metrics — application metrics you write via the API, OpenTelemetry, or Prometheus. Billed per time series.

Uptime checks probe an HTTP(S)/TCP endpoint from multiple global locations and record availability and latency. They pair naturally with an alerting policy:

gcloud monitoring uptime create my-api-uptime \
  --resource-type=uptime-url \
  --resource-labels=host=api.example.com \
  --path=/healthz --port=443 --protocol=https \
  --period=60 --timeout=10

Metrics Scope lets one central “scoping project” observe metrics from many monitored projects — the multi-project observability model.

Querying: MQL and PromQL

Cloud Monitoring supports two query languages:

• MQL (Monitoring Query Language) — Google’s native, pipe-based language.
• PromQL — supported directly against both native metrics and Managed Service for Prometheus data, so Grafana/Prometheus users feel at home.

# MQL: mean CPU utilization per instance, aligned to 1m
fetch gce_instance
| metric 'compute.googleapis.com/instance/cpu/utilization'
| align mean_aligner(1m)
| group_by [resource.instance_id], mean(val())

# PromQL equivalent flavor: p95 request latency by service
histogram_quantile(0.95,
  sum by (le, service_name) (
    rate(run_googleapis_com:request_latencies_bucket[5m])))

Cloud Logging

Log entries are structured JSON documents routed through the Log Router, which evaluates sinks to decide where each entry goes.

• Log buckets — storage containers for logs. Every project has _Default (user logs, 30-day retention) and _Required (admin/audit, 400-day, non-configurable, free) buckets. You can create custom buckets with custom retention (1–3650 days).
• Sinks — a filter + destination. Destinations: another log bucket, BigQuery (for SQL analytics), Cloud Storage (cheap long-term archive), or Pub/Sub (stream to third-party SIEM/Splunk).
• Log Analytics — upgrades a log bucket so you can run SQL (BigQuery engine) over log entries directly, without exporting.
• Log-based metrics — turn log matches into metrics: counter metrics (count matching entries) or distribution metrics (extract a numeric field into a histogram).

# Route all ERROR+ logs to BigQuery for analytics
gcloud logging sinks create errors-to-bq \
  bigquery.googleapis.com/projects/PROJECT/datasets/app_logs \
  --log-filter='severity>=ERROR'

# Grant the sink's writer identity permission on the dataset (returned by the command)

A logging query (Logging Query Language) filters entries:

resource.type="cloud_run_revision"
resource.labels.service_name="my-api"
severity>=WARNING
httpRequest.status>=500
timestamp>="2026-07-18T00:00:00Z"

A log-based metric built on top of that filter:

gcloud logging metrics create http_5xx_count \
  --description="Count of 5xx responses from my-api" \
  --log-filter='resource.type="cloud_run_revision"
    resource.labels.service_name="my-api" httpRequest.status>=500'

You then alert on logging.googleapis.com/user/http_5xx_count.

Cloud Trace

Cloud Trace collects latency data from distributed requests as spans grouped into traces. It auto-instruments App Engine and integrates with OpenTelemetry / instrumentation libraries elsewhere. It surfaces latency distributions, detects regressions between releases, and shows a request’s waterfall across services. There is no separate ingestion charge for the first large monthly tier; you export spans via the OTel exporter or the Cloud Trace API.

Error Reporting

Error Reporting automatically groups and deduplicates exceptions from your logs (stack traces in stderr/structured logs) into distinct error groups, tracks first/last-seen and occurrence counts, and can notify you when a new error type appears. It works out of the box for Cloud Run, App Engine, GKE, GCE (with agent), and Cloud Functions.

Cloud Profiler

Cloud Profiler is a statistical, always-on production profiler with negligible overhead. A small library in your app periodically samples CPU time, heap allocation, contention, and threads, and renders flame graphs in the console. Supported for Go, Java, Node.js, and Python. Use it to find hot functions and memory leaks under real traffic rather than in synthetic benchmarks.

Managed Service for Prometheus (GMP)

GMP is a Prometheus-compatible, globally-scalable managed backend built on Monarch (the same planet-scale time-series database behind Cloud Monitoring). You keep PromQL, Prometheus exposition format, and existing exporters, but Google runs storage and query at scale with no local Prometheus to operate.

• Managed collection — Google-run collectors on GKE scrape your PodMonitoring/ClusterPodMonitoring targets.
• Self-deployed collection — run the drop-in gmp binary and remote-write to GMP.
• Query with PromQL via the Prometheus HTTP API, Cloud Monitoring, or Grafana.

# GKE PodMonitoring CR — scrape pods labeled app=my-api on port "metrics"
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: my-api
spec:
  selector:
    matchLabels:
      app: my-api
  endpoints:
  - port: metrics
    interval: 30s

OpenTelemetry on GCP

OpenTelemetry (OTel) is the recommended, vendor-neutral way to instrument apps. GCP provides exporters for traces (→ Cloud Trace), metrics (→ Cloud Monitoring), and logs, plus the OpenTelemetry Collector with a googlecloud exporter for a centralized pipeline. Instrument once with OTel SDKs; route to Google backends today and stay portable tomorrow.

Key Concepts

Alerting policies

An alerting policy = one or more conditions + notification channels + optional documentation. Conditions can be metric-threshold, metric-absence, or (for logs) log-match. Modern policies express conditions as PromQL or MQL and use notification channels (email, PagerDuty, Slack, Pub/Sub, SMS, webhook).

Example policy that alerts when the 5xx log-based metric exceeds 5/min for 5 minutes:

# gcloud alpha monitoring policies create --policy-from-file=policy.yaml
displayName: "my-api high 5xx rate"
combiner: OR
conditions:
- displayName: "5xx > 5/min for 5m"
  conditionThreshold:
    filter: >
      metric.type="logging.googleapis.com/user/http_5xx_count"
      resource.type="cloud_run_revision"
    comparison: COMPARISON_GT
    thresholdValue: 5
    duration: 300s
    aggregations:
    - alignmentPeriod: 60s
      perSeriesAligner: ALIGN_RATE
notificationChannels:
- projects/PROJECT/notificationChannels/CHANNEL_ID
alertStrategy:
  autoClose: 1800s

SLOs and burn-rate alerting

Cloud Monitoring has native SLO support: define an SLI (availability or latency) against a service, set a target (e.g., 99.9% over 28 days), and Google computes the error budget. Burn-rate alerts fire when you consume budget too fast — a multi-window (fast + slow) burn-rate policy catches both acute outages and slow degradations while minimizing false pages.

Dashboards

Dashboards are collections of charts (line, stacked, heatmap, scorecard, gauge). Define them via the console, the API, or Terraform (google_monitoring_dashboard with a JSON body) so they are version-controlled. For teams wanting Grafana, Managed Service for Grafana and the Cloud Monitoring Grafana data source both work against the same metrics.

Log retention & cost model

You are billed for log ingestion (per GiB) and retention beyond defaults. The _Required bucket is free and immutable. Big savings come from exclusion filters (drop noisy, low-value logs at the router before they are stored/billed) and from routing archive copies to cheap Cloud Storage.

Best Practices

1. Install the Ops Agent on every GCE VM. Rationale: without it you get hypervisor-level metrics only — no memory, disk, or process visibility, and no guest logs. The single Ops Agent replaces the legacy split agents.

2. Instrument applications with OpenTelemetry, not vendor SDKs. Rationale: one instrumentation layer exports to Cloud Trace/Monitoring today and keeps you portable if backends change, avoiding costly re-instrumentation.

3. Define SLOs and alert on burn rate, not raw thresholds. Rationale: multi-window burn-rate alerts tie paging to user-visible error-budget consumption, cutting alert fatigue versus static CPU/latency thresholds.

4. Use multi-window, multi-condition alerting policies. Rationale: a fast-burn window catches acute incidents; a slow-burn window catches gradual degradation — together they reduce both missed incidents and false pages.

5. Prefer log-based metrics + metric alerts over raw log alerts for high-volume signals. Rationale: aggregating into a metric is cheaper to evaluate and gives you rate/percentile math and historical trends.

6. Aggressively exclude low-value logs at the Log Router. Rationale: ingestion is the main Logging cost; exclusion filters drop health-check and debug noise before it is billed, without touching the free _Required bucket.

7. Export audit/security logs to a dedicated, long-retention sink. Rationale: compliance often needs 1+ year; route to a custom bucket or BigQuery/GCS so you meet retention without paying premium hot-storage rates for everything.

8. Enable Log Analytics on buckets you query often. Rationale: running BigQuery-grade SQL directly on logs avoids a separate export pipeline and lets you join logs with other datasets.

9. Centralize observability with a Metrics Scope / logging aggregation. Rationale: a single scoping (or org-level log) project gives SRE one pane of glass across many projects instead of per-project silos.

10. Standardize on Managed Service for Prometheus for Kubernetes metrics. Rationale: keep PromQL and exporters while offloading storage/query scaling to Monarch — no self-managed Prometheus HA, sharding, or long-term storage to operate.

11. Turn on Error Reporting and alert on new error groups. Rationale: deduplicated, first-seen notifications surface regressions the moment a new exception type ships, far faster than scanning logs.

12. Run Cloud Profiler continuously in production. Rationale: statistical sampling has near-zero overhead and reveals real hot paths and leaks under production traffic that benchmarks miss.

13. Attach uptime checks to public endpoints and alert on them. Rationale: multi-region external probes catch DNS, TLS, and edge failures that internal metrics cannot see.

14. Manage dashboards and alerting policies as code (Terraform). Rationale: version control, review, and reproducibility beat click-ops; you can promote identical monitoring across environments.

15. Add resource and metric labels deliberately, but watch cardinality. Rationale: good labels enable slicing by service/version/tenant; unbounded labels (user IDs, request IDs) explode time-series count and cost.

16. Set log bucket retention explicitly per data class. Rationale: 30 days is fine for app debug logs but too short for security/audit; matching retention to purpose controls both compliance risk and spend.

References

• Cloud Operations (formerly Stackdriver) overview
• Cloud Monitoring documentation
• Alerting policies overview
• Monitoring Query Language (MQL)
• PromQL in Cloud Monitoring
• Uptime checks
• SLO monitoring
• Cloud Logging documentation
• Log Router & sinks
• Log-based metrics
• Log Analytics
• Logging Query Language
• Cloud Trace documentation
• Error Reporting documentation
• Cloud Profiler documentation
• Managed Service for Prometheus
• Ops Agent
• OpenTelemetry on Google Cloud