Cloud Data PlatformCloud Data Platforms

Part of the Data Engineer Roadmap knowledge base.

Overview

Almost every data platform built today is built on someone else’s servers. That is not a footnote — it is the defining operational fact of modern data engineering, and it is worth understanding why before cataloguing which service does what. The general mechanics of the three major clouds — IAM, networking, compute instances, block/object storage, managed relational databases — are already covered in ../cloud/README.md and its AWS/GCP deep dives; this note assumes that foundation and does not repeat it. What it covers instead is the layer that sits directly on top of that foundation and is specific to this roadmap: the data warehouses, lakes, and ETL/orchestration services that data engineers actually spend their days operating, plus Snowflake, the cloud-agnostic warehouse that has become a fourth platform in its own right alongside AWS, Azure, and GCP.

Three forces explain why cloud platforms displaced on-premises data warehouses so completely over the last decade. First, analytical workloads are inherently spiky: a nightly batch load or an end-of-quarter reporting rush needs far more compute than an average Tuesday afternoon, and a fixed on-prem cluster sized for the peak sits mostly idle the rest of the time — cloud elasticity lets compute scale up for the spike and back down (or to zero) afterward, so you pay closer to what you actually use. Second, and more architecturally significant, modern cloud warehouses separate storage from compute — a trait pioneered by Snowflake and now shared by BigQuery and modern Redshift (Redshift Serverless, and RA3 nodes with managed storage) — so that the two scale, and are billed, independently. You can hold petabytes of historical data cheaply in object storage while running a small compute cluster against it most of the time, then spin up much larger compute for a specific job, without ever having to move or re-shard the data itself. Third, managed services shift operational burden — patching, node failure recovery, backup, tuning, capacity planning — onto the cloud provider, so a team of a few data engineers can now run infrastructure that a decade ago required a dedicated on-prem database administration team.

Fundamentals

The separation of storage and compute

Understanding this one architectural idea unlocks most of what makes modern cloud warehouses behave the way they do, so it is worth dwelling on before touring individual services.

A classic on-premises MPP (massively parallel processing) appliance — think a Teradata or an early-generation Redshift cluster with dense-compute nodes — bundles storage and compute into the same physical nodes. Each node holds a shard of the data and the CPU/RAM that queries that shard. This is fast for well-distributed workloads, but it means storage and compute can only be scaled together: adding capacity to hold more historical data also means paying for more compute whether you need it or not, and resizing the cluster is a slow, often disruptive operation because data has to be redistributed across the new node count.

Separating the two layers — object storage (S3, ADLS, GCS, or Snowflake’s internal storage, all ultimately backed by the same kind of cheap, durable, effectively infinite blob storage) holds the data, while independent, ephemeral compute clusters attach to that storage on demand — breaks that coupling. Storage grows by simply writing more files, at cents per GB per month, with no compute implication at all. Compute is provisioned, resized, paused, and billed as a completely separate dimension, often per-second or per-query. This is what makes it economically sane to keep years of raw history “just in case” (storage is nearly free) while still controlling costs tightly on the compute side (pause it when nobody is querying, scale it up only for the workload that needs it). Snowflake built its entire architecture around this idea from day one; BigQuery took it further into a fully serverless model where you barely think about compute clusters at all; Redshift retrofitted it via RA3 node types and Redshift Serverless, moving away from its original tightly-coupled dense-compute design.

AWS data stack

AWS’s data services are best understood as a set of well-defined, individually-provisioned building blocks that a data engineer wires together, rather than one integrated product.

Amazon S3 is the foundation almost everything else in the AWS data stack sits on top of. It is the de facto standard data lake storage layer across the industry, not just AWS — cheap, durable object storage with virtually unlimited capacity, versioning, lifecycle policies to tier cold data into Glacier, and — critically for pipelines — the ability to emit an S3 event notification the instant an object is created, which is the trigger most AWS-native ingestion pipelines are built around.

Amazon Redshift is AWS’s MPP data warehouse, covered in architectural depth in ./06-data-modeling-and-warehousing.md (columnar storage, distribution styles, sort keys). Redshift now offers two operating modes: provisioned clusters (you choose node type and count, pay per node-hour, and — with RA3 nodes — get separated managed storage backed by S3) and Redshift Serverless (you specify a capacity range in Redshift Processing Units and it scales automatically, billed per second of actual usage). Redshift Spectrum lets a Redshift cluster query data sitting directly in S3 as external tables, without loading it first — a lightweight lake/warehouse bridge that predates the broader lakehouse trend.

RDS and Aurora are AWS’s managed OLTP databases (PostgreSQL, MySQL, and others) — the operational systems that generate the data a warehouse ultimately ingests, rather than analytical engines themselves. Aurora adds AWS-specific storage and replication engineering on top of the open-source engines for higher throughput and faster failover. Data engineers mostly interact with RDS/Aurora as a source system: extracting via change data capture (Aurora and RDS both support this, often paired with AWS Database Migration Service) or scheduled batch extracts feeding into the warehouse or lake.

AWS Glue is AWS’s managed ETL and metadata service, and it does two distinct jobs that are easy to conflate. Glue ETL runs serverless Apache Spark jobs (or simpler Python shell jobs) to transform data, generated either from a visual job editor or hand-written PySpark. The Glue Data Catalog is a persistent, Hive-metastore-compatible metadata store: it holds table definitions (schema, location, partitioning) for data sitting in S3, and a huge portion of the AWS analytics ecosystem — Athena, Redshift Spectrum, EMR, Glue ETL jobs themselves — reads from this same shared catalog, so registering a table’s schema once in Glue makes it visible to every one of those engines. Amazon Athena is AWS’s serverless, pay-per-query SQL engine (built on Trino/Presto) that queries data directly in S3 using the Glue Data Catalog for schema — the AWS-native example of the “serverless query engine” pattern that BigQuery epitomizes on GCP.

EC2 underlies almost every AWS service in some form (Redshift nodes, EMR clusters, Glue’s execution environment all ultimately run on EC2 capacity), but data engineers rarely provision raw EC2 instances directly for data workloads today — general compute fundamentals are covered in the cloud fundamentals notes, and this note treats EC2 only as the substrate other services build on.

A typical AWS-native pipeline composes these pieces predictably: application databases (RDS/Aurora) or external sources land raw files in S3; an S3 event triggers a Lambda function or step function that kicks off a Glue ETL job; Glue registers the resulting tables in the Data Catalog; Redshift (loaded via COPY from S3, or queried live via Spectrum) or Athena serve the analytical queries; and a BI tool (QuickSight or a third party) sits on top.

Azure data stack

Azure’s data stack mirrors AWS’s shape closely, service-for-service, which is often the fastest way to learn it if you already know AWS.

Azure Blob Storage is Azure’s object storage and data lake foundation, playing the same role S3 plays on AWS. Azure Data Lake Storage Gen2 (ADLS Gen2) is Blob Storage with a hierarchical namespace enabled on top, giving it real directory semantics and POSIX-like ACLs — the configuration data engineers actually want for a lake, rather than plain flat Blob Storage.

Azure SQL Database is Azure’s managed OLTP relational database (SQL Server engine as a service) — the operational-source counterpart to AWS’s RDS, and, like RDS, mostly relevant to a data engineer as a source system to extract from rather than a target to build a warehouse on.

Azure Data Factory (ADF) is Azure’s single answer to what AWS splits across Glue and a separate orchestrator (like Managed Airflow or Step Functions): it is both an ETL/ELT tool (with a visual pipeline designer and “mapping data flows” that compile down to Spark under the hood, similar in spirit to Glue ETL) and a full pipeline orchestrator with scheduling, triggers, and dependency management, similar in spirit to Airflow. This dual role is the single biggest structural difference to internalize when moving between clouds: where an AWS pipeline typically wires Glue (transform) to a separate orchestrator, an Azure pipeline often does both inside ADF alone.

Azure Cosmos DB is Azure’s globally-distributed, multi-model NoSQL database — document, key-value, graph (Gremlin API), and column-family (Cassandra API) access patterns over the same underlying engine, with tunable consistency levels from strong to eventual. It is covered from the NoSQL-modeling side in ./05-nosql-databases.md; in the cloud-platform context, Cosmos DB most often shows up as a low-latency operational source (or occasionally a serving layer for a data product) that ADF or Azure Functions extract from on a schedule or via change feed.

Azure Virtual Machines play the same substrate role EC2 plays on AWS — data-specific services (Data Factory integration runtimes, Synapse’s dedicated SQL pools) run on top of VM capacity, but data engineers rarely provision bare VMs directly for pipeline work.

Note that Azure’s dedicated cloud warehouse offering — Azure Synapse Analytics — is the closest Azure analogue to Redshift/BigQuery/Snowflake (an MPP SQL warehouse with a serverless on-demand query pool alongside provisioned dedicated pools), but the roadmap groups Azure’s core data-warehousing story primarily around Data Factory orchestrating data into Azure SQL/Synapse/Cosmos DB rather than treating Synapse as a separate topic; worth knowing it exists and where it would slot into the AWS/GCP comparison (see the table below).

GCP data stack

GCP’s stack looks structurally similar to AWS and Azure’s at the storage layer, but its warehouse is a genuinely different operating model, not just a rebranded MPP cluster.

Google Cloud Storage (GCS) is GCP’s object storage and data lake foundation — the same role S3 and Blob Storage play, with GCS-native concepts like storage classes (Standard, Nearline, Coldline, Archive) for lifecycle-based cost tiering.

BigQuery is GCP’s serverless data warehouse, and “serverless” here is doing real architectural work, not just marketing: there is no cluster to size, provision, or resize. You load or stream data in, write SQL, and BigQuery’s execution engine (Dremel-derived, using a distributed shuffle-based architecture called Borg under the hood) allocates whatever compute a query needs from a shared multi-tenant pool, transparently, per query. Pricing follows this same serverless philosophy: on-demand pricing bills per byte scanned per query (encouraging column pruning and partition filtering to control cost), while capacity-based (flat-rate) pricing lets high-volume users reserve a fixed number of “slots” (BigQuery’s unit of query compute) for predictable, workload-independent cost. This is a meaningfully different mental model from Redshift or Snowflake, where you still choose and manage a warehouse/cluster size even if it can auto-scale — with BigQuery there is no size to choose in the on-demand model at all.

Dataflow is GCP’s fully managed execution service for Apache Beam pipelines, handling both batch and streaming with the same programming model and the same managed autoscaling — a direct analogue to what Glue ETL/Spark does on AWS or Data Factory’s mapping data flows do on Azure, but built on Beam’s unified batch/streaming abstraction rather than Spark’s.

Compute Engine is GCP’s raw VM service, playing the same substrate role as EC2/Azure VMs — again, rarely provisioned directly by data engineers for pipeline logic itself.

Google Deployment Manager is GCP’s original native infrastructure-as-code service (YAML/Jinja/Python templates describing GCP resources declaratively) — GCP’s original answer to CloudFormation, though in practice most GCP-based data teams today reach for Terraform instead, for the same cross-cloud-consistency reasons discussed below, and Google has been steering new IaC work toward Terraform’s GCP provider and Config Connector rather than continued Deployment Manager investment.

Snowflake: the cloud-agnostic warehouse

Snowflake did not invent cloud data warehousing, but it is the platform most responsible for popularizing the separated storage/compute/services architecture as the expected baseline for a modern warehouse, and for making that architecture available identically across all three major clouds.

Snowflake’s architecture has three distinct layers. The storage layer holds all data, compressed and organized into Snowflake’s proprietary micro-partition format, physically sitting on whichever underlying cloud object storage (S3, Azure Blob, or GCS) the Snowflake account is deployed on — but this is deliberately abstracted away from the user entirely; you never touch a bucket or configure storage directly. The compute layer is made of virtual warehouses — independent clusters of compute resources, sized in “T-shirt sizes” (X-Small through 6X-Large and beyond), each of which can be spun up, resized, or torn down in seconds, and multiple virtual warehouses can query the same underlying data concurrently without contending with each other for compute, because each warehouse gets its own dedicated compute allocation. The cloud services layer handles authentication, metadata, query parsing and optimization, and access control, coordinating across the other two layers.

Two operational features fall directly out of this architecture and explain much of Snowflake’s popularity. Auto-suspend and auto-resume let a virtual warehouse automatically pause after a configurable idle period (stopping compute billing entirely — you pay nothing while paused, since storage billing is fully separate) and automatically resume the moment a new query arrives, typically within a second or two. This makes “pay only for compute you actually use” a default behavior rather than something you have to engineer yourself. Multi-cluster warehouses let a single logical warehouse automatically add or remove parallel clusters in response to concurrent query load, solving the “many analysts hit the dashboard at 9am” concurrency problem without any manual intervention or a human deciding to resize anything.

Snowflake’s other headline capability is Secure Data Sharing (and the broader Snowflake Marketplace built on it): data can be shared live, read-only, and without copying, between Snowflake accounts — including across different cloud providers and regions — because the underlying storage format and metadata are consistent regardless of which cloud a given account happens to be deployed on. A company running its Snowflake account on Azure can share a live dataset with a partner running theirs on AWS with no ETL, no file transfer, and no data duplication involved.

Taken together — the storage/compute separation done well, a genuinely simple operational model (no tuning distribution keys, sort keys, or vacuum jobs the way Redshift historically required), consistent behavior and pricing regardless of which underlying cloud you pick, and frictionless data sharing — explain why Snowflake grew from “another warehouse option” into a platform many organizations treat as effectively a fourth cloud, sitting on top of AWS, Azure, or GCP rather than competing purely as a service within one of them.

Key Concepts

Infrastructure as code for data platforms

Provisioning a warehouse, its databases/schemas, virtual warehouses (Snowflake) or clusters (Redshift), IAM roles and grants, and the pipelines that feed them by hand — through a web console — does not scale past a handful of environments, and it leaves no reviewable history of why a given permission or resource exists. The general mechanics of infrastructure as code — desired-state reconciliation, declarative vs. imperative tools, state management, drift — are covered in depth in ../devops/en/12-infrastructure-as-code.md; this note only adds what is specific to data platforms.

Terraform, and its open-source fork OpenTofu (created after HashiCorp relicensed Terraform from MPL to the more restrictive BUSL in 2023 — see the IaC note for the full story), are the dominant tools for provisioning data infrastructure specifically because they have first-class, actively maintained providers for all four platforms in this note: the official snowflake provider manages databases, schemas, warehouses, users, roles, and grants inside a Snowflake account as code, right alongside aws, azurerm, and google provider resources for S3 buckets, Redshift clusters, BigQuery datasets, or ADF pipelines in the same or a companion configuration. This matters more for data platforms than almost any other infrastructure domain, because a huge fraction of day-to-day data platform work is permission and access management — who can query which schema, which warehouse a given role can use, which service account can write to which bucket — and Terraform/OpenTofu turn all of that into version-controlled, reviewable, diffable configuration rather than one-off console clicks.

AWS CDK (Cloud Development Kit) is worth knowing as the alternative model: instead of a declarative configuration language (HCL), you write actual TypeScript, Python, Java, or another general-purpose language, and CDK synthesizes that down to a CloudFormation template. This buys real programming constructs — loops, conditionals, functions, unit tests, shared libraries of organizational best-practice constructs — at the cost of being AWS-only. Teams already deep in the AWS ecosystem, with engineers more comfortable writing code than HCL, often prefer CDK for exactly that reason; multi-cloud or Snowflake-inclusive teams almost always end up on Terraform/OpenTofu instead, simply because CDK has no path to managing Snowflake, Azure, or GCP resources.

Serverless options for data processing

Serverless query engines and event-driven compute are a recurring pattern across every cloud discussed in this note, and they matter specifically because analytical and pipeline workloads are so often spiky and unpredictable rather than steady-state.

Serverless query engines — BigQuery’s on-demand mode, Athena on AWS, and Redshift Serverless — all share the same value proposition: no cluster sits around idle waiting for the next query, so cost tracks actual usage (bytes scanned, or seconds of compute) rather than provisioned capacity. This fits ad-hoc analytics, infrequent reporting, and workloads with unpredictable or bursty query volume especially well, since there is no capacity-planning decision to get wrong in either direction (too small and queries queue; too large and idle capacity is wasted).

Event-driven, file-arrival-triggered compute is the other major serverless pattern, and it is the mechanism behind most modern ingestion pipelines: an object landing in S3 (or Blob Storage, or GCS) fires an event notification, which invokes a serverless function (Lambda, Azure Functions, Cloud Functions/Cloud Run) that kicks off downstream processing — validating the file, triggering a Glue job or Dataflow pipeline, or loading it into a warehouse. This suits data ingestion particularly well because file arrival is inherently irregular (a partner drops a file whenever their system produces it, not on your schedule), and paying only for the few seconds of compute needed to react to each arrival beats running a poller or a fixed-schedule job that is idle most of the time and occasionally late.

Choosing the right technology

There is no single “best” cloud data stack, and treating this as a pure technology comparison misses the factors that actually decide it in practice.

Existing cloud commitment dominates in most organizations: a company already running its application infrastructure on AWS will default to Redshift/Glue/Athena for the same reasons it defaults to any other AWS service — existing IAM setup, existing networking, existing vendor relationship and enterprise discount, existing team familiarity. The same logic applies symmetrically to Azure- and GCP-committed organizations. This single factor overrides most feature-level comparisons for the majority of teams.

Team skill set matters independently of cloud commitment: a team of strong SQL analysts and few dedicated data engineers often gravitates toward BigQuery or Snowflake precisely because of how little infrastructure they have to manage day to day, whereas a team with deep Spark and infrastructure expertise may extract more value from Redshift’s or Databricks’ greater tunability.

Cost model is a real architectural decision, not just a line-item comparison: pay-per-query (BigQuery on-demand, Athena, Snowflake with aggressive auto-suspend) rewards spiky, unpredictable workloads and punishes constant, heavy, steady-state usage, where reserved/flat-rate capacity (BigQuery flat-rate slots, Redshift reserved instances, Snowflake with warehouses running near-continuously) becomes cheaper. Teams frequently get this backwards — provisioning reserved capacity for a bursty workload, or running on-demand pricing against a workload that never really stops — and it is one of the most common sources of cloud data-platform cost overruns.

Data volume and velocity push toward specific tools regardless of cloud preference: very high-throughput streaming ingestion tends to pull in Kafka/Kinesis/Pub/Sub plus a streaming-capable processor (Dataflow, Flink) regardless of which warehouse sits downstream, while pure batch, moderate-volume analytical workloads are comfortably served by almost any of the four platforms in this note.

Vendor lock-in tolerance is the most strategic and least reversible factor. Deep use of cloud-native, proprietary services (Glue’s specific job model, ADF’s proprietary pipeline format, BigQuery-specific SQL extensions) is efficient but expensive to unwind later; Snowflake’s cross-cloud portability and reliance on standard SQL and open connectors reduces lock-in to any one cloud but still creates Snowflake-specific lock-in; leaning on open table formats (Iceberg, Delta Lake — see ./07-data-lakes-and-modern-architectures.md) and open compute engines (Spark, Trino) on top of plain object storage minimizes lock-in the most, at the cost of taking on more integration and operational work yourself. None of these are wrong choices in isolation — they are different points on a real trade-off between operational simplicity and long-term flexibility, and the right answer depends on how much that flexibility is actually worth to a given organization.

Comparison table: AWS vs. Azure vs. GCP vs. Snowflake

Best Practices

• Match the platform to existing cloud commitment first, features second. The integration cost of fighting your organization’s existing IAM, networking, and billing setup usually outweighs marginal feature differences between warehouses.
• Default to separated storage and compute wherever it’s offered (Redshift RA3/Serverless over old dense-compute nodes, BigQuery over any clustered alternative, Snowflake’s native model) — the cost and flexibility benefits are close to free once you’re on the modern architecture, and expensive to retrofit later.
• Turn on auto-suspend / auto-scaling by default, then tune it, rather than leaving compute running “just in case.” Idle warehouse compute is one of the largest and most avoidable sources of cloud data-platform overspend.
• Register schemas in a shared catalog (Glue Data Catalog, Dataplex/Data Catalog, Purview) as a first-class deliverable, not an afterthought — the value of a serverless query engine like Athena or a lakehouse table format collapses if nothing can discover the tables’ schema and location.
• Choose the ETL/orchestration split deliberately. Know whether your cloud’s model is “one tool does both” (Azure Data Factory) or “separate transform and orchestration tools” (AWS Glue + Step Functions/Airflow, GCP Dataflow + Cloud Composer) before designing pipelines, so you don’t accidentally build orchestration logic inside a transform tool not meant to hold it, or vice versa.
• Provision data infrastructure — warehouses, roles, grants, buckets, pipelines — with Terraform or OpenTofu rather than console clicks, especially for anything involving Snowflake, where access control changes are otherwise invisible and unreviewable; reserve AWS CDK for teams fully committed to AWS-only infrastructure who value general-purpose-language ergonomics over multi-cloud portability.
• Pick pay-per-query vs. reserved/flat-rate pricing based on actual usage shape, not habit. Model a bursty, ad-hoc workload against on-demand pricing and a constant, heavy workload against reserved/flat-rate capacity before committing — the wrong pairing is a common, quietly compounding source of cloud cost overruns.
• Treat vendor lock-in as a spectrum to choose deliberately, not something to avoid at all costs. Proprietary managed services (Glue jobs, ADF pipelines, BigQuery SQL extensions) buy real operational simplicity; open table formats and open compute engines buy flexibility. Decide which one your organization actually needs more of, rather than defaulting to either extreme.
• Reserve Snowflake (or any cross-cloud strategy) for organizations that genuinely need multi-cloud reach or external data sharing. A single-cloud-committed team with no data-sharing requirement often gets equivalent value from that cloud’s native warehouse with less operational surface area to reason about.

References

• AWS — Analytics Services Overview
• Amazon Redshift Documentation
• AWS Glue Documentation
• Microsoft — Azure Data Factory Documentation
• Microsoft — Azure Cosmos DB Documentation
• Google Cloud — BigQuery Documentation
• Google Cloud — Dataflow Documentation
• Snowflake — Key Concepts & Architecture
• roadmap.sh — Data Engineer Roadmap