Big Data & Distributed ComputingBig Data & Distributed Computing

Part of the Data Engineer Roadmap knowledge base.

Overview

Every data platform eventually hits the same wall: a table, a join, or a nightly aggregation job that used to finish in minutes now takes hours, or simply doesn’t fit in the memory of the machine running it. At that point there are two ways forward. Vertical scaling buys a bigger machine — more RAM, more CPU cores, faster disks — which is simple but has a hard ceiling and gets disproportionately expensive near the top of the hardware market. Horizontal scaling instead splits the data and the work across many ordinary machines that cooperate as a single system. This is the fundamental shift that “big data” technologies are built around, and it is the same trade-off backend engineers make when scaling a stateless web service (see ../backend/README.md for the general scaling vocabulary) — except a distributed data processing system also has to move and shuffle large volumes of state, not just stateless requests, which introduces problems that don’t exist when scaling web servers behind a load balancer.

This note covers what a data engineer needs to actually reason about distributed computing: partitioning, shuffling, and fault tolerance as the core mechanics; clusters and resource managers (YARN, Kubernetes) as the machinery that runs the work; the Hadoop ecosystem (HDFS, MapReduce) as the historical foundation nearly every modern system either descends from or reacted against; and Apache Spark as the tool a data engineer will actually use day to day. It closes with a note on why cloud object storage has largely replaced self-managed HDFS in modern stacks, and how this topic connects to data lakes (./07-data-lakes-and-modern-architectures.md), streaming (./11-streaming-and-realtime-data.md), container orchestration (./12-containers-and-orchestration.md), and managed cloud platforms (./13-cloud-data-platforms.md).

Fundamentals

Why distributed computing

A single machine has three finite resources that a data workload can exhaust: memory, disk, and CPU throughput. A dataset that is a few gigabytes fits comfortably in RAM on a laptop; a dataset that is tens of terabytes does not, no matter how much RAM you buy, because RAM capacity per machine has practical and economic limits. Even when data does fit on disk, a single CPU (or even a multi-core single machine) processing it serially puts a hard ceiling on throughput — reading, transforming, and aggregating a petabyte of data on one machine could take days or weeks even if the disk could hold it.

Distributed computing addresses both constraints by splitting a large problem into many smaller pieces, sending each piece to a different machine in a cluster, and combining the partial results. If splitting the work N ways scaled perfectly, N machines would finish in 1/N of the time. In practice it never scales perfectly — coordinating machines, moving data between them, and combining results all cost time — but going from one machine to a cluster of dozens or hundreds turns jobs that are impossible on a single node into jobs that finish in minutes.

It’s worth being honest about when this matters. Most analytical workloads — even at companies that consider themselves “big data” shops — fit comfortably on a single well-resourced machine using tools like DuckDB, Polars, or a single-node Spark/pandas job. Distributed computing adds real operational complexity (cluster provisioning, network partitions, partial failures, debugging across dozens of machines) that is only worth paying for once data volume or processing throughput genuinely exceeds what a single powerful machine can do. A good data engineer reaches for a cluster because the numbers demand it, not by default.

Data partitioning

Partitioning is how a distributed system decides which rows or records live on which node (or, within Spark, which partition of an executor’s memory). A dataset is split into partitions — logical chunks — and each partition is processed independently and in parallel by a task running on some node in the cluster. Partitioning is what makes horizontal scaling possible at all: without it, there would be no way to divide the work.

How data is partitioned matters enormously for performance. If a dataset of user events is partitioned by a hash of user_id, then all events for a given user land in the same partition — which is exactly what you want for an operation that needs all of one user’s events together (a groupBy(user_id), for instance). If the same dataset is partitioned round-robin (row 1 to partition 1, row 2 to partition 2, and so on with no regard to content), no such guarantee holds, and an operation that needs to group by user_id will have to move data between partitions to bring matching keys together. Choosing the right partitioning key — and avoiding partition skew, where one key is far more common than others and one partition ends up doing most of the work while the rest sit idle — is one of the most consequential decisions in tuning a distributed job.

The shuffle problem

A shuffle is what happens when an operation needs data that currently lives on different nodes to be brought together onto the same node — most commonly for JOIN, GROUP BY, DISTINCT, and repartitioning operations. Concretely: if orders is partitioned by order_id and customers is partitioned by customer_id, joining them on customer_id requires physically moving rows across the network so that matching customer_id values end up co-located on the same executor. This involves serializing data, writing it to disk, transferring it over the network, and deserializing it on the receiving node — all of which is orders of magnitude slower than an in-memory, local computation.

This is the thing that makes distributed processing slower than engineers new to the field expect. A job that looks fast on paper (aggregate a billion rows, easy, just split it 1,000 ways) can be dominated end to end by shuffle time if the aggregation requires moving most of the data across the network. Symptoms of shuffle-heavy jobs include long “stage” times in the Spark UI that don’t correspond to obvious computation, disk spill (when shuffle data doesn’t fit in memory and has to be written to and read back from local disk), and network saturation across the cluster.

The practical response is to design jobs to minimize shuffling: filter and project (select only needed columns/rows) before a join or group-by rather than after, so less data needs to move; pre-partition data on disk by the key you’ll frequently join or group on, so it’s already co-located; use broadcast joins when one side of a join is small enough to fit in memory on every node (avoiding a shuffle of the large side entirely); and be deliberate about repartition()/coalesce() calls, since both trigger a shuffle (repartition redistributes data across a new number of partitions; coalesce merges partitions without a full shuffle when reducing partition count).

Fault tolerance

At the scale of hundreds of machines running for hours, hardware and network failures are not an edge case — they are a statistical certainty. A disk fails, a node loses network connectivity, a process gets killed by the OS for using too much memory. A distributed processing system has to be built assuming this will happen mid-job, and designed so a single node failure doesn’t require restarting the entire computation from scratch.

Two ideas make this possible. First, replication: data is stored with multiple copies (HDFS defaults to 3x replication) across different nodes and racks, so losing one node’s disk doesn’t lose the data. Second, lineage and recomputation: modern engines like Spark track the sequence of transformations that produced each partition of data (this is exactly what the RDD lineage graph or DataFrame query plan records). If a node holding a partition dies mid-job, Spark doesn’t need a checkpoint of that exact partition — it can recompute just that partition from its lineage, using the original input data and the transformations that were supposed to produce it, and re-run only the lost work rather than the whole job. This combination — durable input via replication, recoverable intermediate state via lineage — is what lets a job spanning a 500-node cluster survive a handful of node failures without an operator needing to intervene.

Key Concepts

Cluster computing and cluster management

A cluster is a group of machines (physical or virtual) that cooperate to behave as a single system to the applications running on it. From a data engineer’s point of view, a cluster provides pooled CPU, memory, and (sometimes) storage, and a scheduler decides which machine runs which piece of work.

Someone has to decide, at any given moment, which jobs get which slice of the cluster’s CPU and memory — this is the job of a resource manager. In the Hadoop ecosystem this role belongs to YARN (Yet Another Resource Negotiator): a ResourceManager process tracks available resources across the cluster and negotiates with NodeManager agents on each machine to launch and monitor “containers” (units of allocated CPU/memory, not to be confused with Docker containers) in which application tasks actually run. YARN decoupled Hadoop’s resource management from MapReduce specifically so other frameworks — Spark included — could run on the same cluster and share resources with it, rather than each processing engine needing its own dedicated fleet of machines.

Increasingly, Kubernetes fills this same role for data workloads, not just for stateless microservices. Spark, Flink, and other engines can run natively on Kubernetes, using it as a general-purpose scheduler that allocates pods (not YARN containers) across a cluster, handles restarts of failed executors, and lets data workloads share infrastructure with the rest of an organization’s containerized applications rather than requiring a separate Hadoop/YARN cluster. This convergence is covered in more depth in ./12-containers-and-orchestration.md; the short version is that for new deployments, “Spark on Kubernetes” has become at least as common a target as “Spark on YARN,” particularly for organizations that already run Kubernetes for everything else.

The Hadoop ecosystem

Hadoop is the project that popularized the current generation of open-source distributed data processing, built out of two core components plus a resource manager layer added later.

HDFS (Hadoop Distributed File System) solves the problem of storing files far larger than any single disk. A file is split into fixed-size blocks (128 MB by default in modern Hadoop), and each block is stored — and, crucially, replicated — across multiple machines (3 copies by default, typically spread across different racks so a single rack failure doesn’t cause data loss). Two kinds of nodes coordinate this:

The NameNode is a well-known single point of failure in classic Hadoop deployments (mitigated in production clusters with HDFS High Availability, running a standby NameNode with shared edit logs via a quorum journal). If a DataNode goes down, the NameNode notices missing heartbeats and re-replicates the blocks that node was holding onto other healthy nodes, restoring the replication factor automatically.

MapReduce was Hadoop’s original processing model, and it’s still worth understanding because its two-phase structure (map, then reduce) is the conceptual ancestor of virtually every distributed processing API that followed, including Spark’s. A MapReduce job has two user-defined phases:

• Map — runs a function over every input record independently and in parallel (typically one map task per HDFS block), producing intermediate key-value pairs. Because each map task only ever looks at its own slice of input, this phase parallelizes trivially.
• Reduce — receives all values associated with each key (Hadoop’s shuffle phase sorts and groups the map output by key across the cluster before reduce runs) and aggregates them into a final result, which is written back to HDFS.

A classic word-count example: the map phase emits (word, 1) for every word in every document; the shuffle groups all 1s by word across the cluster; the reduce phase sums the counts per word. The pattern generalizes to filtering, joining, and aggregating far larger datasets than word counts, but the same shuffle mechanics described above apply — and in classic MapReduce, every intermediate result between map and reduce, and between chained MapReduce jobs, is written to disk. This is the main reason MapReduce has been largely superseded by Spark for most workloads: a multi-step MapReduce pipeline pays a full disk write and read for every stage, whereas Spark keeps intermediate data in memory across steps whenever it fits, only spilling to disk under memory pressure. For a chain of 10 transformations, that difference is not incremental — it can be an order of magnitude in wall-clock time. MapReduce is nonetheless still foundational to understand: HDFS, YARN, and the map/shuffle/reduce mental model underpin how Spark itself executes work internally, and some organizations still run legacy MapReduce jobs they haven’t migrated.

Apache Spark in depth

Spark was built directly in response to MapReduce’s disk-bound execution model, with in-memory processing across steps as its central improvement: intermediate results from one transformation can stay in a cluster’s collective RAM and feed directly into the next transformation, rather than round-tripping through disk. For iterative workloads (machine learning training, iterative graph algorithms) and multi-step pipelines, this typically makes Spark 10–100x faster than equivalent MapReduce jobs.

Spark’s core abstraction is the RDD (Resilient Distributed Dataset): an immutable, partitioned collection of objects spread across the cluster, along with the lineage (the sequence of transformations) needed to recompute any lost partition — this lineage is exactly the fault-tolerance mechanism described earlier. RDDs are low-level and most data engineering code today is written against the higher-level DataFrame API instead — a distributed table with named, typed columns, conceptually similar to a pandas or R DataFrame but partitioned across a cluster, and built on top of RDDs under the hood. DataFrames benefit from Catalyst, Spark’s query optimizer, and Tungsten, its off-heap memory/execution engine, which together let Spark plan and execute DataFrame operations far more efficiently than equivalent hand-written RDD code — for this reason, DataFrames (and Spark SQL, below) are the recommended entry point for nearly all new Spark code, with raw RDDs reserved for unusual low-level needs.

Central to how Spark executes any of this is lazy evaluation. Calling .filter(), .select(), or .groupBy() on a DataFrame does not compute anything immediately — it appends a step to a logical query plan. Only an action — .collect(), .count(), .write(), .show() — actually triggers execution, at which point Spark’s Catalyst optimizer looks at the entire chain of transformations at once (not one at a time), reorders and merges operations where it’s profitable (pushing filters as early as possible, for instance, so less data flows into a later join), and only then generates and runs the physical execution plan across the cluster. This is why calling .explain() on a DataFrame before running it is one of the most useful debugging habits in Spark: it shows exactly what physical plan — including where shuffles occur — Catalyst decided on.

Spark SQL lets you express the same DataFrame operations as SQL queries against registered tables/views, which Catalyst compiles down to the identical physical plan as the equivalent DataFrame code — SQL and the DataFrame API are two syntaxes for the same underlying engine, and can be freely mixed in the same job.

A real PySpark example

The following reads a CSV of e-commerce orders, filters to completed orders, computes per-country revenue, and writes the result out as partitioned Parquet — a small but complete, realistic ETL step.

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = (
    SparkSession.builder
    .appName("daily-revenue-by-country")
    .config("spark.sql.shuffle.partitions", "200")  # tune down from default 200 for smaller clusters
    .getOrCreate()
)

# Read: schema inference is convenient for exploration, but an explicit
# schema avoids a full extra read pass and surprising type inference in production.
orders = (
    spark.read
    .option("header", "true")
    .schema(
        "order_id STRING, customer_id STRING, country STRING, "
        "status STRING, amount DOUBLE, order_ts TIMESTAMP"
    )
    .csv("s3://raw-bucket/orders/2026-07-18/*.csv")
)

# Transformations are lazy: nothing runs yet, Spark is just building a plan.
completed = orders.filter(F.col("status") == "completed")

revenue_by_country = (
    completed
    .groupBy("country")                     # triggers a shuffle: rows for the
                                             # same country must land on one executor
    .agg(
        F.sum("amount").alias("total_revenue"),
        F.count("order_id").alias("order_count"),
        F.countDistinct("customer_id").alias("unique_customers"),
    )
    .orderBy(F.desc("total_revenue"))
)

# Inspect the physical plan before running anything — see where the shuffle happens.
revenue_by_country.explain()

# .write() is an action: this is what actually triggers execution across the cluster.
(
    revenue_by_country
    .write
    .mode("overwrite")
    .partitionBy("country")
    .parquet("s3://curated-bucket/revenue_by_country/2026-07-18/")
)

# Equivalent logic expressed as Spark SQL against the same DataFrame,
# compiled by Catalyst to the same physical plan:
completed.createOrReplaceTempView("completed_orders")
spark.sql("""
    SELECT country,
           SUM(amount)              AS total_revenue,
           COUNT(order_id)          AS order_count,
           COUNT(DISTINCT customer_id) AS unique_customers
    FROM completed_orders
    GROUP BY country
    ORDER BY total_revenue DESC
""").show(10)

spark.stop()

A few things worth calling out in this example: the groupBy("country") is the point in the job where a shuffle happens — every executor’s local rows for a given country need to end up together before the aggregation can run, so this is exactly the kind of operation the “shuffle problem” section above warns about, and spark.sql.shuffle.partitions controls how many partitions that shuffle produces (the default of 200 is often too high for small jobs and too low for very large ones — tuning it is one of the highest-leverage Spark performance knobs). Writing with .partitionBy("country") lays the output out as one subdirectory per country in the target path, which lets downstream readers prune partitions and read only the countries they need instead of scanning everything. And nothing above actually executes until .write() or .show() is called — this is lazy evaluation in practice, not just in theory.

Distributed file systems beyond HDFS

In an on-premises Hadoop cluster, HDFS is the natural place to store the data a cluster computes over — the storage and compute live on the same machines. In modern cloud-native data platforms, that arrangement has largely been replaced by cloud object storage — Amazon S3, Google Cloud Storage (GCS), Azure Data Lake Storage (ADLS) — as the primary distributed storage layer, with compute (Spark on EMR/Dataproc/Databricks, for instance) reading and writing to it over the network rather than from local disk.

A few reasons this shift happened and stuck: object storage requires no cluster to manage — no NameNode to keep highly available, no DataNode disks to provision or replace, no capacity planning for storage growth, since it’s effectively unlimited and elastic. It is billed pay-per-use (per GB stored, per request, per GB transferred) rather than requiring upfront, always-on hardware sized for peak load. And it integrates natively with managed compute platforms — Databricks, AWS EMR, Google Dataproc — that spin ephemeral compute clusters up and down against data that persists independently in object storage, decoupling the lifecycle (and cost) of storage from the lifecycle of compute, which is a much better fit for workloads that don’t need to run 24/7. The trade-off is that object storage is not a true POSIX filesystem — no in-place file appends or renames as atomic metadata operations the way HDFS provides — which is exactly the gap that table formats like Delta Lake, Iceberg, and Hudi exist to fill; see ./07-data-lakes-and-modern-architectures.md for that layer.

MapReduce vs. Spark

Understanding this table is less about picking a side in a settled argument — Spark won for the overwhelming majority of new work years ago — and more about recognizing why: the shuffle problem and disk I/O costs are shared by both engines, but Spark’s in-memory execution and lazy, optimizer-driven planning attack exactly the bottleneck that made MapReduce slow, without abandoning the same underlying cluster/partitioning/fault-tolerance model MapReduce established.

Best Practices

• Reach for a cluster only when the numbers justify it. Benchmark against a single well-resourced machine (DuckDB, Polars, single-node Spark) before assuming distributed processing is needed — the operational cost of a cluster is real and only pays for itself past a certain data volume or throughput requirement.
• Design around the shuffle, don’t discover it in production. Filter and select columns before joins/aggregations, pre-partition data on disk by frequently-joined keys, and use broadcast joins for small dimension tables to avoid shuffling the large side of a join.
• Watch for and address partition skew explicitly. A job that’s slow because one partition is 100x larger than the rest looks identical to “the cluster needs more resources” in surface-level monitoring — check the Spark UI’s task duration distribution before scaling up hardware.
• Prefer the DataFrame/Spark SQL API over raw RDDs for new code — Catalyst and Tungsten optimize DataFrame/SQL operations in ways hand-written RDD transformations don’t benefit from.
• Read .explain() output before running expensive jobs, especially to confirm where shuffles happen and whether a broadcast join is being used where expected.
• Set an explicit schema on read rather than relying on inference in production pipelines — inference costs an extra read pass and can silently mis-type columns on malformed input.
• Tune spark.sql.shuffle.partitions to cluster size and data volume rather than leaving the default — too few partitions underutilizes the cluster; too many adds scheduling overhead per task.
• Default to cloud object storage over self-managed HDFS for new platforms unless there’s a specific reason (on-prem constraints, extreme low-latency small-file access patterns) to run HDFS — the operational savings are substantial.
• Treat node/task failures as expected, not exceptional — verify a job’s retry and checkpointing behavior under simulated failures rather than assuming fault tolerance “just works” without configuration (checkpointing, in particular, matters a great deal for long streaming jobs — see ./11-streaming-and-realtime-data.md).

References

• Apache Spark — Official Documentation
• Apache Spark — RDD Programming Guide
• Apache Spark — SQL, DataFrames and Datasets Guide
• Apache Hadoop — HDFS Architecture Guide
• Apache Hadoop — YARN Architecture
• Apache Hadoop — MapReduce Tutorial
• Matei Zaharia et al. — Resilient Distributed Datasets (RDD paper)
• roadmap.sh — Data Engineer