Source: apache_tez · apache_tez.md · updated 2026-05-29 · 🔒 secret gist

Synced verbatim from gist.github.com/bl9.

Apache Tez: A Deep Dive Into Architecture, Internals & Source Code

Table of Contents

1. What Is Apache Tez?
2. Why Tez Exists — The MapReduce Problem
3. Architecture Overview
4. Core Concepts
5. Source Code Structure
6. Deep Dive: Key Modules
7. The DAG Execution Lifecycle
8. The Shuffle Pipeline
9. Runtime Reconfiguration
10. How Hive and Pig Use Tez
11. Tez on YARN — The Full Picture
12. Reading the Source: A Guided Path
13. Simpler Open Source Alternatives
14. References

1. What Is Apache Tez?

Apache Tez is a distributed data-processing execution engine built on top of Apache Hadoop YARN. It generalizes the rigid two-phase MapReduce model into an arbitrary directed acyclic graph (DAG) of tasks, enabling higher-level frameworks like Hive and Pig to express complex query plans as a single, optimized execution graph rather than a chain of independent MapReduce jobs.

Tez is not an end-user framework. You don't typically write Tez applications directly (though you can). Instead, Tez acts as the execution backend for tools like:

• Apache Hive (SQL on Hadoop)
• Apache Pig (data flow scripting)
• Cascading (Java data pipelines)

Key Stats

2. Why Tez Exists — The MapReduce Problem

The Pain Points of MapReduce

Classic MapReduce forces every computation into a rigid two-phase pattern:

Input → Map → Shuffle/Sort → Reduce → Output (HDFS)

For a complex Hive query that involves multiple joins and aggregations, MapReduce had to chain several independent jobs:

MR Job 1: Scan + Filter     → write to HDFS
MR Job 2: Join               → read from HDFS, write to HDFS
MR Job 3: Aggregate          → read from HDFS, write to HDFS
MR Job 4: Final sort + output → read from HDFS, write to HDFS

Each boundary between jobs meant:

• Materializing intermediate data to HDFS (disk I/O + replication overhead)
• Launching new YARN containers (JVM startup latency)
• No pipelining between stages (the next job can't start until the previous one fully finishes writing)

What Tez Changes

Tez lets you express the same computation as a single DAG:

Scan+Filter ─┐
              ├─→ Join ─→ Aggregate ─→ Sort ─→ Output
Scan+Filter ─┘

Intermediate data flows through in-memory edges (or local disk, never HDFS) between vertices in the same DAG. Containers are reused across vertices. The entire plan runs as one YARN application, not four.

Performance Impact

For Hive queries, switching from MapReduce to Tez typically yields:

• 2–5x faster for simple queries (fewer job launches, no HDFS writes between stages)
• 10–100x faster for complex multi-stage queries (eliminates cascading HDFS I/O)
• Near-interactive latency for many queries that previously took minutes

3. Architecture Overview

Tez has a clean three-layer architecture:

┌─────────────────────────────────────────────────┐
│              CLIENT APPLICATION                  │
│  (Hive, Pig, or custom Tez app)                  │
│                                                   │
│  Uses DAG API to build: Vertices + Edges          │
│  Submits DAG to the Tez ApplicationMaster         │
└────────────────────┬────────────────────────────┘
                     │ Submit DAG
                     ▼
┌─────────────────────────────────────────────────┐
│          TEZ APPLICATION MASTER (AM)             │
│  (runs as a YARN ApplicationMaster)              │
│                                                   │
│  • DAG state machine (DAGImpl)                    │
│  • Vertex state machine (VertexImpl)              │
│  • Task state machine (TaskImpl)                  │
│  • TaskAttempt state machine (TaskAttemptImpl)    │
│  • Container management & reuse                   │
│  • Speculative execution                          │
│  • Runtime reconfiguration                        │
│  • Scheduling & data locality                     │
└────────────────────┬────────────────────────────┘
                     │ Launch tasks in containers
                     ▼
┌─────────────────────────────────────────────────┐
│            TASK RUNTIME (per container)           │
│                                                   │
│  • Input (reads data: HDFS, shuffle, etc.)        │
│  • Processor (user logic: map, reduce, join)      │
│  • Output (writes data: shuffle, HDFS, etc.)      │
│                                                   │
│  Tasks run the Input-Processor-Output pipeline    │
│  within YARN containers on cluster nodes          │
└─────────────────────────────────────────────────┘

The Two Core Components (Per the Tez README)

At its heart, Tez has just two components:

1. The data-processing pipeline engine — where you plug in Input, Processor, and Output implementations to perform arbitrary data processing within each task.

2. The DAG-based application master — where you compose those tasks into a DAG and the master handles scheduling, fault tolerance, and resource negotiation with YARN.

4. Core Concepts

DAG (Directed Acyclic Graph)

The top-level execution unit. A DAG contains vertices connected by edges. It maps to a single YARN application.

DAG dag = DAG.create("my-query");
dag.addVertex(scanVertex);
dag.addVertex(joinVertex);
dag.addEdge(Edge.create(scanVertex, joinVertex, edgeConfig));

Vertex

A logical stage in the pipeline. Each vertex has:

• A parallelism (number of tasks)
• A Processor class (the computation logic)
• Zero or more Inputs and Outputs

A vertex with parallelism=100 runs 100 tasks, each executing the same Processor on different data partitions.

Edge

Defines how data flows between two vertices. An edge specifies:

• Data movement type: ONE_TO_ONE, SCATTER_GATHER, BROADCAST
• Scheduling type: SEQUENTIAL (downstream waits) or CONCURRENT
• Data source type: PERSISTED (written to disk) or EPHEMERAL (pipelined)
• Edge manager plugin: controls routing of data between producer and consumer tasks

Task

A single unit of work within a vertex. Runs in a YARN container with an Input → Processor → Output pipeline.

Input / Processor / Output

The pluggable components of each task:

• Input: reads key-value pairs (from HDFS, from shuffle, from another source)
• Processor: transforms data (the user's computation logic)
• Output: writes key-value pairs (to shuffle, to HDFS, to another sink)

This is the fundamental abstraction. MapReduce hardcodes the inputs and outputs; Tez makes them pluggable.

5. Source Code Structure

The repository (github.com/apache/tez) is organized into these Maven modules:

tez/
├── tez-api/                  # Public-facing API (DAG, Vertex, Edge, configs)
├── tez-common/               # Shared utilities, counters, security
├── tez-dag/                  # ★ THE CORE: ApplicationMaster, state machines
├── tez-runtime-internals/    # ★ Task runtime framework (Input/Processor/Output lifecycle)
├── tez-runtime-library/      # ★ Built-in I/O implementations (shuffle, sort, merge)
├── tez-mapreduce/            # MapReduce compatibility layer
├── tez-examples/             # WordCount, OrderedWordCount examples
├── tez-plugins/              # History logging, YARN plugin
├── tez-tests/                # Integration tests
├── tez-ui/                   # Web UI (Ember.js)
└── tez-dist/                 # Distribution packaging

Complexity by Module

6. Deep Dive: Key Modules

6.1 tez-api — The Public API

This is the module that Hive, Pig, and custom applications interact with. The key classes:

DAG.java

The builder for a DAG. Lets you:

• Create a named DAG
• Add vertices and edges
• Set DAG-level configuration and credentials
• Attach security tokens

DAG dag = DAG.create("word-count");
Vertex tokenizer = Vertex.create("Tokenizer", ProcessorDescriptor.create(
    TokenProcessor.class.getName()), numTasks);
Vertex summation = Vertex.create("Summation", ProcessorDescriptor.create(
    SumProcessor.class.getName()), numTasks);

dag.addVertex(tokenizer)
   .addVertex(summation)
   .addEdge(Edge.create(tokenizer, summation, edgeConfig));

Vertex.java

Defines a processing stage. Configuration includes:

• Processor class
• Parallelism (task count)
• Data sources (inputs that don't come from other vertices)
• Environment settings (JVM opts, environment variables)
• Vertex manager plugin (for dynamic parallelism)

Edge.java

Connects two vertices. The EdgeProperty specifies:

EdgeProperty.create(
    DataMovementType.SCATTER_GATHER,  // How data is routed
    DataSourceType.PERSISTED,          // Disk or in-memory
    SchedulingType.SEQUENTIAL,         // When downstream starts
    outputDescriptor,                  // Output class on producer side
    inputDescriptor                    // Input class on consumer side
);

Data movement types — the critical routing strategies:

TezClient.java

The client-side entry point. Creates a YARN application, uploads the DAG plan, and submits it to the Tez ApplicationMaster.

6.2 tez-dag — The ApplicationMaster (The Hard Core)

This is where most of the complexity lives. The AM runs as a YARN ApplicationMaster and manages the entire DAG execution.

State Machines

The AM is built around four nested state machines, each implemented with Hadoop's StateMachineFactory:

DAGImpl (DAG lifecycle)
 └── VertexImpl (per-vertex lifecycle)
      └── TaskImpl (per-task lifecycle)
           └── TaskAttemptImpl (per-attempt lifecycle)

DAGImpl.java — DAG State Machine

States: NEW → INITED → RUNNING → COMMITTING → SUCCEEDED / FAILED / KILLED / ERROR

Key transitions:

This state machine orchestrates the entire execution. When a vertex completes, the DAG checks which downstream vertices have all their input-vertices completed and starts them.

Tip: You can generate a visual Graphviz diagram of this state machine:

mvn compile -Pvisualize \
  -Dtez.dag.state.classes=org.apache.tez.dag.app.dag.impl.DAGImpl \
  -DskipTests=true

This outputs Tez.gv which you can render with dot -Tpng Tez.gv -o dag-states.png.

VertexImpl.java — Vertex State Machine

States: NEW → INITIALIZING → INITED → RUNNING → COMMITTING → SUCCEEDED / FAILED / KILLED

This is the most complex state machine. A vertex:

1. Initializes its tasks based on input splits or configured parallelism
2. Waits for the VertexManager to signal readiness
3. Schedules tasks on available containers
4. Tracks task completion, handles failures and retries
5. Commits output when all tasks succeed

Key complexity here: vertex reconfiguration. At runtime, a vertex can change its parallelism based on actual data sizes (see Section 9).

TaskImpl.java and TaskAttemptImpl.java

Task manages one logical unit of work. TaskAttempt tracks a single execution attempt of that task (there may be multiple attempts due to failures or speculative execution).

TaskAttempt handles:

• Container assignment
• Launch on a NodeManager
• Status updates and heartbeats
• Output commit
• Failure handling (retry logic)

Container Management

The AM doesn't just schedule tasks — it manages container reuse. Key classes:

• TaskSchedulerManager — coordinates container requests with YARN
• ContainerLauncherManager — launches tasks in containers
• TaskCommunicatorManager — heartbeat protocol with running tasks

Container reuse is a critical optimization. When a task finishes, the AM can reassign the same container to the next task in the DAG without releasing it back to YARN and requesting a new one. This avoids the JVM startup penalty that made chained MapReduce jobs slow.

6.3 tez-runtime-internals — The Task Runtime

This module manages the lifecycle of a single task running inside a YARN container.

LogicalIOProcessorRuntimeTask.java

The main class that runs inside each container. It:

1. Initializes the configured Inputs
2. Initializes the Processor
3. Initializes the configured Outputs
4. Calls processor.run(inputs, outputs) — this is where user code executes
5. Handles cleanup and status reporting

The Input-Processor-Output Model

┌──────────┐     ┌──────────────┐     ┌───────────┐
│  Input 1  │────→│              │────→│  Output 1  │
├──────────┤     │   Processor   │     ├───────────┤
│  Input 2  │────→│              │────→│  Output 2  │
└──────────┘     └──────────────┘     └───────────┘

Each component is pluggable:

• Inputs implement LogicalInput — they know how to read from a data source
• Processors implement LogicalIOProcessor — they transform data
• Outputs implement LogicalOutput — they know how to write to a destination

The task runtime wires them together based on the DAG plan. This is the key abstraction that makes Tez flexible — MapReduce hardcodes the wiring; Tez lets you plug in any combination.

6.4 tez-runtime-library — Built-in I/O Implementations

This is the other highly complex module. It provides the actual data movement implementations.

Key Classes

The Shuffle Pipeline (Scatter-Gather)

This is the most performance-critical code path. When a producer vertex sends data to a consumer vertex via a SCATTER_GATHER edge:

Producer side (OrderedPartitionedKVOutput):

Records from Processor
  → Partition (by key, using configured Partitioner)
  → Sort (in-memory sort buffer)
  → Optional combine (reduce locally before shipping)
  → Spill to disk when buffer full
  → Merge spills into a single sorted, partitioned file

Consumer side (OrderedGroupedKVInput):

Fetch partition data from all producer tasks (HTTP)
  → Merge sorted streams (k-way merge)
  → Present to Processor as a sorted key-value stream

This is conceptually the same as MapReduce's shuffle, but with two critical improvements:

1. No HDFS materialization — intermediate data goes to local disk or stays in memory
2. Pipelining — consumers can start fetching before all producers finish (configurable via slow-start)

7. The DAG Execution Lifecycle

Here's what happens when Hive submits a query via Tez:

Step 1: DAG Construction (Client Side)

Hive's query optimizer produces a physical plan. The Tez execution engine (inside Hive) translates this into a Tez DAG:

Hive Query: SELECT dept, COUNT(*) FROM employees GROUP BY dept

Tez DAG:
  Vertex "Map 1" (parallelism=10) → reads employees table from HDFS
      │
      │ SCATTER_GATHER edge (partition by dept)
      ▼
  Vertex "Reducer 2" (parallelism=4) → aggregates counts, writes to HDFS

Step 2: DAG Submission

The TezClient submits the DAG:

1. Serializes the DAG plan as a Protocol Buffer message
2. Uploads the plan + JARs + configuration to HDFS
3. Submits a YARN application (or reuses an existing Tez session)

Step 3: ApplicationMaster Initialization

The Tez AM starts on a cluster node:

1. DAGAppMaster.main() — entry point
2. Registers with YARN ResourceManager
3. Deserializes the DAG plan
4. Creates DAGImpl, which creates VertexImpl objects for each vertex

Step 4: Vertex Scheduling

The DAG starts root vertices (those with no upstream dependencies):

1. VertexImpl determines parallelism (from input splits or configured value)
2. VertexImpl creates TaskImpl objects
3. Each task requests a YARN container (with data locality preferences)
4. The TaskSchedulerManager negotiates containers from the ResourceManager

Step 5: Task Execution

For each scheduled task:

1. The AM launches the task in an allocated container
2. TezTaskRunner starts in the container's JVM
3. The runtime initializes Input → Processor → Output
4. Processor.run() executes (this is where the actual computation happens)
5. The task reports status back to the AM via heartbeats

Step 6: Inter-Vertex Data Flow

When "Map 1" tasks produce output:

1. OrderedPartitionedKVOutput writes sorted, partitioned data to local disk
2. The AM notifies "Reducer 2" that data is available
3. "Reducer 2" tasks fetch their partitions from "Map 1" tasks via HTTP
4. ShuffleManager + MergeManager handle the fetch and merge

Step 7: Completion

1. All tasks in the final vertex complete
2. Output is committed (to HDFS or wherever configured)
3. The DAG transitions to SUCCEEDED
4. The AM reports completion to the client
5. Containers are released (or held for session reuse)

8. The Shuffle Pipeline

The shuffle is the most performance-sensitive code in Tez. Here's how data moves through it in detail.

Producer Side

The flow in OrderedPartitionedKVOutput:

1. Processor writes key-value pairs
2. Each record is:
   a. Partitioned → which downstream task gets it
   b. Serialized → converted to bytes
   c. Written to a circular in-memory buffer (sort buffer)
3. When buffer reaches threshold (default 80%):
   a. Sort the buffer by (partition, key)
   b. Optionally run combiner on each partition's data
   c. Spill sorted data to local disk as an IFile
4. After all records:
   a. Merge all spill files into a single output file
   b. Create an index file mapping partition → offset in the output file
   c. Register the output with the AM

Key configuration:

Consumer Side

The flow in OrderedGroupedKVInput:

1. ShuffleManager determines which source tasks to fetch from
2. For each source task:
   a. HTTP GET to fetch this consumer's partition from the source's output
   b. If small enough → keep in memory
   c. If too large → write to local disk
3. MergeManager performs k-way merge:
   a. In-memory merge when memory segments exceed threshold
   b. On-disk merge when disk segments exceed threshold
   c. Final merge produces a single sorted stream
4. Processor reads merged sorted key-value pairs

The merge uses a priority queue (min-heap) over sorted segments — the same algorithm as MapReduce, but with better memory management and configurable thresholds.

9. Runtime Reconfiguration

One of Tez's most powerful features — and a major source of code complexity.

The Problem

When building the DAG, you often don't know the right parallelism for downstream vertices. Example: if "Map 1" produces 10 GB of shuffle data, you might want 100 reducers. If it produces 100 MB, you might want 2.

MapReduce forces you to guess at job submission time. Get it wrong, and you either waste resources (too many reducers) or create a bottleneck (too few).

How Tez Solves It

VertexManager plugins can dynamically reconfigure a vertex at runtime based on actual data from upstream:

public class ShuffleVertexManager extends VertexManagerPlugin {

    @Override
    public void onVertexManagerEventReceived(VertexManagerEvent event) {
        // Receive actual output size from upstream tasks
        actualOutputSize += event.getOutputSize();
    }

    @Override
    public void onVertexStateUpdated(VertexStateUpdate update) {
        // When enough upstream tasks have reported:
        int newParallelism = actualOutputSize / desiredTaskInputSize;
        getContext().reconfigureVertex(newParallelism, ...);
    }
}

ShuffleVertexManager (built-in) is the most important VertexManager. It:

1. Collects output-size statistics from completed upstream tasks
2. Estimates the total output size
3. Calculates the optimal parallelism for the downstream vertex
4. Reconfigures the vertex (changes task count and routing) before tasks launch

This means a Hive query doesn't need SET mapreduce.job.reduces=100 — Tez figures it out automatically.

Key Source Files

• ShuffleVertexManager.java in tez-runtime-library — the auto-parallelism logic
• VertexImpl.java — handles the reconfigure event, re-creates tasks
• VertexManager.java in tez-api — the plugin interface

10. How Hive and Pig Use Tez

Hive on Tez

When you run a Hive query with hive.execution.engine=tez:

1. Hive compiles the HiveQL into an operator tree (Scan → Filter → Join → Aggregate → File)
2. Hive's optimizer (Calcite-based) optimizes the plan
3. TezCompiler translates the operator tree into a Tez DAG:
   • Each "work" unit (a group of operators that run together) becomes a Vertex
   • Dependencies between work units become Edges
   • Shuffle boundaries (GROUP BY, JOIN, DISTRIBUTE BY) become SCATTER_GATHER edges
   • Broadcast joins become BROADCAST edges
4. TezSessionState submits the DAG to a running Tez session (or starts a new one)
5. Tez executes the DAG on YARN
6. Hive reads the output from HDFS

Example: Multi-Stage Query

SELECT d.name, COUNT(*)
FROM employees e
JOIN departments d ON e.dept_id = d.id
WHERE e.salary > 50000
GROUP BY d.name
ORDER BY COUNT(*) DESC;

As MapReduce (3 separate jobs):

Job 1: Scan employees, filter salary > 50000, scan departments → Join → HDFS
Job 2: Read join output → Group by dept name, count → HDFS
Job 3: Read aggregated output → Sort by count DESC → HDFS

As Tez (1 DAG, 4 vertices):

Vertex "Map 1" (scan employees + filter) ──┐
                                            ├─→ Vertex "Join" ─→ Vertex "GroupBy" ─→ Vertex "Sort"
Vertex "Map 2" (scan departments) ─────────┘

No HDFS writes between stages. Containers reused. Dynamic parallelism at each boundary.

Pig on Tez

Pig's integration is similar. Pig's TezCompiler converts Pig's logical plan into a Tez DAG. Each Pig operator (LOAD, FILTER, GROUP, FOREACH, STORE) maps to Tez vertices and edges.

11. Tez on YARN — The Full Picture

Session Mode vs. Non-Session Mode

Non-session mode: Each DAG submission creates a new YARN application. The AM starts, runs the DAG, and exits. Simple but has launch overhead.

Session mode: A long-lived Tez AM stays running between DAG submissions. Multiple DAGs can be submitted to the same session sequentially. This is what Hive uses for interactive queries — the first query pays the AM startup cost, but subsequent queries start immediately.

TezClient (session mode)
  → DAG 1: submitted, runs, completes
  → DAG 2: submitted immediately (no AM restart), runs, completes
  → DAG 3: ...
  → session.stop() — AM releases all containers and exits

Container Reuse

Within a session (and even within a single DAG), Tez reuses containers:

1. Task A finishes in container C on node N
2. The AM checks if any pending task prefers node N (data locality)
3. If yes: reassign container C to that task — no YARN negotiation needed
4. The container's JVM runs the new task's Input/Processor/Output

This is managed by AMContainerMap and the TaskSchedulerManager in tez-dag.

Speculative Execution

Tez supports speculative execution: if a task is running significantly slower than its peers, the AM launches a duplicate attempt on a different node. Whichever finishes first wins; the other is killed.

Controlled by: tez.am.speculation.enabled=true

12. Reading the Source: A Guided Path

If you want to understand how Tez works by reading the code, follow this sequence:

Level 1: The API (1–2 hours)

Start with the examples and the public API to understand the programming model.

1. tez-examples/WordCount.java (~200 lines) — A complete Tez application. Shows how to define vertices, edges, processors, and submit a DAG. This is the "Hello World" of Tez.

2. tez-examples/OrderedWordCount.java — Adds a second vertex for sorting. Shows a multi-vertex DAG with a SCATTER_GATHER edge.

3. tez-api/DAG.java — Read the builder methods. Clean API.

4. tez-api/Vertex.java — How vertices are configured.

5. tez-api/Edge.java + EdgeProperty.java — How edges are defined. Pay attention to DataMovementType.

Level 2: The Task Runtime (2–4 hours)

Understand what happens inside each task.

6. tez-runtime-internals/LogicalIOProcessorRuntimeTask.java — The task entry point. Follows the I/P/O lifecycle clearly.

7. tez-api/Processor.java interface — Simple: just run(Map<String, LogicalInput>, Map<String, LogicalOutput>).

8. tez-runtime-library/OrderedPartitionedKVOutput.java — The producer side of shuffle. Follow from write() through sort and spill.

9. tez-runtime-library/OrderedGroupedKVInput.java — The consumer side. Follow from initialization through fetch and merge.

Level 3: The ApplicationMaster (4–8 hours)

The hardest part. Read the state machines.

10. Generate the state diagram first:

    mvn compile -Pvisualize \
      -Dtez.dag.state.classes=org.apache.tez.dag.app.dag.impl.DAGImpl \
      -DskipTests=true
    

    Render the .gv file and keep it open as a reference.

11. tez-dag/DAGImpl.java — Focus on the StateMachineFactory at the top of the file. Each .addTransition() call defines one edge in the state diagram. Read the transition handlers to understand what happens at each step.

12. tez-dag/VertexImpl.java — The most complex file. Focus on:

    • The state machine definition
    • handleInitEvent() — how a vertex initializes
    • scheduleTasks() — how tasks are scheduled
    • reconfigureVertex() — runtime parallelism changes

13. tez-dag/TaskImpl.java — Simpler. Focus on attempt management.

14. tez-dag/TaskAttemptImpl.java — Focus on launch, completion, and failure handling.

Level 4: Container & Scheduling (Advanced)

15. tez-dag/TaskSchedulerManager.java — How the AM interacts with YARN for containers.

16. tez-dag/AMContainerMap.java — Container reuse logic.

17. tez-runtime-library/ShuffleVertexManager.java — Auto-parallelism. The onVertexManagerEventReceived() and reconfigureVertex() methods are where the magic happens.

13. Simpler Open Source Alternatives

If the Tez codebase feels overwhelming, these projects implement the same core ideas in much less code. Study them first to build intuition, then come back to Tez.

For Understanding DAG Execution Logic

Luigi (Python, by Spotify)

Repository: github.com/spotify/luigi Size: ~15,000 lines of Python Best for: Understanding DAG dependency resolution and task scheduling

Luigi implements the core job of Tez's DAG engine — scheduling tasks in dependency order, handling retries, tracking state — in readable Python. The key files:

• scheduler.py — the central scheduler that resolves dependencies
• worker.py — the worker that pulls and executes tasks
• task.py — the Task base class with requires() and run()

The requires() pattern maps directly to Tez's Edge concept — each task declares its upstream dependencies.

Dask (Python)

Repository: github.com/dask/dask Size: ~100k total, but local.py is ~400 lines Best for: Understanding the minimal DAG execution algorithm

Dask's dask/local.py contains a complete single-machine DAG executor in under 400 lines. It's the clearest possible implementation of the core algorithm:

1. Build a DAG of function calls
2. Identify tasks with no dependencies
3. Execute them
4. Remove completed tasks from the graph
5. Repeat until empty

For the distributed version, distributed/scheduler.py adds work-stealing and data locality — concepts directly relevant to Tez's TaskSchedulerManager.

Prefect (Python)

Repository: github.com/PrefectHQ/prefect Size: ~50,000 lines Best for: Modern task/flow model with clean state management

Prefect's Task and Flow abstractions map almost 1:1 to Tez's Processor and DAG. Their state machine is simpler and better documented than Tez's.

For Understanding Distributed Execution

Ray (Python / C++)

Repository: github.com/ray-project/ray Best for: Understanding distributed task execution without YARN complexity

Ray's task model (remote functions, object store, scheduling) is conceptually the closest modern equivalent to Tez. Their architecture whitepaper and documentation are excellent. The key insight Ray shares with Tez: tasks produce objects, and downstream tasks consume those objects — the system handles transfer.

Spark Core (Scala)

Repository: github.com/apache/spark Best for: Comparing Tez's approach to the main competing engine

Spark's DAGScheduler plays the same role as Tez's DAGImpl + VertexImpl. Comparing the two is illuminating:

For Understanding the Shuffle Pipeline

There's no great "simple" version of a distributed shuffle, because it's inherently complex. The most readable reference implementations:

• Spark's SortShuffleWriter and ExternalSorter — same concepts as Tez's OrderedPartitionedKVOutput, but in Scala with better comments
• Hadoop MapReduce's MapOutputBuffer — the original implementation that Tez's shuffle is based on (in hadoop-mapreduce-client-core)

Summary Comparison Table

Recommended Learning Path

1. Read Dask local.py — understand the core algorithm (1 hour)
2. Read Luigi's scheduler.py — add dependency resolution and retry (half day)
3. Read Tez tez-examples/WordCount.java — see how Tez exposes the DAG API (1 hour)
4. Read Tez tez-api/DAG.java + Vertex.java + Edge.java — the user-facing API (2 hours)
5. Generate Tez's state diagram — visualize the AM's state machine (30 min)
6. Read Tez DAGImpl.java with the state diagram open — the AM core (half day)
7. Read Tez OrderedPartitionedKVOutput.java — the shuffle producer (half day)

14. References

Official Resources

• Apache Tez Website: https://tez.apache.org
• GitHub Repository: https://github.com/apache/tez
• Tez Design Documents: https://cwiki.apache.org/confluence/display/TEZ
• How to Contribute: https://cwiki.apache.org/confluence/display/TEZ/How+to+Contribute+to+Tez

Key Papers & Talks

• "Apache Tez: Accelerating Hadoop Query Processing" — Bikas Saha, Arun Murthy (Hortonworks, 2013)
• "Hive + Tez: A Performance Deep Dive" — Jitendra Pandey, Gopal V
• InfoQ Article: "What is Apache Tez?" — https://www.infoq.com/articles/apache-tez-saha-murthy/

Source Code Reading Guides

• Tez source reading notes by @oza: https://gist.github.com/oza/470e961ff10b60778772
• BUILDING.txt in the repo: Module structure and build instructions

Simpler Alternatives (GitHub)

• Luigi: https://github.com/spotify/luigi
• Dask: https://github.com/dask/dask
• Prefect: https://github.com/PrefectHQ/prefect
• pydags: https://github.com/DavidTorpey/pydags
• simple-dag: https://github.com/leokster/simple_dag
• Dagu: https://github.com/dagucloud/dagu
• Ray: https://github.com/ray-project/ray