Monitoring & Debugging - Spark UI, execution plans, debugging failed jobs

The Debugging Mental Model

Before diving into individual tools, anchor yourself in this hierarchy:

Cluster --> Driver --> Executor --> Stage --> Task --> Table (Delta)

Rule of thumb:

• Debug top-down (start broad, narrow down)
• Optimize bottom-up (fix root causes at the task level)

Spark UI

The Spark UI is the primary tool for understanding job performance. It gives you a live, structured view of every job, stage, and task running on the cluster.

How to access: Compute page --> Select your cluster --> Spark UI tab

Spark UI Tabs

Jobs Tab - What to Look For

• Job duration: Which job is taking the longest?
• Status: Failed jobs show red -- click to drill into the failing stage.
• Number of stages: Many stages may indicate excessive shuffles from wide transformations (joins, groupBy, repartition).

Stages Tab - What to Look For

This is where most debugging happens.

Tasks Tab - What to Look For

Drill into a specific stage to see individual task metrics.

Executors Tab - What to Look For

DAG Visualization

• Available in the Jobs tab (click a job) or SQL tab.
• Shows the directed acyclic graph of transformations.
• Narrow stages (map, filter): no shuffle, fast.
• Wide stages (join, groupBy, repartition): shuffle boundary, expensive.
• Look for unnecessarily wide stages and redundant shuffles.

Execution Plans

What is an Execution Plan?

When you write a DataFrame transformation or SQL query, Spark does not execute it immediately (lazy evaluation). The Catalyst Optimizer transforms your code through multiple stages before execution:

Your Code --> Parsed Logical Plan --> Analyzed --> Optimized --> Physical Plan --> Execution

Catalyst Optimizer Pipeline

EXPLAIN Modes

# PySpark
df.explain("simple")     # Only physical plan (default)
df.explain("extended")   # All 4 plans: parsed, analyzed, optimized, physical
df.explain("codegen")    # Physical plan + generated Java code
df.explain("cost")       # Optimized plan with CBO cost estimates
df.explain("formatted")  # Clean tree with node IDs and splits

-- SQL
EXPLAIN SELECT * FROM table;
EXPLAIN EXTENDED SELECT * FROM table;
EXPLAIN FORMATTED SELECT * FROM table;

Key Things to Look For in Execution Plans

Example: Reading an Execution Plan

df = spark.read.table("sales")
result = df.filter(df.region == "US").groupBy("product").sum("amount")
result.explain("extended")

Output walkthrough:

1. Parsed: Unresolved sales, region, product, amount
2. Analyzed: Resolved to catalog table with typed columns
3. Optimized: Filter pushed down to scan, only product and amount columns read (column pruning)
4. Physical: FileScan with pushed filters --> HashAggregate (partial) --> Exchange hashpartitioning --> HashAggregate (final)

Debugging Failed Jobs

Step-by-Step Debugging Workflow

1. Find the failed run/task
       |
2. Read the error message
       |
3. Check driver and executor logs
       |
4. Reproduce in a notebook
       |
5. Fix and prevent

Step 1: Find the Failed Run

• Go to Jobs UI (Lakeflow Jobs) --> Click the job name --> Runs tab
• The matrix view shows a history grid of all task runs -- red cells = failures
• Hover over a failed task to see: start/end time, duration, cluster name, error summary
• Click the failed task to open Task Run Details with full error message and logs

Step 2: Read the Error Message

Step 3: Check Logs

Step 4: Reproduce in a Notebook

• Attach to the same cluster type and runtime version
• Run the failing logic in isolation
• Use count() on intermediate DataFrames to identify which transformation fails
• Use df.explain(True) to check the execution plan

Step 5: Fix and Prevent

• Fix the root cause (not just the symptom)
• Repair run: In the Jobs UI, click "Repair Run" to re-run only failed/skipped tasks (no need to re-run successful tasks)
• Add monitoring: Set up email/Slack alerts for job failures
• Add data quality checks: Use DLT expectations or custom assertions
• Persist logs: Configure cluster log delivery to ADLS/S3 for post-mortem analysis

Log Types in Databricks

Cluster Event Logs

Driver Logs

• stdout: Application output, print() statements
• stderr: Python exceptions, tracebacks, Spark errors
• log4j: Spark internal messages (INFO, WARN, ERROR)

The driver log is where you'll find the root cause error message for most job failures.

Executor Logs

• Task-level errors and stack traces
• OOM kills, GC overhead, shuffle failures
• Each executor has its own stderr/stdout
• Access via: Spark UI --> Executors tab --> click executor ID --> stderr

Cluster Log Delivery

For post-mortem analysis after cluster termination, configure log delivery to cloud storage:

• Azure: ADLS Gen2 or Azure Blob Storage
• AWS: S3 bucket
• Logs are organized by cluster ID and date
• Not supported on serverless compute (at the time of writing)

Best practice: Always enable log delivery for production clusters.

Common Debugging Patterns

1. Data Skew

Symptom: A few tasks take 10x-100x longer than the rest in the Stages tab.

Detection:

• Spark UI --> Stages --> Task Duration Summary: large gap between median and max
• One executor with disproportionately high shuffle read

Solutions:

• Salting: Add a random prefix to the skewed key to distribute data evenly
• Broadcast join: If the smaller table fits in memory (< 10MB default, configurable)
• Adaptive Query Execution (AQE): Enable spark.sql.adaptive.enabled=true (auto-handles skew in Spark 3.0+)
• Repartition: Increase spark.sql.shuffle.partitions to create smaller partitions

2. Shuffle Spill

Symptom: "Spill (Memory)" and "Spill (Disk)" metrics visible in the Stages tab.

Solutions:

• Increase spark.sql.shuffle.partitions (default 200 -- try 400-2000 for large datasets)
• Increase executor memory
• Reduce data volume before shuffle (filter early, aggregate early)

3. Small File Problem

Symptom: Stage has thousands of tasks but each processes only a few KB of data.

Detection: DESCRIBE DETAIL table shows high numFiles with small sizeInBytes.

Solutions:

• Run OPTIMIZE table to compact small files into larger ones
• Enable Auto Optimize: delta.autoOptimize.optimizeWrite = true
• Increase batch size for streaming ingestion

4. OOM Errors

5. Straggler Tasks

Symptom: Job mostly complete but waiting for 1-2 slow tasks.

Solutions:

• Enable speculative execution: spark.speculation=true (re-launches slow tasks on other executors)
• Check for data skew (the most common cause)
• Check for spot instance evictions causing task retries

Symptom-to-Tool Quick Reference

Python Logging Best Practice in Databricks

For production pipelines, use Python's logging module instead of print():

import logging

logger = logging.getLogger("my_pipeline")
logger.setLevel(logging.INFO)

# File handler -- persist logs to Unity Catalog Volume
file_handler = logging.FileHandler("/Volumes/catalog/schema/volume/pipeline.log")
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
file_handler.setFormatter(formatter)

# Avoid duplicate handlers in interactive notebooks
if not logger.hasHandlers():
    logger.addHandler(file_handler)
    logger.addHandler(logging.StreamHandler())  # Also print to stdout

logger.info("Pipeline started")
logger.warning("Schema drift detected in source table")
logger.error("Failed to write to target table", exc_info=True)

Why not print()?

• No log levels (can't distinguish INFO vs ERROR)
• No timestamps or module names
• Can't route to files or external systems
• Not production-grade

Interview Question

#databricks #spark