Compute - Clusters, Policies & Cost Optimization

Compute refers to the computing resources available to run your data engineering, data science, and analytics workloads. Every notebook, job, pipeline, or SQL query needs compute to execute.

A Spark cluster in Databricks consists of two types of nodes:

+-----------+
|  Driver   |  <-- Coordinates, plans, collects results
+-----------+
   / | \
  /  |  \
+--------+ +--------+ +--------+
| Worker | | Worker | | Worker |  <-- Execute tasks in parallel
+--------+ +--------+ +--------+

Types of Compute

Databricks broadly groups compute into three categories:

1. Classic Compute

a) All-Purpose Clusters (Interactive)

• Created manually via UI, CLI, or API
• Long-running, persistent -- stays alive until manually stopped or auto-terminated
• Supports Python, SQL, Scala, R
• Multiple users can attach simultaneously
• Use case: development, exploration, ad-hoc analysis, collaborative notebooks
• More expensive (~$0.55/DBU on Premium tier)

b) Job Clusters (Automated)

• Created automatically when a job is triggered
• Terminated automatically when the job completes
• Isolated per job run -- no shared state between jobs
• Use case: production ETL, scheduled batch processing, ML training
• Significantly cheaper (~$0.15/DBU on Premium tier)

2. SQL Warehouses

Compute resources specifically optimized for SQL analytics and BI workloads. They come with Photon enabled by default.

3. Serverless Compute

Fully managed by Databricks no cluster configuration needed. Available for:

• Notebooks
• Jobs
• Delta Live Tables (Lakeflow Declarative Pipelines)

Benefits: faster startup, automatic scaling, lower operational overhead, no infrastructure management.

Instance Pools

An Instance Pool is a set of pre-provisioned, idle VMs ("warm" instances) that clusters can draw from, reducing startup time.

Key properties:

• Minimum idle instances (always warm)
• Maximum capacity (cost cap)
• Idle instance auto-termination
• Can be shared across multiple clusters and teams

Use when: teams run frequent short-lived clusters, need fast startup, or want to share resources.

Cluster Modes

Note: High Concurrency mode is being replaced by Access Modes in newer Databricks versions.

Access Modes

Access mode determines who can use the compute and what security features are enforced:

Cluster Configuration Options

Databricks Runtime (DBR)

The runtime is the set of core components that run on your cluster. Includes Apache Spark, pre-installed libraries, and Databricks optimizations.

Photon Engine

Photon is Databricks' native vectorized query engine. It accelerates SQL and DataFrame workloads by 2-4x.

• Written in C++ for high performance
• Enabled by default on SQL Warehouses
• Optional on clusters (toggle on during creation)
• Best for: SQL-heavy workloads, large aggregations, joins, data scans
• May not benefit: simple batch ETL under 2 seconds, pure Python/RDD workloads

Node Type Selection

Autoscaling

Dynamically adds or removes worker nodes based on pending tasks in Spark's task queue.

• Min workers: baseline floor for SLA
• Max workers: budget ceiling
• When pending tasks exceed available cores, workers are added
• When workers sit idle, they are removed

Best practices:

• Narrow min-max range for latency-sensitive jobs (e.g., 4-8)
• Wider range for throughput-heavy batch (e.g., 2-50)
• Not beneficial for ultra-short jobs (startup overhead > benefit)

Auto-Termination

Automatically shuts down a cluster after a period of inactivity.

• Default: 120 minutes for all-purpose clusters
• Recommended: 10-30 minutes for interactive clusters
• Job clusters auto-terminate by design when the job completes
• Critical for cost control -- idle clusters are the #1 source of waste

Tags

Custom key-value metadata attached to clusters for:

• Cost allocation and tracking (e.g., team: analytics, env: prod)
• Governance and compliance
• Filtering in cost reports

Compute Policies

Policies are admin-defined rules that limit what users can configure when creating clusters.

Benefits

• Enforce cost controls (max nodes, instance types, auto-termination)
• Simplify cluster creation UI (fix/hide complex settings)
• Prevent over-provisioning
• Standardize configurations across teams
• Limit per-cluster maximum cost (DBUs/hour)

Policy Families (Built-in Templates)

Policies can be defined via UI dropdown menus or raw JSON. Libraries can also be attached to policies (auto-installed on cluster creation).

Pricing -- The Dual-Bill Model

Databricks cost consists of two separate bills:

Total Cost = Databricks DBU Fees + Cloud Infrastructure Fees

What is a DBU?

A Databricks Unit (DBU) is a normalized unit of processing power per hour. It is not tied to a specific VM size -- Databricks normalizes consumption across instance types.

Approximate DBU Rates (Premium Tier, 2026)

Key insight: There is a 3-4x cost gap between All-Purpose Compute and Jobs Compute. This is the single biggest lever for cost reduction.

The cloud infrastructure bill (VMs, storage, networking) is charged separately by your cloud provider (Azure, AWS, GCP).

Spot Instances

When to Use Spot Instances

• Fault-tolerant batch ETL jobs
• Non-critical dev/test clusters
• ML training with checkpointing
• Worker nodes (never use spot for the driver node)

Best Practice: Hybrid Approach

• Driver node: Always on-demand (no eviction risk)
• Worker nodes: Spot instances (cost savings, tolerable if evicted)
• Configure fallback to on-demand if spot capacity is unavailable

Cost Optimization Best Practices

Summary Comparison Table

Interview Question

#databricks