Databricks Interview Questions

Here I will be adding the questions as I learn and work on the databricks.

Question I think are important

1. Difference b/w Serverless Compute Plane and Classic Compute Plane

The primary difference between a serverless and a classic compute plane comes down to infrastructure ownership, startup latency, and billing granularity. In a classic compute plane, the virtual machines run inside my own cloud account, giving me full control over network topology and instance tuning at the cost of management overhead. In a serverless compute plane, the infrastructure runs in the vendor’s account, abstracting away the hardware layer completely so I can focus purely on data processing and transformation code.

2. What is a Databricks Workspace?

A workspace is the central collaborative environment where users organize and manage all Databricks assets -- notebooks, files, folders, dashboards, queries, libraries, experiments, and repos. It provides a unified web UI to access every Databricks feature.

3. What are the workspace folder types?

• Users/ -- Private per-user space
• Shared/ -- Team collaboration space
• Repos/ -- Git-connected folders (GitHub, Azure DevOps, GitLab)

4. What is the difference between a Workspace and a Cluster?

Workspace is the organizational layer (where you manage assets). Cluster is the compute layer (where code actually runs). A notebook in the workspace must be attached to a cluster to execute.

5. (IMP) What is the difference between %run and dbutils.notebook.run()?

6. What are notebook permission levels?

• Creators get CAN MANAGE by default
• Notebooks inherit permissions from parent folder

7. What is a Temporary View vs Global Temporary View?

8. What are Databricks Widgets? Why are they used?

Widgets add interactive input parameters to notebooks (text fields, dropdowns, comboboxes, multiselect). Used to parameterize notebooks for reusable pipelines -- e.g., passing env, date, region at runtime.

# to create a dropdown widget
dbutils.widgets.dropdown("env", "dev", ["dev", "staging", "prod"]) 

# get value from a widget
env = dbutils.widgets.get("env")

9. What are the key dbutils modules?

10. (IMP) How do you manage secrets in Databricks?

1. Create a secret scope (backed by Databricks or Azure Key Vault)
2. Store secrets in the scope
3. Access in notebooks via dbutils.secrets.get("scope", "key")
4. Secret values are redacted in notebook output -- never printed in plain text

11. What is DBFS? Is it still recommended?

DBFS (Databricks File System) is a distributed file system mounted into the workspace. It abstracts cloud storage. However, DBFS root is now considered legacy. Unity Catalog with External Locations and Volumes is the recommended approach for data access.

12. (IMP) What is the difference between mounting storage and using Unity Catalog External Locations?

13. How does Databricks integrate with Azure?

Azure Databricks is a first-party Azure service jointly developed by Databricks and Microsoft. It is deployed as an Azure resource and integrates natively with the Azure ecosystem.

14. What Azure services does Databricks integrate with?

15. (IMP) What is the difference between Azure Databricks and Databricks on AWS/GCP?

The core platform is the same. Azure-specific differences:

• Deployed via Azure Portal as a native Azure resource
• Uses ADLS Gen2 (not S3/GCS) as primary storage
• Identity via Microsoft Entra ID (not AWS IAM)
• Governance exports to Microsoft Purview
• BI via Power BI optimized connector
• Networking uses Azure VNet injection and Private Link

16. How does data flow in an Azure Databricks architecture?

Sources --> Ingestion --> Databricks (Medallion) --> Serving
  |            |              |                        |
  |-- Batch -> ADF --------> Bronze (raw)              |-> Power BI
  |-- Stream-> Event Hubs -> Silver (clean)            |-> DBSQL
  |-- CDC ---> Auto Loader-> Gold (aggregated)         |-> ML Models
                              |
                        ADLS Gen2 (Delta Lake)

17. How do you connect Azure Databricks to ADLS Gen2?

Three methods (know all three):

1. Unity Catalog External Locations (recommended) -- governed, cross-workspace
2. Service Principal + OAuth -- via Spark config with client ID/secret
3. Mount Points (legacy) -- dbutils.fs.mount() with OAuth config

18. What is the difference between a Driver Node and Worker Nodes in a Spark cluster?

Key follow-up: The driver is the single point of failure. That's why you never put it on a spot instance, and why collect() on large datasets can crash the driver (all data is pulled to one node).

19. What is the difference between All-Purpose Clusters and Job Clusters? When would you use each?

The most common follow-up: "Why not use All-Purpose for everything?"
Answer: Cost. At 3-4x the price, running production pipelines on All-Purpose clusters is the single biggest source of unnecessary Databricks spend. All-Purpose is for humans iterating; Job Clusters are for automated workloads.

20. How does Databricks autoscaling work? What are the key considerations?

Databricks autoscaling monitors the pending task queue in Spark:

1. When pending tasks exceed available executor cores - scales up (adds worker nodes from the cloud provider or instance pool)
2. When workers sit idle beyond a threshold - scales down (removes workers)

You configure two parameters:

• Min workers: baseline floor (keeps cluster responsive, avoids cold start delays)
• Max workers: budget ceiling (caps cost)

Key considerations:

21. What factors do you consider when creating a Databricks cluster?

22. What is Photon and how does it improve performance?

Photon is Databricks' native vectorized query engine written in C++. It replaces the standard JVM-based Spark execution for SQL and DataFrame operations.

How it improves performance:

• Vectorized execution: processes data in batches (columnar) instead of row-by-row
• Native C++ execution: bypasses JVM overhead (no garbage collection, no serialization)
• Optimized for I/O: better scan, filter, and aggregation performance on Delta Lake
• Result: 2-4x faster (and up to 10x in some benchmarks) for SQL-heavy workloads

When Photon helps:

• Large aggregations, joins, and scans
• SQL Warehouse queries (Photon is enabled by default)
• DataFrame transformations on wide tables

When Photon may NOT help:

• Pure Python/RDD workloads (Photon operates at the SQL/DataFrame layer)
• Ultra-short jobs completing in under 2 seconds
• UDF-heavy pipelines (UDFs bypass Photon's optimized execution)

Cost implication: Photon uses more DBUs per hour, but jobs complete faster -- net cost is often lower.

23. What are the different SQL Warehouse types? When would you use each?

All SQL Warehouses come with Photon enabled by default.

Key differences from clusters:

• SQL Warehouses are SQL-only (no Python, Scala, R)
• Optimized for concurrent BI queries (Power BI, Tableau, Looker)
• Serverless warehouses have auto-suspend -- no idle cost when not querying

24. How would you reduce Databricks costs in a production environment?

Ordered by impact:

The first three alone typically cut costs by 50-70%.

25. Your cluster cost increased significantly last month. How would you investigate and optimize?

Step-by-step investigation:

1. Check cluster uptime -- Are all-purpose clusters running idle overnight or over weekends? This is the most common culprit.
2. Identify compute type -- Are production jobs running on All-Purpose ($0.55/DBU) instead of Job Clusters ($0.15/DBU)?
3. Review auto-termination settings -- Is it disabled or set too high (e.g., 120 min default)?
4. Inspect autoscaling config -- Are max workers set unreasonably high? Are clusters scaling to max and staying there?
5. Check instance types -- Are teams using expensive GPU or memory-optimized VMs when general-purpose would suffice?
6. Review cluster policies -- Are guardrails in place, or can any user create unrestricted clusters?
7. Look at job history -- Any failed jobs retrying repeatedly? A retry loop on a 50-node cluster burns cost fast.
8. Check tags -- Identify which team/project drove the spike.
9. SQL Warehouse usage -- Is a serverless warehouse running expensive, unoptimized queries at scale?
10. Use cloud cost tools -- Azure Cost Management / AWS Cost Explorer to trace the infrastructure portion of the bill.

Optimization actions: Apply the strategies from Q24 based on findings. Typically, enforcing auto-termination + switching to Job Clusters resolves 60-70% of cost spikes.

26. A production ETL job that usually completes in 20 minutes is now taking 2 hours. The cluster configuration hasn't changed. How would you debug this?

Systematic approach:

1. Check Spark UI -- Look at the stage timeline. Is one stage taking disproportionately long? That signals a data skew (one partition has far more data than others).

2. Check input data volume -- Did the source data grow significantly? A table that was 10 GB last week might be 100 GB now due to upstream changes.

3. Check for shuffle spills -- In the Spark UI, look for "Spill (Memory)" and "Spill (Disk)". Spills mean workers don't have enough memory for the shuffle, causing disk I/O.

4. Check for small file problem -- Too many small files in Delta Lake cause excessive task overhead. Run OPTIMIZE to compact files.

5. Check for missing VACUUM/OPTIMIZE -- Delta Lake tables accumulate old files over time. Without maintenance, read performance degrades.

6. Check upstream schema changes -- Did a column type change? Did new columns get added? Schema evolution can trigger full table scans.

7. Check cluster health -- Are any workers in an unhealthy state? Check for spot instance evictions causing task retries.

8. Check concurrent workloads -- Is another heavy job sharing the same cluster and competing for resources?

9. Compare execution plans -- Run EXPLAIN on the query and compare with a previous successful run. Look for missing predicate pushdown or broadcast join changes.

10. Check Delta Lake stats -- Run DESCRIBE DETAIL on the table. Check numFiles, sizeInBytes. If file count exploded, that's the problem.

27. Your team uses All-Purpose clusters for everything -- dev, test, and production. Management wants to cut Databricks costs by 50%. What changes would you propose?

Phased proposal:

Phase 1 -- Immediate wins (Week 1-2, ~40% savings):

• Move all scheduled/production jobs to Job Clusters ($0.15 vs $0.55/DBU = 3-4x savings)
• Set auto-termination to 15 minutes on all remaining All-Purpose clusters
• Enable Spot Instances on worker nodes for all non-critical workloads

Phase 2 -- Governance (Week 3-4, ~10-15% additional savings):

• Implement Cluster Policies restricting max workers, instance types, and enforcing auto-termination
• Migrate SQL/BI workloads to Serverless SQL Warehouses (auto-suspend = zero idle cost)
• Set up Instance Pools for dev teams needing fast startup

Phase 3 -- Optimization (Month 2, ~5-10% additional savings):

• Enable Photon on SQL-heavy pipelines
• Right-size VMs based on actual utilization metrics
• Implement tagging for cost attribution and set up alerts for budget thresholds
• Schedule dev clusters to auto-terminate outside business hours

Expected result: 50-65% cost reduction without any loss in functionality or performance.

The key insight: The bulk of savings (40%) comes from one single change -- switching production from All-Purpose to Job Clusters. Everything else is incremental.

28. How do you monitor and debug a failed Spark job in Databricks?

Follow a structured 5-step workflow:

Step 1: Find the failed run and task

• Go to Jobs UI (Lakeflow Jobs) --> click the job --> Runs tab
• Use the matrix view to spot red (failed) and grey (skipped) tasks
• Hover over the failed task for metadata (duration, cluster, error summary)

Step 2: Read the error message properly

• Click the failed task to open Task Run Details
• Don't stop at the first line -- scroll to the root cause (often buried in the stack trace)
• Classify the error: OOM? Schema drift? Permission issue? Timeout?

Step 3: Check logs

• Driver logs (stdout/stderr): Application-level errors, Python tracebacks
• Executor logs (via Spark UI --> Executors tab --> stderr): Task-level OOM, GC pressure
• Cluster Event Log: Unexpected terminations, init script failures, autoscaling issues

Step 4: Reproduce in a notebook

• Attach to the same cluster type and runtime version
• Run the failing logic in isolation
• Use count() on intermediate DataFrames to pinpoint the failing transformation
• Use df.explain(True) to inspect the execution plan

Step 5: Fix and prevent

• Fix the root cause, not the symptom
• Use "Repair Run" in Jobs UI to re-run only failed/skipped tasks
• Set up email/Slack alerts for failures
• Enable cluster log delivery to ADLS/S3 for post-mortem analysis

29. What are the key tabs in the Spark UI and what does each tell you?

The most important tab for performance debugging is Stages. Start there, then drill into Tasks for skew analysis.

30. Your Databricks job is running 4x slower in production. How would you identify and fix the bottleneck?

Systematic approach:

1. Identify the slow stage:

• Open Spark UI --> Jobs tab --> find the longest-running job
• Drill into the longest stage in that job
• Check the Task Duration Summary: is there a large gap between median and max duration?

2. Check for data skew:

• If median task = 2 sec but max task = 5 min, one partition has disproportionately more data
• Fix: Enable AQE (spark.sql.adaptive.enabled=true), salt the skewed key, or use broadcast join

3. Check for shuffle spill:

• In the Stages tab, look for "Spill (Memory)" and "Spill (Disk)"
• Fix: Increase spark.sql.shuffle.partitions, increase executor memory, or filter data earlier

4. Check for small file problem:

• Run DESCRIBE DETAIL table -- if numFiles is very high with small average size
• Fix: Run OPTIMIZE table, enable auto-optimize

5. Check input data volume:

• Compare current input size vs historical. Did source data grow significantly?
• Fix: Add incremental processing, partition pruning, or Z-Order on filter columns

6. Check execution plan:

• Run df.explain("formatted") and compare with a previous successful run
• Look for: missing predicate pushdown, SortMergeJoin where BroadcastHashJoin was expected, extra Exchange nodes

7. Check cluster health:

• Executors tab: any dead executors? High GC time? Spot evictions?
• Cluster Event Log: unexpected scaling or termination events?

31. What is a Spark Execution Plan? How do you read it?

When you write a transformation, Spark's Catalyst Optimizer transforms your code through four stages before executing:

Parsed Logical Plan --> Analyzed --> Optimized --> Physical Plan

How to view:

df.explain("simple")     # Physical plan only
df.explain("extended")   # All 4 stages
df.explain("formatted")  # Clean tree with node IDs

What to look for in the physical plan:

• BroadcastHashJoin = efficient (small table broadcast). SortMergeJoin = expensive shuffle on both sides.
• PushedFilters in FileScan = predicate pushdown is working.
• Exchange hashpartitioning = shuffle happening. Minimize these.
• *(n) prefix = whole-stage code generation (good -- operators fused into native code).
• Missing * = operators running outside codegen (often UDFs).

32. How do you identify and handle data skew in Spark?

Detection:

• Spark UI --> Stages tab --> Task Duration Summary
• If a few tasks take 10x-100x longer than the median, you have skew
• Also check: one executor with disproportionately high shuffle read

Common causes:

• Joining on a key where one value dominates (e.g., country = "US" is 90% of data)
• GroupBy on a low-cardinality column
• Null values in join keys (all nulls go to one partition)

Solutions (ordered by preference):

33. How would you debug intermittent (non-deterministic) job failures in production?

Intermittent failures are harder than consistent failures because they can't be reliably reproduced. Approach:

1. Check for spot instance evictions:

• Cluster Event Log --> look for "NODE_LOST" or "SPOT_INSTANCE_TERMINATION"
• If workers are being evicted mid-task, Spark retries tasks but some may fail
• Fix: Use on-demand for workers on critical jobs, or configure fallback to on-demand

2. Check for resource contention:

• Is the cluster shared? Other workloads might compete for memory/CPU
• Fix: Use isolated job clusters for critical pipelines

3. Check for upstream data changes:

• Source files may arrive late, be partially written, or have inconsistent schemas
• Fix: Add retry logic, data validation checks, and schema enforcement

4. Check for API/external dependency timeouts:

• Network issues, rate limiting on external APIs, database connection pool exhaustion
• Fix: Add retry with exponential backoff, increase connection timeouts

5. Check for race conditions:

• Concurrent writes to the same Delta table without proper isolation
• Fix: Use Delta Lake's ACID guarantees, avoid concurrent OPTIMIZE on the same table

6. Implement structured logging:

• Log job parameters, input record counts, and key metrics at each step
• When the failure recurs, logs provide context that the error message alone doesn't

7. Enable retry policies:

• Configure job-level retries in Workflows (e.g., 2 retries with 5-minute delay)
• This handles transient failures automatically

34. What are common executor issues and how do you troubleshoot them?

Debugging steps:

1. Spark UI --> Executors tab: Check active vs dead, GC time, shuffle metrics
2. Click a dead executor --> stderr: Read the full stack trace
3. Spark UI --> Stages tab: Find the stage with failed tasks, check task duration distribution
4. Adjust configuration:
   • spark.executor.memory -- increase for OOM
   • spark.sql.shuffle.partitions -- increase for spill
   • spark.dynamicAllocation.enabled -- disable for short jobs with frequent executor churn

35. A streaming pipeline's processing time exceeds the batch interval. How would you diagnose and fix this?

When processing time > batch interval, the pipeline falls behind and a backlog builds up. Eventually, it can crash or produce severely delayed results.

Diagnosis:

1. Spark UI --> Streaming tab: Check the Processing Time graph. If it consistently exceeds the batch interval (e.g., 10-second batch, 15-second processing), the pipeline is falling behind.
2. Check Input Rate: Is the source sending more events than expected? A sudden spike in input can overwhelm the pipeline.
3. Check Stages tab for the streaming job: Look for data skew, shuffle spill, or long-running tasks.
4. Check for micro-batch overhead: Too many small files being written per batch.

Fixes:

• Scale up: Add more workers or increase executor cores to process more partitions in parallel.
• Increase batch interval: If latency tolerance allows, increase processingTime trigger (e.g., from 10s to 30s).
• Optimize the query: Remove unnecessary shuffles/joins. Use mapPartitions instead of row-level UDFs.
• Increase shuffle partitions: Default 200 may be too few for high-volume streams.
• Enable Auto Loader with cloud files: More efficient file discovery than fileStream.
• Compact output files: Enable autoOptimize on the Delta sink to prevent small file accumulation.
• Use trigger.availableNow: For catch-up scenarios, processes all available data in one batch.

36. After deploying a code change, your Delta Lake MERGE job started failing with OOM errors. The data volume hasn't changed. What would you investigate?

The data volume hasn't changed, so the problem is in the code change or execution plan, not the data.

Investigation:

1. Compare execution plans (before vs after):

df.explain("extended")

• Did a broadcast join change to a SortMergeJoin (or vice versa)?
• Did predicate pushdown stop working due to a code change?
• Is the MERGE condition now matching more rows than intended?

2. Check the MERGE condition:

• A poorly written MERGE condition (e.g., missing a key column) can cause a cross-join, exploding the result set
• Verify the join cardinality: source.join(target, condition).count() -- is it much larger than expected?

3. Check for new columns or schema changes:

• Adding wide columns (large strings, nested structs) increases row size, which means more memory per partition
• Check if schema evolution was triggered

4. Check for changes in partitioning:

• Did the code change alter repartition() or coalesce() calls?
• Fewer partitions = larger partitions = higher memory per task

5. Check for removed filters:

• Was a filter accidentally removed? Without partition pruning, MERGE scans the entire target table.
• Run DESCRIBE DETAIL target_table and check how many files are being read.

6. Fix:

• Restore the correct MERGE condition
• Add explicit partition filters to limit the scan
• Increase spark.executor.memory as a short-term mitigation
• Run OPTIMIZE on the target table if file fragmentation is a factor

37. Your team has no visibility into why jobs fail. Management asks you to set up a monitoring strategy for Databricks. What would you propose?

A comprehensive monitoring strategy has three layers:

Layer 1: Built-in Databricks Monitoring (Day 1)

Layer 2: Centralized Logging (Week 1-2)

Layer 3: Dashboards & Alerting (Month 1)

Result: The team goes from "we don't know why jobs fail" to "we see failures in real-time, can diagnose root causes in minutes, and have historical trend data for capacity planning."

Questions which are asked in the interview

#databricks #todo #interview