Skip to main content
DatabricksPySparkSnowflake

Cost Optimization in Databricks — Clusters, Jobs & SQL Warehouses

Databricks is powerful, but costs can spiral if clusters, jobs, and SQL warehouses aren’t managed carefully.

This guide covers strategies for optimizing compute usage, scheduling, and storage, so data teams can maximize performance while controlling costs.

A Real-World Story​

Meet Arjun, a data engineer managing multiple Databricks workloads.

  • Jobs run overnight, consuming huge clusters
  • SQL dashboards run repeatedly, scanning full tables
  • Monthly bills surprise the management team

By applying cost optimization best practices, Arjun reduces cloud costs by 40% while maintaining query speed and reliability.

1. Cluster Optimization​

a) Use Auto-Scaling Clusters​

Min: 1 node | Max: 8 nodes
  • Automatically adjusts resources based on workload
  • Avoids over-provisioning

b) Choose Appropriate Cluster Types​

  • Light workloads → Serverless SQL warehouses or single-node clusters
  • Heavy ETL → Standard clusters with optimal cores

c) Termination Settings​

  • Set auto-termination to prevent idle compute costs
Auto-terminate after 15 minutes idle

2. Job Scheduling & Optimization​

  • Batch similar jobs to share clusters
  • Use job clusters for ephemeral workloads
  • Schedule non-critical workloads off-peak
Job A: Nightly ETL → shared cluster
Job B: Ad-hoc analytics → serverless warehouse

3. SQL Warehouse Cost Management​

  • Avoid always-on warehouses for infrequent queries
  • Use auto-stop and auto-scale features
  • Optimize queries to scan minimal data
  • Enable caching for repeated queries
CACHE TABLE silver_sales_summary;

4. Data Storage & Scan Optimization​

  • Partition and Z-Order Delta tables to reduce scan size
  • **Avoid SELECT *** on large tables
  • Use optimized file formats like Parquet

5. Monitoring & Alerts​

  • Use Databricks Cost Insights to track spending by clusters, jobs, and warehouses
  • Set alerts for unexpected spikes
  • Analyze query execution and cluster utilization for inefficiencies

Input & Output Example​

Scenario​

  • Large ETL job runs on 10-node cluster for 4 hours nightly

Optimization​

  • Auto-scaling cluster: 2–6 nodes
  • Optimized SQL queries and caching

Result​

  • Reduced runtime by 35%
  • Monthly compute cost dropped by 40%

Summary​

Cost optimization in Databricks requires holistic attention across clusters, jobs, and SQL warehouses.

Key takeaways:

  • Use auto-scaling clusters and terminate idle nodes
  • Schedule jobs strategically and share clusters where possible
  • Optimize SQL warehouses with auto-stop, caching, and query tuning
  • Partition and Z-order tables to reduce scan size
  • Monitor usage continuously with Cost Insights

Applying these practices ensures efficient, performant workloads without unexpected bills.

📌 Next Article in This Series: Query Profiling & Spark UI for Databricks SQL

Career
All Tutorials
Interview Q & A
quizzes