Skip to main content
PySparkSnowflakeDatabricks

Databricks Jobs — Scheduling Batch Processing

🎬 Story Time — “The ETL That Never Slept…”​

Nidhi, a data engineer at a logistics company, receives complaints from every direction.

Analytics team:

“Why is our daily ETL running manually?”

Finance:

“Why didn’t yesterday’s batch complete?”

Managers:

“Can’t Databricks run jobs automatically?”

Nidhi knows the truth:
Someone runs the ETL notebook manually every morning.

She smiles and opens Databricks.

“Time to put this into a Job and let it run like clockwork.”

That’s where Databricks Jobs come in — reliable, automated batch processing in the Lakehouse.

🚀 1. What Are Databricks Jobs?​

Databricks Jobs allow you to schedule and automate:

  • Notebooks
  • Python scripts
  • Spark jobs
  • JAR files
  • Delta Live Tables
  • ML pipelines
  • SQL tasks

Jobs ensure processing happens on schedule, with retries, alerts, logging, and monitoring — without human involvement.

🧱 2. Creating Your First Databricks Job​

Nidhi starts with a simple daily ETL.

In the Databricks Workspace:​

Workflows → Jobs → Create Job

She configures:

  • 📘 Task: notebook path (e.g., /ETL/clean_orders)
  • ⚙️ Cluster: new job cluster (cost-optimized)
  • 🕒 Schedule: daily at 1:00 AM
  • 🔁 Retries: 3 attempts
  • 🔔 Alert: email on failure

Within minutes — her ETL is automated.

🔧 3. Example: Notebook-Based ETL Job​

The ETL notebook:

df = spark.read.format("delta").load("/mnt/raw/orders")

clean_df = df \
    .filter("order_status IS NOT NULL") \
    .withColumn("cleaned_ts", current_timestamp())

clean_df.write.format("delta").mode("overwrite").save("/mnt/clean/orders")

Databricks Job runs this nightly.

⏱️ 4. Scheduling Jobs​

Databricks offers flexible scheduling:

🟦 Cron Schedule​

0 1 * * *

🟩 UI-based Scheduling​

  • Daily
  • Weekly
  • Hourly
  • Custom

🟧 Trigger on File Arrival (Auto Loader + Jobs)​

Perfect for streaming-batch hybrid architectures.

🏗️ 5. Job Clusters vs All-Purpose Clusters​

Nidhi must choose between:

✔ Job Cluster (recommended)​

  • Auto-terminated after job finishes
  • Cheaper
  • Clean environment per run
  • Best for production

✔ All-Purpose Cluster​

  • Shared
  • Not ideal for scheduled jobs
  • More expensive

She selects job clusters to cut compute waste.

🔄 6. Multi-Step ETL With Dependent Tasks​

A single Databricks Job can contain multiple tasks, such as:

  1. Extract
  2. Transform
  3. Validate
  4. Load into Delta
  5. Notify Slack

Example DAG:

extract → transform → load → validate → notify

📌 7. Retry Policies​

Batch jobs fail sometimes.

Nidhi configures:

  • 3 retries
  • 10-minute delay
  • Exponential backoff

Databricks handles failures automatically.

📤 8. Logging & Monitoring​

Databricks Jobs provide:

  • Run page logs
  • Driver and executor logs
  • Spark UI
  • Execution graphs
  • Cluster metrics

She can debug any failure easily.

📦 9. Real-World Enterprise Use Cases​

⭐ E-commerce​

Nightly ETL loading sales, product, and customer data.

⭐ Finance​

Batch jobs calculating daily P&L and risk metrics.

⭐ Manufacturing​

Daily IoT ingestion and device telemetry cleaning.

⭐ Logistics​

Route optimization pipelines.

⭐ SaaS Platforms​

Customer-level usage aggregation.

🧠 Best Practices​

  1. Use job clusters for cost efficiency
  2. Keep each task modular
  3. Add alerts for failures
  4. Store logs in DLT + Delta tables
  5. Use retries for robustness
  6. Use version-controlled notebooks/scripts
  7. Document every pipeline task

🎉 Real-World Ending — “The Batch Runs Automatically Now”​

With Databricks Jobs:

  • No more manual ETL runs
  • No more failures unnoticed
  • Costs reduced by 35% with job clusters
  • Alerts keep teams informed
  • Nidhi sleeps peacefully

Her manager says:

“This is production-grade analytics. Our pipelines finally look professional.”

📘 Summary​

Databricks Jobs enable:

  • ✔ Automated scheduling

  • ✔ Reliable batch processing

  • ✔ Multi-task workflows

  • ✔ Alerts, retries, logging

  • ✔ Cost-effective orchestration

A fundamental building block for production data pipelines on Databricks.

👉 Next Topic

Multi-Task Job Workflows — Dependencies Across Tasks