Skip to main content
DatabricksPySparkSnowflake

Multi-Task Job Workflows — Dependencies Across Tasks

🎬 Story Time — “One Task Fails And Everything Breaks…”​

Arjun, a senior data engineer, maintains a pipeline that:

  1. Extracts data from APIs
  2. Cleans & transforms it
  3. Loads it to Delta Lake
  4. Validates quality
  5. Sends success notifications

Unfortunately, these steps were split across five separate jobs.

When the extraction job fails, the transform job still runs.
When transformation fails, the notification job still says “pipeline completed.”

Arjun sighs:

“I need something that ties everything together… with dependencies… and intelligence.”

Enter Databricks Multi-Task Job Workflows — the Lakehouse-native orchestration layer.

🔥 1. What Are Multi-Task Job Workflows?​

A workflow in Databricks is a single job that contains multiple tasks with:

  • Task dependencies
  • Conditional logic
  • Modular execution
  • Shared compute clusters
  • Automatic DAG orchestration

Perfect for building end-to-end ETL pipelines in a single pane.

🧱 2. Creating a Multi-Task Workflow​

Arjun opens:

Workflows → Jobs → Create Job

Then clicks “Add Task” multiple times.

Example workflow:


extract → transform → load → validate → notify

Each task can be:

  • Notebook
  • Python script
  • SQL query
  • JAR
  • Delta Live Table pipeline
  • dbt task (new)
  • dbt CLI runner
  • Or a combination

🔗 3. Defining Task Dependencies​

Databricks uses a clean dependency UI:


[extract] → [transform] → [load]
↓
[validate]
↓
[notify]

A task only runs after its upstream tasks succeed.

Example:​

{
  "task_key": "transform",
  "depends_on": [{"task_key": "extract"}]
}

Dependencies can form:

  • Linear DAGs
  • Fan-in DAGs
  • Fan-out DAGs
  • Branching pipelines

🧪 4. Example: Notebook-Based Multi-Task Pipeline​

Step 1 — Extract​

df_raw = spark.read.format("json").load("/mnt/raw/api_logs/")
df_raw.write.format("delta").mode("overwrite").save("/mnt/stage/logs_raw")

Step 2 — Transform​

df = spark.read.format("delta").load("/mnt/stage/logs_raw")
df_clean = df.filter("event IS NOT NULL")
df_clean.write.format("delta").mode("overwrite").save("/mnt/clean/logs_clean")

Step 3 — Validation​

from pyspark.sql import functions as F

df = spark.read.format("delta").load("/mnt/clean/logs_clean")
if df.filter(F.col("event").isNull()).count() > 0:
    raise Exception("Data validation failed")

Step 4 — Notify​

dbutils.notebook.exit("Success: ETL Pipeline Completed")

⚙️ 5. Shared Job Cluster​

Arjun selects:

  • Job cluster (cheaper than all-purpose clusters)
  • Applies it to all tasks
  • Auto-terminate after 5 minutes

This avoids cluster spin-ups for every task.

🔄 6. Retry Logic Per Task​

Instead of retrying the whole job: Arjun can retry only the failing task.

Task-level retry settings:

  • Retry attempts
  • Backoff
  • Timeout
  • Cluster retry vs task retry

This makes workflows extremely resilient.

🧯 7. Error Handling Across Tasks​

Databricks supports:

  • ✔ Stop entire pipeline on failure

  • ✔ Run downstream tasks only if upstream succeeds

  • ✔ Add "failure notification" as a separate branch

  • ✔ On-failure triggers for Slack/email

Example branch:

validate_failed → slack_alert

🌉 8. Branching Logic Inside Workflows​

Arjun builds a logic:

high_volume → process_big_data
else → process_small_data

Branches allow conditional processing depending on:

  • Input size
  • Date
  • Event type
  • External parameters

This is Databricks' version of lightweight if-else orchestration.

📊 9. Real-World Enterprise Use Cases​

⭐ Finance​

Multi-step risk scoring → aggregation → validation → reporting.

⭐ Retail​

Daily SKU extraction → price rules → promotions → BI delivery.

⭐ Healthcare​

PHI ingestion → anonymization → validation → controlled-zone storage.

⭐ Logistics​

GPS ingest → cleaning → route clustering → ML scoring → dashboard refresh.

⭐ Manufacturing​

Sensor data → dedupe → QC → anomaly detection.

🧠 Best Practices​

  1. Keep tasks modular (single purpose per task)
  2. Use job clusters for cost control
  3. Add alerts + slack notifications
  4. Add validation task before loading curated data
  5. Use task parameters instead of hardcoding
  6. Enable run-as service principals for security
  7. Store job configs in repos for version control

🎉 Real-World Ending — “The Pipeline is Finally Smart”​

Now Arjun’s ETL:

  • understands dependencies
  • retries failures automatically
  • alerts the team instantly
  • uses clean DAG orchestration
  • cuts compute cost with shared job clusters

His manager says:

“This is the pipeline architecture we should have done years ago.”

And everyone finally stops blaming Arjun’s pipelines.

📘 Summary​

Databricks Multi-Task Job Workflows provide:

  • ✔ DAG orchestration

  • ✔ Multiple task types

  • ✔ Dependency management

  • ✔ Shared job clusters

  • ✔ Conditional branching

  • ✔ Retry & alerting

  • ✔ Production-grade pipeline automation

A core building block for enterprise-scale data workflows.

👉 Next Topic

Databricks Workflows (New) — Production Orchestration

Career
All Tutorials
Interview Q & A
quizzes