Skip to main content
DatabricksPySparkSnowflakeApache Airflow

Cross-DAG Dependencies in Apache Airflow

Connecting DAGs Without Breaking Your Platform 🔗​

The Story: When One DAG Isn’t Enough​

Imagine a growing data platform.

  • DAG A ingests raw customer data
  • DAG B transforms that data
  • DAG C trains a machine learning model

Each DAG belongs to a different team.
Each DAG runs on a different schedule.

Now comes the big question:

How do you ensure DAG B only runs after DAG A finishes — and DAG C waits for both?

This challenge is known as Cross-DAG Dependencies, and handling it poorly is one of the fastest ways to create fragile, unreliable Airflow systems.

Let’s learn how to do it the right way.

What Are Cross-DAG Dependencies?​

A Cross-DAG Dependency exists when:

One DAG depends on the successful execution or output of another DAG.

Simple Example​

  • DAG load_sales_data must finish
  • Before DAG generate_sales_report starts

Unlike task dependencies (task1 >> task2), these dependencies cross DAG boundaries.

Why Cross-DAG Dependencies Are Hard​

Cross-DAG dependencies introduce complexity because:

  • DAGs may have different schedules
  • DAGs may run in different timezones
  • Failures can cascade silently
  • Backfills become harder
  • Tight coupling hurts scalability

That’s why how you implement them matters.

Traditional Approach: ExternalTaskSensor​

How It Works​

The ExternalTaskSensor waits for a task (or DAG) in another DAG to complete.

Example: ExternalTaskSensor​

 from airflow import DAG
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.operators.python import PythonOperator
 from datetime import datetime

def process_data():
 print("Processing downstream data")

with DAG(
 dag_id="downstream_dag",
 start_date=datetime(2024, 1, 1),
 schedule="@daily",
 catchup=False
 ):
 wait_for_upstream = ExternalTaskSensor(
 task_id="wait_for_upstream_dag",
 external_dag_id="upstream_dag",
 external_task_id=None,
 mode="reschedule"
 )

 process = PythonOperator(
 task_id="process_data",
 python_callable=process_data
 )

 wait_for_upstream >> process

Input​

Input
Upstream DAG run completed

Output​

Output
Downstream DAG starts processing

Problems with ExternalTaskSensor ❌​

❌ Tight coupling between DAGs
❌ Breaks easily during backfills
❌ Sensor tasks occupy scheduler resources
❌ Hard to manage at scale

ExternalTaskSensor works — but it doesn’t scale well.

Modern Approach: Cross-DAG Dependencies with Datasets ✅​

Airflow Datasets are now the recommended way to manage Cross-DAG dependencies.

Instead of waiting on a DAG, you wait on data.

Dataset-Based Cross-DAG Dependency (Best Practice)​

Step 1: Define a Shared Dataset​

 from airflow.datasets import Dataset

customer_dataset = Dataset("warehouse.curated.customers")

Step 2: Producer DAG (Publishes Dataset)​

 from airflow import DAG
 from airflow.operators.python import PythonOperator
 from datetime import datetime

def load_customers():
 print("Customer data loaded")

with DAG(
 dag_id="load_customers_dag",
 start_date=datetime(2024, 1, 1),
 schedule="@daily",
 catchup=False
 ):
 PythonOperator(
 task_id="load_customers",
 python_callable=load_customers,
 outlets=[customer_dataset]
 )

Step 3: Consumer DAG (Triggered by Dataset)​

 from airflow import DAG
 from airflow.operators.python import PythonOperator
 from airflow.datasets import Dataset
 from datetime import datetime

customer_dataset = Dataset("warehouse.curated.customers")

def transform_customers():
 print("Transforming customer data")

with DAG(
 dag_id="transform_customers_dag",
 start_date=datetime(2024, 1, 1),
 schedule=[customer_dataset],
 catchup=False
 ):
 PythonOperator(
 task_id="transform_customers",
 python_callable=transform_customers
 )

Input​

Input
Dataset warehouse.curated.customers updated

Output​

Output
Downstream DAG triggered automatically

Why Datasets Are Better Than Sensors​

AspectExternalTaskSensorDatasets
CouplingTightLoose
ScalabilityPoorExcellent
Resource UsageHighMinimal
Backfill SafetyRiskySafe
Data Awareness❌ No✅ Yes

Multi-DAG Dependency with Multiple Datasets​

A DAG can depend on multiple datasets.

 with DAG(
 dag_id="ml_training_dag",
 start_date=datetime(2024, 1, 1),
 schedule=[
 Dataset("warehouse.curated.customers"),
 Dataset("warehouse.curated.orders")
 ],
 catchup=False
 ):
 ...

Input​

Dataset
Customers updated
Orders updated

Output​

Output
ML training starts only when both are ready

When ExternalTaskSensor Is Still Acceptable​

Use it only if:

  • You are on Airflow < 2.4
  • You must wait for a specific task, not data
  • Migration to datasets is not yet possible

If you use it:
✅ Always use mode="reschedule"
✅ Set timeouts
✅ Avoid chaining sensors

Common Mistakes in Cross-DAG Dependencies​

❌ Making DAGs depend on DAG names instead of data
❌ Long sensor chains blocking workers
❌ Circular dependencies between DAGs
❌ Ignoring backfill behavior

Best Practices for Cross-DAG Dependencies​

✅ Prefer Datasets over Sensors
✅ Treat datasets as data contracts
✅ Keep DAGs loosely coupled
✅ Document dataset ownership
✅ Design DAGs to be independently runnable

Summary 🧠​

  • Cross-DAG dependencies are unavoidable in real platforms.
  • Traditional sensors work but don’t scale well.
  • Airflow Datasets provide a modern, event-driven solution.
  • Data-driven orchestration is more reliable than DAG-driven orchestration.

Key Takeaways​

  • DAGs should depend on data, not other DAGs.
  • Datasets reduce coupling and improve reliability.
  • ExternalTaskSensor is a legacy tool — use sparingly.
  • Modern Airflow architectures are dataset-first.

What’s Next?​

➡️ Custom Plugins in Apache Airflow – Hooks, Operators, and Sensors