Skip to main content
DatabricksPySparkSnowflakeApache Airflow

Unit Testing Airflow DAGs

pytest & airflow.testing Utilities 🧪🛠️​

The Story: The DAG That Failed After Deployment​

Your DAG passed code review.
It worked on your laptop.
It even ran once in staging.

Then it hit production — and failed instantly.

  • Wrong schedule
  • Missing task dependency
  • Typo in task ID
  • Broken import

All of these failures had one thing in common:

They could have been caught before deployment.

That’s why unit testing Airflow DAGs is a professional requirement, not an optional luxury.

This article explains how to unit test Airflow DAGs, what should and should not be tested, and how to integrate tests into CI/CD pipelines.

What Does “Unit Testing a DAG” Mean?​

Unit testing Airflow DAGs does not mean running full pipelines.

Instead, it means validating:

  • DAG loads without errors
  • DAG has correct schedule and configuration
  • Tasks exist and are correctly named
  • Dependencies are correctly defined
  • Task logic behaves as expected (when applicable)

📌 No scheduler, no executor, no production systems involved.

Why Unit Testing DAGs Is Critical​

Without tests, DAG issues are found:

❌ At runtime
❌ By the scheduler
❌ In production

With tests, DAG issues are found:

✅ During development
✅ In pull requests
✅ Before deployment

What Should You Test in an Airflow DAG?​

AreaTest Type
DAG loadingImport test
DAG propertiesSchedule, start_date
TasksExistence & type
DependenciesUpstream / downstream
Python logicFunction-level tests

What You Should NOT Unit Test​

❌ External systems (databases, APIs)
❌ Actual task execution in Airflow ❌ Scheduler behavior
❌ Performance and load

Unit tests validate structure and logic, not execution.

Testing Tools You’ll Use​

Core Tools​

  • pytest
  • airflow.testing
  • DagBag

Project Structure for Testing​

 airflow_project/
 ├── dags/
 │ └── sales_etl.py
 └── tests/
 └── test_sales_etl_dag.py

📌 Tests live outside the dags/ folder.

Test 1: DAG Loads Without Errors​

This is the most important test.

 from airflow.models import DagBag

def test_dag_loaded():
 dag_bag = DagBag(dag_folder="dags/", include_examples=False)
 assert len(dag_bag.import_errors) == 0

Input​

| DAG files | sales_etl.py |

Output​

ResultValue
Import errors0

❌ If this fails, your DAG will not appear in the UI.

Test 2: DAG Exists and Has Correct ID​

 def test_dag_id():
 dag_bag = DagBag()
 dag = dag_bag.get_dag("sales_etl")
 assert dag is not None

Input​

| Expected DAG ID | sales_etl |

Output​

| Result | Pass |

Test 3: Validate DAG Configuration​

 from datetime import datetime

def test_dag_schedule_and_start_date():
 dag = DagBag().get_dag("sales_etl")
 assert dag.schedule_interval == "@daily"
 assert dag.start_date == datetime(2024, 1, 1)

Input​

PropertyExpected
Schedule@daily
Start date2024-01-01

Output​

✅ Configuration matches expectation

Test 4: Validate Tasks Exist​

 def test_tasks_exist():
 dag = DagBag().get_dag("sales_etl")
 task_ids = dag.task_ids

 assert "extract_data" in task_ids
 assert "transform_data" in task_ids
 assert "load_data" in task_ids

Input​

Expected Tasks
extract_data
transform_data
load_data

Output​

| Missing tasks | None |

Test 5: Validate Task Dependencies​

 def test_task_dependencies():
 dag = DagBag().get_dag("sales_etl")

 extract = dag.get_task("extract_data")
 transform = dag.get_task("transform_data")
 load = dag.get_task("load_data")

 assert transform in extract.downstream_list
 assert load in transform.downstream_list

Input​

DependencyExpected
extract → transformTrue
transform → loadTrue

Output​

✅ DAG flow is correct

Test 6: Unit Test Python Task Logic​

Test Python logic outside of Airflow.

 from dags.sales_etl import calculate_total

def test_calculate_total():
 data = [10, 20, 30]
 result = calculate_total(data)
 assert result == 60

Input​

| Data | [10, 20, 30] |

Output​

| Result | 60 |

📌 This keeps tests fast and reliable.

airflow.testing Utilities (Advanced)​

Airflow provides helpers for DAG testing.

 from airflow.testing.utils import assert_dag_loaded

def test_dag_with_airflow_utils():
 assert_dag_loaded("sales_etl")

📌 These utilities simplify common validation patterns.

Running Tests Locally​

 pytest tests/

Output Example​

 =================== test session starts ===================
 collected 6 items

test_sales_etl_dag.py ...... [100%]

==================== 6 passed ============================

Unit Testing in CI/CD Pipelines​

Common CI flow:

 Git Push
 → Run pytest
 → Validate DAGs
 → Build image
 → Deploy Airflow

📌 Broken DAGs never reach production.

Input & Output – Realistic Testing Scenario​

Input​

  • 12 DAGs
  • 80 tasks total
  • 40 tests

Output​

ResultValue
DAG import errors0
Failed tests0
DeploymentAllowed

Common Testing Mistakes​

❌ No DAG import test
❌ Testing external systems
❌ Hardcoding dates incorrectly
❌ Mixing unit and integration tests
❌ Skipping tests due to time pressure

Best Practices for Professional Teams​

✅ Every DAG must have tests
✅ Test DAG structure first
✅ Test Python logic separately
✅ Run tests in CI
✅ Fail fast, deploy safely

Summary 🧠​

  • Unit testing prevents broken DAGs
  • Test DAG loading, structure, and logic
  • pytest + airflow.testing are sufficient
  • Tests are fast and automation-friendly
  • CI + tests = stable Airflow

Key Takeaways​

  • ✅ DAGs are code — test them
  • ✅ Import errors are critical failures
  • ✅ Dependencies must be validated
  • ✅ Python logic should be isolated
  • ✅ Testing saves production incidents

What’s Next? 🚀​

➡️ DAG Validation & Linting Tools
(Using airflow dags test, airflow dags list, and flake8 to enforce quality before runtime)