Skip to main content
DatabricksPySparkSnowflakeApache Airflow

Airflow File Structure & Best Practices

Designing a Scalable Platform 🏗️📂​

The Story: When Airflow Turns into a Junk Drawer​

At first, everything is clean.

One DAG.
One folder.
Life is good.

Six months later:

  • 200 DAGs in one directory
  • Random helper scripts
  • Hardcoded configs
  • No naming standards
  • New engineers afraid to touch anything

Airflow didn’t fail — structure did.

A well-designed file structure turns Airflow from a scheduler into a maintainable data platform.

Why Airflow File Structure Matters​

Airflow continuously:

  • Scans files
  • Imports Python modules
  • Parses DAGs

Poor structure leads to:

  • Slow DAG parsing
  • Scheduler overload
  • Hard-to-debug failures
  • Unmaintainable pipelines

📌 File structure is performance, reliability, and governance combined.

Default Airflow Directory Layout​

A typical Airflow home looks like this:

 $AIRFLOW_HOME/
 ├── dags/
 ├── logs/
 ├── plugins/
 ├── airflow.cfg
 └── webserver_config.py

Each directory has a specific responsibility.

The dags/ Directory (Most Important)​

This is where Airflow looks for DAG definitions.

Basic Example​

 dags/
 ├── example_dag.py
 └── sales_etl.py
 dags/
 ├── sales/
 │ ├── sales_etl_dag.py
 │ ├── tasks.py
 │ └── helpers.py
 ├── marketing/
 │ ├── campaign_dag.py
 │ └── transformations.py
 └── common/
 ├── constants.py
 └── utils.py

📌 Rule:

Only DAG definitions should be parsed at top-level imports.

DAG Parsing Best Practices​

Airflow parses DAGs every few seconds.

Do ✅​

  • Keep imports lightweight
  • Move heavy logic inside tasks
  • Use helper modules

Avoid ❌​

  • Database connections at import time
  • API calls at import time
  • Large computations in global scope

❌ Bad Example​

 data = load_big_dataset() # runs on every parse

✅ Good Example​

 @task
 def load_data():
 return load_big_dataset()

The plugins/ Directory​

Used for custom Airflow extensions.

What Belongs Here?​

  • Custom Operators
  • Hooks
  • Sensors
  • Macros
  • UI plugins

Example Structure​

 plugins/
 ├── operators/
 │ └── custom_api_operator.py
 ├── hooks/
 │ └── custom_db_hook.py
 └── macros/
 └── date_macros.py

📌 Plugins are loaded globally — keep them clean and minimal.

The logs/ Directory​

Stores:

  • Task logs
  • Scheduler logs
  • Webserver logs

Local Logging Structure​

 logs/
 └── dag_id=example_dag/
 └── task_id=task1/
 └── execution_date=2024-01-10/

📌 In production, logs usually live in remote storage.

Configuration Files​

airflow.cfg​

Main Airflow configuration file.

Controls:

  • Executor type
  • Logging backend
  • Database connection
  • Scheduler behavior

📌 Avoid editing directly in containers — use env vars.

webserver_config.py​

Controls:

  • Authentication
  • UI behavior
  • RBAC settings

Used mostly in enterprise setups.

Environment-Based Structure (Best Practice)​

Separate environments cleanly:

 airflow/
 ├── dags/
 ├── plugins/
 ├── config/
 │ ├── dev.env
 │ ├── staging.env
 │ └── prod.env

📌 One codebase, multiple environments.

ItemConvention
DAG IDlowercase_with_underscores
Task IDverb_noun
File namessnake_case.py
Foldersdomain-based

📌 Consistency = maintainability.

Input & Output Example (Realistic)​

Input​

Team with:

  • 10 data domains
  • 20 DAGs per domain
  • Shared utilities

Output​

  • Faster DAG parsing
  • Clear ownership
  • Fewer production incidents
  • Easier onboarding

Common File Structure Mistakes​

❌ All DAGs in one folder
❌ Business logic at import time
❌ Random scripts inside dags/
❌ No naming conventions
❌ Environment-specific logic in DAG code

Best Practices Summary​

✅ Organize DAGs by domain
✅ Keep DAG files thin
✅ Move logic into tasks and helpers
✅ Use plugins for reusable components
✅ Separate config from code
✅ Structure for scale, not convenience

Summary 🧠​

  • Airflow file structure defines platform quality
  • Clean DAG folders improve performance
  • Plugins enable extensibility
  • Configuration should be environment-driven
  • Structure is critical as DAG count grows

Key Takeaways​

  • ✅ DAG parsing performance depends on structure
  • ✅ Clean folders reduce cognitive load
  • ✅ Plugins prevent code duplication
  • ✅ Naming standards improve collaboration
  • ✅ Structure is a long-term investment

What’s Next? 🚀​

➡️ Secrets Management in Airflow
Environment Variables, HashiCorp Vault, AWS Secrets Manager, and best practices for secure pipelines