Skip to main content
DatabricksPySparkSnowflakeApache Airflow

Airflow REST API & CLI Automation

Running Airflow Like a Production Platform 🤖⚙️​

The Story: When Clicking Is No Longer an Option​

At the beginning, everything happens in the Airflow UI:

  • Trigger a DAG manually
  • Clear failed tasks
  • Pause a DAG
  • Check logs

It works… until:

  • You have hundreds of DAGs
  • Multiple environments (dev, stage, prod)
  • CI/CD pipelines deploying DAGs daily
  • External systems needing to trigger workflows

At this stage, clicking buttons is no longer professional.

Production Airflow is automated Airflow.

That’s where the REST API and CLI become essential.

Why REST API & CLI Matter in Airflow​

Using API and CLI enables:

✅ CI/CD-driven workflows
✅ Event-based DAG triggering
✅ Automated backfills
✅ Platform governance
✅ Integration with external systems

In short: Airflow becomes controllable by code.

Airflow REST API Overview​

Apache Airflow provides a stable REST API (v1) that allows you to:

  • Trigger DAG runs
  • Pause / unpause DAGs
  • Fetch DAG and task status
  • Clear task instances
  • Retrieve logs and metadata

📌 Available from Airflow 2.x onward.

Authentication Basics (High Level)​

Airflow REST API typically uses:

  • Basic Auth
  • JWT / OAuth (with RBAC)
  • Cloud provider IAM (managed Airflow)

Authentication depends on your Airflow deployment — but API usage remains consistent.

Triggering a DAG Using REST API​

Example: Trigger DAG Run​

 curl -X POST "http://localhost:8080/api/v1/dags/sales_report_dag/dagRuns" \
 -H "Content-Type: application/json" \
 -u admin:admin \
 -d '{
 "conf": {
 "region": "EU",
 "run_type": "manual"
 }
 }'

Input​

Input
DAG ID: sales_report_dag
Config: { region: "EU" }

Output​

 {
 "dag_id": "sales_report_dag",
 "dag_run_id": "manual__2024-01-10T12:00:00",
 "state": "queued"
 }

Using DAG Run Configuration in Code​

 def generate_report(**context):
 conf = context["dag_run"].conf
 region = conf.get("region", "US")
 print(f"Generating report for {region}")

📌 This enables parameterized, event-driven DAGs.

Pause & Unpause DAGs via REST API​

 curl -X PATCH "http://localhost:8080/api/v1/dags/sales_report_dag" \
 -H "Content-Type: application/json" \
 -u admin:admin \
 -d '{ "is_paused": true }'

Output​

Output
DAG paused successfully

Fetch DAG Run Status​

 curl -X GET "http://localhost:8080/api/v1/dags/sales_report_dag/dagRuns" \
 -u admin:admin

Output​

 {
 "total_entries": 1,
 "dag_runs": [
 {
 "state": "success",
 "execution_date": "2024-01-10T00:00:00"
 }
 ]
 }

Airflow CLI Overview​

The Airflow CLI is ideal for:

  • Local development
  • Debugging
  • Operational scripts
  • Admin automation

Commonly Used Airflow CLI Commands​

Trigger a DAG​

 airflow dags trigger sales_report_dag \
 --conf '{"region":"US"}'

Input​

Input
DAG ID
JSON config

Output​

Output
DAG triggered successfully

Pause / Unpause DAG​

 airflow dags pause sales_report_dag
 airflow dags unpause sales_report_dag

Clear Task Instances​

 airflow tasks clear sales_report_dag \
 --start-date 2024-01-01 \
 --end-date 2024-01-05 \
 --yes

📌 Extremely useful for automated recovery.

REST API vs CLI (When to Use What)​

Use CaseREST APICLI
CI/CD pipelines✅❌
External system triggers✅❌
Local debugging❌✅
Admin scripts⚠️✅
Cloud automation✅❌

Real-World Automation Use Cases​

✅ Trigger DAGs from web applications
✅ Auto-pause failing DAGs
✅ Release pipelines triggering backfills
✅ Monitoring systems reacting to failures
✅ Multi-environment orchestration

Common Mistakes in Airflow Automation​

❌ Hardcoding credentials
❌ No RBAC restrictions
❌ Overusing CLI in production automation
❌ Missing idempotency in API triggers

Best Practices for REST API & CLI Automation​

✅ Use REST API for external automation
✅ Secure API with RBAC & least privilege
✅ Log and monitor API-triggered DAG runs
✅ Version-control automation scripts
✅ Prefer API over database-level hacks

Summary 🧠​

  • Airflow UI is for humans — APIs are for systems.
  • REST API enables scalable, secure automation.
  • CLI is perfect for debugging and admin tasks.
  • Automation is mandatory for production-grade Airflow.

Key Takeaways​

  • Manual operations don’t scale.
  • REST API unlocks event-driven workflows.
  • CLI simplifies operational tasks.
  • Professional Airflow platforms are automated by design.

What’s Next?​

➡️ Airflow Governance & Code Review Standards (Enterprise-Ready Airflow)