Skip to main content
DatabricksPySparkSnowflakeApache Airflow

Secrets Management in Apache Airflow

Secure by Design 🔐🛡️​

The Story: The Password That Took Down Production​

Everything worked perfectly — until someone pushed a DAG to Git.

Inside the code:

 password = "SuperSecret123"

That one line caused:

  • Credential leaks
  • Repo rotation
  • Emergency access revocation
  • Production downtime

Airflow didn’t fail.
Secrets management did.

In this article, you’ll learn how Airflow handles secrets, where they should live, and how to manage them securely in real-world production systems.

What Are Secrets in Airflow?​

In Airflow, secrets include anything sensitive, such as:

  • Database passwords
  • API keys
  • Cloud credentials
  • Tokens
  • Certificates

They are commonly used by:

  • Connections
  • Variables
  • Hooks
  • Operators

📌 Rule #1:

Secrets must never live in DAG code.

How Airflow Uses Secrets Internally​

Airflow retrieves secrets at runtime, not parse time.

 Task starts
 → Airflow requests secret
 → Secret backend responds
 → Task uses secret
 → Secret never written to logs or DB

📌 This design allows secure rotation without code changes.

Secret Storage Options in Airflow​

Airflow supports multiple secret backends, in this priority order:

  1. Environment Variables
  2. Secrets Backend (Vault, AWS, GCP, Azure)
  3. Metadata Database (fallback)

1️⃣ Environment Variables (Simple but Limited)​

How It Works​

Secrets are injected as environment variables.

 export AIRFLOW_CONN_POSTGRES_DB=postgresql://user:pass@host:5432/db

Airflow automatically detects:

  • AIRFLOW_CONN_*
  • AIRFLOW_VAR_*

Example: Using an Environment Variable Connection​

Input​

 AIRFLOW_CONN_MY_API=http://user:token@api.example.com

DAG Code​

 from airflow.hooks.base import BaseHook

conn = BaseHook.get_connection("my_api")
 print(conn.host)

Output​

 api.example.com

Pros & Cons​

✅ Easy to set up
✅ Good for local/dev
❌ Hard to rotate
❌ Not scalable
❌ Visible to system users

📌 Use only for development or small setups.

2️⃣ HashiCorp Vault (Enterprise-Grade)​

Why Vault?​

Vault provides:

  • Dynamic secrets
  • Automatic rotation
  • Fine-grained access control
  • Audit logs

📌 Best for large organizations.

Vault Integration Overview​

 [secrets]
 backend = airflow.providers.hashicorp.secrets.vault.VaultBackend
 backend_kwargs = {
 "connections_path": "airflow/connections",
 "variables_path": "airflow/variables",
 "url": "http://vault:8200",
 "token": "VAULT_TOKEN"
 }

Example Vault Secret​

Input (Stored in Vault)​

 {
 "conn_type": "postgres",
 "host": "db.prod",
 "login": "airflow",
 "password": "vault-generated-pass",
 "schema": "analytics"
 }

Output in Airflow​

  • Used dynamically
  • Never stored in metadata DB

📌 Vault secrets can rotate without restarting Airflow.

3️⃣ AWS Secrets Manager (Cloud-Native Favorite)​

Why AWS Secrets Manager?​

  • Automatic rotation
  • IAM-based access
  • Native AWS integration
  • High availability

📌 Best for AWS-based Airflow.

Configuration Example​

 [secrets]
 backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
 backend_kwargs = {
 "connections_prefix": "airflow/connections",
 "variables_prefix": "airflow/variables"
 }

Example Secret in AWS​

Input (Stored in AWS)​

 {
 "username": "etl_user",
 "password": "rotated-password",
 "host": "rds.amazonaws.com",
 "port": 5432
 }

Output​

  • Automatically resolved by Airflow
  • IAM controls access
  • Rotation supported

Secrets Priority Order (Important)​

Airflow resolves secrets in this order:

  1. Environment Variables
  2. Secrets Backend
  3. Metadata DB

📌 This allows secure overrides without code changes.

Secrets vs Variables vs Connections​

ItemPurposeStore Secrets?
VariablesConfig values❌ Avoid
ConnectionsCredentials✅ Yes
Secrets BackendSecure storage✅ Best

📌 Never store secrets in Variables unless encrypted and unavoidable.

Input & Output Example (Secure Pipeline)​

Input​

  • Database password stored in Vault
  • DAG uses PostgresHook
 from airflow.providers.postgres.hooks.postgres import PostgresHook

hook = PostgresHook(postgres_conn_id="analytics_db")
 hook.get_conn()

Output​

  • Secure connection
  • No credentials in code
  • No secrets in logs

Common Secrets Management Mistakes​

❌ Hardcoding secrets in DAGs
❌ Storing passwords in Variables
❌ Committing .env files
❌ Logging credentials accidentally
❌ No rotation strategy

Secrets Management Best Practices​

✅ Use a secrets backend in production
✅ Apply least-privilege access
✅ Rotate secrets regularly
✅ Mask sensitive logs
✅ Separate secrets per environment
✅ Audit access to secrets

Security Checklist (Production)​

  • Secrets backend enabled
  • No secrets in DAG code
  • IAM roles / policies enforced
  • Rotation enabled
  • Logs sanitized

Summary 🧠​

  • Secrets are critical production assets
  • Airflow supports multiple secure backends
  • Environment variables are entry-level only
  • Vault and AWS Secrets Manager are production-ready
  • Proper secrets management prevents outages and breaches

Key Takeaways​

  • ✅ Never hardcode secrets
  • ✅ Use secrets backends for scale
  • ✅ Rotate credentials without redeployments
  • ✅ Protect logs and configs
  • ✅ Treat secrets as infrastructure, not code

What’s Next? 🚀​

Monitoring DAGs – SLAs, Alerts, Email, Webserver, Metrics