Skip to main content
DatabricksPySparkSnowflakeApache Airflow

Remote Logging in Apache Airflow

S3, GCS, and Azure Blob Explained ☁️📜​

The Story: Logs That Disappear​

Your DAG ran overnight.
A task failed.
You open the Airflow UI…

“Log file does not exist.”

This happens when:

  • Workers restart
  • Pods are rescheduled
  • Nodes are auto-scaled

Local logs vanish.
Remote logs survive.

Remote logging is not a feature — it’s a requirement for any serious Airflow deployment.

What Is Remote Logging in Airflow?​

Remote logging means storing Airflow task logs in durable external storage instead of local worker disks.

Supported backends:

  • Amazon S3
  • Google Cloud Storage (GCS)
  • Azure Blob Storage
  • Elasticsearch (advanced use cases)

📌 Airflow streams logs to remote storage while the task is running.

Why Remote Logging Is Mandatory in Production​

ProblemLocal LogsRemote Logs
Worker restarts❌ Lost✅ Safe
Kubernetes pods❌ Ephemeral✅ Persistent
Multi-worker access❌ Inconsistent✅ Centralized
Compliance❌ Risky✅ Auditable
Cost control❌ Disk-heavy✅ Lifecycle rules

How Remote Logging Works (Architecture)​

 Task Running on Worker
 → Log written to local buffer
 → Streamed to remote storage
 → Metadata saved in DB
 → Webserver reads from remote storage
 → Log displayed in UI

📌 Airflow UI does not read logs from workers when remote logging is enabled.

Enabling Remote Logging (Core Configuration)​

Remote logging is configured in airflow.cfg or environment variables.

 [logging]
 remote_logging = True
 remote_log_conn_id = remote_log_storage
 remote_base_log_folder = s3://my-airflow-logs

📌 Each provider uses its own connection type.

Remote Logging with Amazon S3​

Required Setup​

  • S3 bucket
  • IAM role or access keys
  • Airflow AWS connection

Example Configuration​

 [logging]
 remote_logging = True
 remote_base_log_folder = s3://airflow-prod-logs
 remote_log_conn_id = aws_default

Example Log Path in S3​

 s3://airflow-prod-logs/
 └── dag_id=sales_etl/
 └── task_id=transform/
 └── execution_date=2024-01-10/
 └── attempt=1.log

Input​

Task prints:

 print("Processing completed successfully")

Output (Log File Content)​

 [2024-01-10 02:14:32] INFO - Processing completed successfully

Remote Logging with Google Cloud Storage (GCS)​

Required Setup​

  • GCS bucket
  • Service account
  • Airflow GCP connection

Example Configuration​

 [logging]
 remote_logging = True
 remote_base_log_folder = gs://airflow-logs-prod
 remote_log_conn_id = google_cloud_default

📌 Ideal for Cloud Composer and GCP-native environments.

Remote Logging with Azure Blob Storage​

Required Setup​

  • Azure Storage account
  • Blob container
  • Airflow Azure connection

Example Configuration​

 [logging]
 remote_logging = True
 remote_base_log_folder = wasb://airflow-logs@storageaccount.blob.core.windows.net
 remote_log_conn_id = azure_blob_default

📌 Often used in Azure Managed Airflow setups.

Security Best Practices for Remote Logging​

✅ Use IAM roles / Managed identities
✅ Avoid hardcoded credentials
✅ Encrypt buckets/containers
✅ Restrict write/read permissions
✅ Mask secrets in logs

📌 Logs often contain sensitive operational data.

Log Retention with Remote Storage​

Remote logging works best when combined with lifecycle rules.

Example Retention Strategy​

AgeAction
0–30 daysHot storage
31–90 daysCold storage
>90 daysDelete

📌 Managed entirely by the storage provider.

Performance Considerations​

  • Logs are streamed asynchronously
  • Large logs can slow UI rendering
  • Excessive logging increases storage costs

📌 Use INFO level by default.

Common Mistakes​

❌ Enabling remote logging without lifecycle rules
❌ Using local logging in Kubernetes
❌ Logging secrets or tokens
❌ Storing massive logs per task
❌ Misconfigured storage permissions

Input & Output Example (Production Scale)​

Input​

  • 200 DAGs
  • 30 tasks each
  • 2 retries
  • Remote logging enabled
  • Retention: 60 days

Output​

MetricValue
Logs/day~12,000
Storage/month~150–300 GB
Data loss riskNear zero

Summary 🧠​

  • Remote logging is essential for reliability
  • S3, GCS, and Azure Blob are all first-class options
  • Logs are streamed in near real-time
  • Airflow UI reads logs from remote storage
  • Lifecycle policies control cost and retention

Key Takeaways​

  • ✅ Local logs do not scale
  • ✅ Remote logging prevents data loss
  • ✅ Cloud storage ensures durability
  • ✅ Lifecycle rules handle retention cleanly
  • ✅ Secure logs like production data

What’s Next? 🚀​

➡️ Airflow File Structure & Best Practices
Understand DAG folders, plugins, logs, configs, and how to structure Airflow like a professional platform