27. What is an Executor in Apache Airflow?

Imagine the Scheduler is a project manager.

The manager decides what work needs to be done, but doesn't actually perform the work.

Instead, the manager assigns tasks to employees.

In Airflow, those employees are called Executors.

An Executor determines how and where tasks are executed after the Scheduler decides they are ready.

Workflow

Scheduler
      │
      ▼
 Executor
      │
      ▼
 Worker(s)
      │
      ▼
 Execute Task

Without an Executor, the Scheduler would know when to run a task but not where to run it.

---

28. What is the difference between LocalExecutor, CeleryExecutor, and KubernetesExecutor?

Airflow provides different Executors depending on the scale of your environment.

Executor	Best For	Characteristics
LocalExecutor	Small to medium projects	Runs tasks in parallel on a single machine
CeleryExecutor	Large distributed systems	Uses multiple workers across different machines
KubernetesExecutor	Cloud-native environments	Creates a separate Kubernetes Pod for each task

LocalExecutor

Machine
 ├── Task A
 ├── Task B
 ├── Task C

Everything runs on one server.

---

CeleryExecutor

Scheduler
      │
      ▼
 Message Broker
   │     │
Worker1 Worker2 Worker3

Tasks are distributed across multiple workers.

---

KubernetesExecutor

Task A → Pod A

Task B → Pod B

Task C → Pod C

Each task gets its own isolated Kubernetes Pod.

Interview Tip

• LocalExecutor → Development or small production environments.
• CeleryExecutor → Most enterprise Airflow deployments.
• KubernetesExecutor → Dynamic cloud workloads with automatic scaling.

---

29. How do you scale Apache Airflow?

Imagine your company grows from processing:

• 10 workflows/day

to

• 10,000 workflows/day.

A single worker will no longer be enough.

Airflow supports horizontal scaling.

Some common approaches include:

• Adding more Celery workers
• Using KubernetesExecutor
• Increasing Scheduler capacity
• Increasing Parallelism
• Using multiple queues

Real-World Example

Instead of one worker:

Worker

You expand to:

Worker 1

Worker 2

Worker 3

Worker 4

Worker 5

Now many tasks execute simultaneously.

---

30. What are Pools in Apache Airflow?

Imagine a database allows only 5 simultaneous connections.

Your DAG contains 100 tasks.

If all 100 tasks connect at once, the database could become overloaded.

Airflow solves this using Pools.

A Pool limits how many tasks can access a shared resource simultaneously.

Example:

Pool Size = 5

Task1 ✔

Task2 ✔

Task3 ✔

Task4 ✔

Task5 ✔

Task6 ⏳ Waiting

Task7 ⏳ Waiting

As tasks finish, waiting tasks begin.

Pools help protect:

• Databases
• APIs
• File systems
• External services

---

31. What are Priority Weights?

Sometimes not every task has the same importance.

Imagine two workflows.

Workflow A

Payroll Processing

Workflow B

Generate Marketing Report

If workers are busy, which workflow should run first?

Most companies choose Payroll because it is business critical.

Airflow allows assigning Priority Weights.

Higher priority tasks are scheduled before lower priority tasks when resources are limited.

This helps ensure important workloads finish first.

---

32. What is Parallelism in Apache Airflow?

Parallelism determines how many tasks Airflow can execute at the same time.

Example:

Without parallelism:

Task A

↓

Task B

↓

Task C

Everything runs sequentially.

With parallelism:

Task A     Task B     Task C

All tasks execute simultaneously if there are enough worker slots.

Increasing parallelism improves throughput but also increases resource usage.

---

33. How can you reduce DAG parsing time?

The Scheduler repeatedly reads every DAG file.

If DAG files perform heavy operations during import, scheduling becomes slow.

Avoid doing things like:

import pandas as pd

df = pd.read_csv("large_file.csv")

at the global level.

Instead, place expensive operations inside task functions.

Good example:

@task
def load_data():
    import pandas as pd
    return pd.read_csv("large_file.csv")

This keeps DAG parsing lightweight.

Best Practices

• Keep global code minimal.
• Avoid API calls during import.
• Avoid database queries during import.
• Load configuration lazily.
• Import heavy libraries only when required.

---

34. What are some best practices for optimizing Python code in Airflow?

Poorly written Python code can slow down an entire Airflow deployment.

Some recommended practices include:

• Keep DAG files lightweight.
• Avoid unnecessary loops.
• Minimize network calls.
• Break large workflows into smaller tasks.
• Use efficient libraries.
• Avoid loading huge datasets into memory.
• Reuse helper functions instead of duplicating code.

Real-World Story

Instead of creating one task that processes 50 GB of data,

split it into smaller independent tasks.

Read Data

↓

Clean Data

↓

Transform Data

↓

Load Database

Smaller tasks are easier to retry, monitor, and scale.

---

Summary

Question	Topic
27	Executors
28	LocalExecutor vs CeleryExecutor vs KubernetesExecutor
29	Scaling Airflow
30	Pools
31	Priority Weights
32	Parallelism
33	Reducing DAG Parsing Time
34	Optimizing Python Code

---

Interview Tip

A very common interview question is:

"Why shouldn't you put database queries or API calls at the top level of a DAG file?"

A strong answer is:

Because the Airflow Scheduler repeatedly parses DAG files. If a DAG performs expensive operations such as database queries, API requests, or reading large files during import, it slows down the Scheduler and affects the performance of the entire Airflow environment. Heavy operations should be placed inside tasks so they execute only during task runtime.