18. What are XComs in Apache Airflow?

Imagine your workflow has two tasks:

Download Customer Data
          ↓
Generate Customer Report

The second task needs to know where the downloaded file is stored.

Instead of saving this information in a database or file, Airflow provides XCom (Cross Communication).

XCom allows tasks to exchange small pieces of data.

Examples include:

• File paths
• Record counts
• IDs
• Status messages
• API responses (small)

---

Example

def extract():
    return "customers.csv"

The returned value is automatically pushed into XCom.

Another task can retrieve it.

filename = ti.xcom_pull(task_ids="extract_task")

---

Interview Tip

XCom is designed for small metadata, not large datasets.

Avoid storing large DataFrames or files in XCom.

---

19. What is the difference between XCom Push and XCom Pull?

Think of two coworkers.

Employee A finishes a report.

Employee B needs that report.

Employee A Pushes the report.

Employee B Pulls the report.

Exactly the same concept exists in Airflow.

XCom Push

Stores information.

ti.xcom_push(
    key="total_sales",
    value=2500
)

---

XCom Pull

Retrieves information.

sales = ti.xcom_pull(
    task_ids="calculate_sales",
    key="total_sales"
)

---

Real-World Example

Download File
      ↓
Push File Name
      ↓
Read File Name
      ↓
Load Database

---

20. What is BranchPythonOperator?

Sometimes your workflow should follow different execution paths depending on a condition.

Example:

Receive Sales Data
        ↓
   Sales > 0 ?
     /      \
   Yes      No
    ↓        ↓
Process   Send Alert

BranchPythonOperator allows you to implement this logic.

Example

from airflow.operators.python import BranchPythonOperator

def choose_path():
    return "process_task"

branch = BranchPythonOperator(
    task_id="branch_task",
    python_callable=choose_path
)

The function returns the task ID that should execute next.

---

21. What is ShortCircuitOperator?

ShortCircuitOperator is used when you want to stop the remaining workflow if a condition is not satisfied.

Example:

Check File Exists
       ↓
True?
  |
Yes
 ↓
Process File

False
 ↓
Stop Workflow

Example

from airflow.operators.python import ShortCircuitOperator

def check_file():
    return True

task = ShortCircuitOperator(
    task_id="check_file",
    python_callable=check_file
)

If the function returns False, downstream tasks are skipped.

---

22. What are Trigger Rules in Apache Airflow?

By default, a task runs only when all upstream tasks succeed.

However, production workflows often require more flexibility.

Airflow provides Trigger Rules.

Some common trigger rules are:

Trigger Rule	Description
all_success	Run only if all upstream tasks succeed
all_failed	Run only if all upstream tasks fail
one_success	Run if at least one task succeeds
one_failed	Run if any upstream task fails
all_done	Run regardless of success or failure

---

Real-World Example

Suppose three data sources are loaded.

Sales
Inventory
Customers
     ↓
Generate Report

Even if one source fails, you might still want to generate a partial report.

Using one_success or all_done makes this possible.

---

23. What is the TaskFlow API?

Before Airflow 2.x, DAGs required a lot of boilerplate code.

TaskFlow API simplifies DAG development using Python decorators.

Traditional approach:

task = PythonOperator(
    task_id="hello",
    python_callable=my_function
)

TaskFlow approach:

from airflow.decorators import task

@task
def hello():
    print("Hello Airflow")

This style is cleaner and easier to maintain.

---

24. What is the @task decorator?

The @task decorator converts a Python function into an Airflow task.

Example:

from airflow.decorators import task

@task
def calculate_total():
    return 100

Benefits include:

• Less boilerplate code
• Automatic XCom support
• Better readability
• Easier testing
• Improved maintainability

---

25. What is Dynamic Task Mapping?

Imagine your company receives reports from 200 stores every day.

Instead of writing 200 separate tasks:

Store1
Store2
Store3
...
Store200

Airflow can automatically create tasks dynamically.

Example:

@task
def process_store(store):
    print(store)

process_store.expand(
    store=["Store1", "Store2", "Store3"]
)

Airflow automatically creates one task for each store.

This greatly reduces duplicate code.

---

26. What is Dynamic DAG Generation?

Sometimes organizations have hundreds of similar workflows.

Instead of manually creating hundreds of DAG files, you can generate them dynamically.

Example:

Suppose you have:

USA
Canada
India
Germany
Australia

Instead of writing five DAGs manually, a single Python script can generate all five.

Advantages

• Less duplicate code
• Easier maintenance
• Consistent workflow structure
• Faster onboarding of new pipelines

---

Summary

Question	Topic
18	XCom
19	XCom Push & Pull
20	BranchPythonOperator
21	ShortCircuitOperator
22	Trigger Rules
23	TaskFlow API
24	@task Decorator
25	Dynamic Task Mapping
26	Dynamic DAG Generation

---

Interview Tip

A common interview question is:

"When should you use XCom instead of a database?"

A simple answer:

• Use XCom for small pieces of metadata, such as file names, IDs, timestamps, or status values.
• Use a database, cloud storage, or data lake for large datasets, files, or DataFrames.

Keeping XCom lightweight improves Airflow's performance and keeps the metadata database efficient.