11. What is PythonOperator in Apache Airflow?

Imagine your company receives a CSV file every morning.

Before loading it into a database, you need to:

• Remove duplicate rows
• Convert column names to lowercase
• Validate missing values

Instead of writing shell commands, you can execute a Python function using PythonOperator.

from airflow.operators.python import PythonOperator

def greet():
    print("Welcome to Apache Airflow!")

task = PythonOperator(
    task_id="greeting_task",
    python_callable=greet
)

Explanation

• python_callable points to the Python function.
• Airflow executes that function when the task runs.
• It is one of the most commonly used operators in Airflow.

---

12. When should you use BashOperator?

Sometimes your workflow needs to execute operating system commands instead of Python code.

Examples include:

• Copying files
• Compressing folders
• Running shell scripts
• Executing Linux commands

For these scenarios, use BashOperator.

from airflow.operators.bash import BashOperator

task = BashOperator(
    task_id="show_files",
    bash_command="ls -lh"
)

Real-World Example

Every night after generating reports:

Generate Reports
        ↓
Compress Reports
        ↓
Move Reports

The compression step can be implemented using BashOperator.

---

13. What are SQL Operators in Apache Airflow?

Many companies store data in relational databases.

After an ETL pipeline completes, you may need to:

• Create tables
• Insert records
• Update data
• Run stored procedures

Instead of writing Python database code, Airflow provides SQL Operators.

Examples include:

• SQLExecuteQueryOperator
• PostgresOperator
• MySqlOperator
• SnowflakeOperator

Example:

from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator

task = SQLExecuteQueryOperator(
    task_id="create_table",
    conn_id="postgres_default",
    sql="""
        CREATE TABLE employees(
            id INT,
            name VARCHAR(100)
        );
    """
)

Why use SQL Operators?

• Cleaner code
• Easier maintenance
• Uses Airflow Connections
• Supports multiple databases

---

14. What are Hooks in Apache Airflow?

Think of a Hook as a connector between Airflow and an external system.

Without Hooks, every task would need to manually create database connections.

Hooks simplify this process.

Common Hooks include:

Hook	Connects To
PostgresHook	PostgreSQL
MySqlHook	MySQL
S3Hook	Amazon S3
GCSHook	Google Cloud Storage
WasbHook	Azure Blob Storage

Example:

from airflow.providers.postgres.hooks.postgres import PostgresHook

hook = PostgresHook(postgres_conn_id="postgres_default")

connection = hook.get_conn()

Real-World Story

Imagine every employee has their own office key.

Instead of making a new key every day, everyone simply uses their assigned key.

Similarly, Hooks reuse existing Airflow Connections.

---

15. What are Sensors in Apache Airflow?

Sometimes a task should not execute immediately.

Instead, it must wait until something happens.

For example:

• Wait for a file to arrive
• Wait for another DAG
• Wait for an API response
• Wait for a database record

This is exactly what Sensors do.

Example:

Wait for Sales File
        ↓
Process Sales Data
        ↓
Generate Report

The processing task starts only after the file becomes available.

Common Sensors

Sensor	Purpose
FileSensor	Waits for a file
ExternalTaskSensor	Waits for another DAG
SqlSensor	Waits for database conditions
HttpSensor	Waits for an API response

---

16. What is the difference between Poke Mode and Reschedule Mode in Sensors?

Sensors continuously check whether a condition has been met.

They support two modes.

Poke Mode

Worker
   ↓
Check
Wait
Check
Wait
Check

• Worker remains occupied.
• Good for short waiting times.
• Higher resource usage.

---

Reschedule Mode

Check
 ↓
Release Worker
 ↓
Wait
 ↓
Acquire Worker Again
 ↓
Check Again

• Frees the worker while waiting.
• Better for long waiting periods.
• Improves resource utilization.

Interview Tip

Most production environments prefer Reschedule Mode because it avoids blocking worker slots unnecessarily.

---

17. What is HttpOperator?

Modern data pipelines frequently communicate with REST APIs.

Examples include:

• Weather APIs
• Payment APIs
• CRM Systems
• Internal Microservices

HttpOperator simplifies API calls.

Example:

from airflow.providers.http.operators.http import SimpleHttpOperator

task = SimpleHttpOperator(
    task_id="fetch_users",
    endpoint="/users",
    method="GET",
    http_conn_id="my_api"
)

Real-World Example

Call Weather API
        ↓
Receive JSON
        ↓
Transform Data
        ↓
Store Database

Airflow can automate this complete workflow.

---

Summary

Question	Topic
11	PythonOperator
12	BashOperator
13	SQL Operators
14	Hooks
15	Sensors
16	Poke vs Reschedule
17	HttpOperator

---

Interview Tip

A common interview question is:

"What is the difference between an Operator and a Hook?"

A simple answer is:

• Operator defines what work should be performed (for example, execute Python code or run a SQL query).
• Hook defines how Airflow connects to an external system (such as PostgreSQL, Amazon S3, or Google Cloud Storage).

Think of it this way:

• Operator = Worker
• Hook = Connection