Skip to main content
PYSPARK • LESSON 79

Filtering Prior to Aggregation

How do we filter for shipped orders before calculating realized net revenue?

Intermediate2 Minutes420 XP
🤔 THE QUESTION

How do we filter for shipped orders before calculating realized net revenue?

💡 WHAT IS IT?

Chaining filter() before aggregate select() to restrict metric calculations to valid status subsets.

🎯 WHAT IS IT USED FOR?

Realized financial revenue accounting, excluding pending or cancelled orders from metrics.

💻 EXAMPLE
from pyspark.sql.functions import col, sum

df.filter(col("status") == "SHIPPED") \
  .select(sum(col("amount")).alias("realized_revenue")) \
  .show()

🎯 Mission Objectives

Practice typing production-grade PySpark code for Filtering Prior to Aggregation.

  • Filter shipped status first
  • Compute aggregate sum on filtered subset
  • Display realized revenue