Fetching Prior Row Values with lag()

How do we look back to fetch the previous order amount for each customer to compute spending changes?

Advanced⏱ 2 Minutes⭐ 630 XP

🤔 THE QUESTION

How do we look back to fetch the previous order amount for each customer to compute spending changes?

💡 WHAT IS IT?

lag(col, offset).over(windowSpec) retrieves the value of a column from offset rows prior to the current row.

🎯 WHAT IS IT USED FOR?

Calculating period-over-period growth, session duration, and customer spending deltas.

💻 EXAMPLE

from pyspark.sql.window import Window
from pyspark.sql.functions import col, lag

window_spec = Window.partitionBy("customer_id").orderBy("order_date")
df = df.withColumn("prev_order_amount", lag(col("amount"), 1).over(window_spec))

🎯 Mission Objectives

Practice typing production-grade PySpark code for Fetching Prior Row Values with lag().

✓ Import lag function
✓ Retrieve prior order amount with offset 1
✓ Enable order-over-order comparative analytics

🚀 Start Mission

← Prev (Dense Ranking without Gaps with dense_rank())Next Lesson (Looking Ahead to Next Steps with lead()) →