Latest Record Deduplication with row_number()

How can we keep only the most recent record for each natural key based on an updated_at timestamp?

Advanced⏱ 3 Minutes⭐ 820 XP

🤔 THE QUESTION

How can we keep only the most recent record for each natural key based on an updated_at timestamp?

💡 WHAT IS IT?

Partitioning by key and ordering by timestamp descending with row_number() assigns rank 1 to the newest event.

🎯 WHAT IS IT USED FOR?

Change Data Capture (CDC) pipelines where entities receive multiple state updates over time.

💻 EXAMPLE

w = Window.partitionBy("customer_id").orderBy(col("updated_at").desc())
df_latest = df.withColumn("rn", row_number().over(w)).filter(col("rn") == 1).drop("rn")

🎯 Mission Objectives

Practice typing production-grade PySpark code for Latest Record Deduplication with row_number().

✓ Window partitioning by key
✓ Timestamp descending ordering
✓ Latest record selection with row_number == 1

🚀 Start Mission

← Prev (Business Key Deduplication with dropDuplicates())Next Lesson (Highest Priority Deduplication) →