How do we deduplicate duplicate orders by partitioning on order_id and keeping the latest updated_at record?
Partitioning by natural business key, ordering by update timestamp descending, and filtering for row_number == 1.
Resolving duplicate event deliveries from Kafka/Kinesis and reconstructing current entity state.
from pyspark.sql.window import Window
from pyspark.sql.functions import col, row_number
dedup_window = Window.partitionBy("order_id").orderBy(col("updated_at").desc())
deduped_df = raw_orders \
.withColumn("rn", row_number().over(dedup_window)) \
.filter(col("rn") == 1) \
.drop("rn")Practice typing production-grade PySpark code for Natural Key Deduplication with Window row_number().