How can we extract all duplicate records (rn > 1) into an audit log table for compliance investigation?
Filtering for row_number() > 1 isolates every discarded duplicate record along with its rank for auditing.
Regulatory compliance reporting and identifying upstream double-billing or re-transmission anomalies.
w = Window.partitionBy("transaction_id").orderBy(col("received_at").desc())
df_duplicates_audit = df.withColumn("rn", row_number().over(w)).filter(col("rn") > 1)Practice typing production-grade PySpark code for Duplicate Identification & Audit Log Extraction.