Skip to main content
DatabricksPySparkSnowflakeApache Airflow
Code Keys/PySpark Academy/Module 28/Lesson 224
PYSPARK • LESSON 224

Highest Priority Deduplication

How can we resolve duplicate records by selecting the record with the highest priority source system?

Advanced⏱ 3 Minutes⭐ 830 XP
🤔 THE QUESTION

How can we resolve duplicate records by selecting the record with the highest priority source system?

💡 WHAT IS IT?

Ordering by a custom priority integer or source confidence score inside a Window function resolves duplicate conflicts deterministically.

🎯 WHAT IS IT USED FOR?

Master Data Management (MDM) merging customer records from multiple enterprise CRM systems.

💻 EXAMPLE
w = Window.partitionBy("account_id").orderBy(col("source_priority").desc(), col("last_modified").desc())
df_master = df.withColumn("rank", row_number().over(w)).filter(col("rank") == 1).drop("rank")

🎯 Mission Objectives

Practice typing production-grade PySpark code for Highest Priority Deduplication.

  • ✓ Source system priority ranking
  • ✓ Deterministic conflict resolution
  • ✓ Master Data Management merging
🚀 Start Mission
← Prev (Latest Record Deduplication with row_number())Next Lesson (Duplicate Identification & Audit Log Extraction) →