Deduplicating Nested Struct Sub-Records

How can we deduplicate array elements within a nested column while preserving root record structure?

Advanced⏱ 3 Minutes⭐ 880 XP

🤔 THE QUESTION

How can we deduplicate array elements within a nested column while preserving root record structure?

💡 WHAT IS IT?

array_distinct() eliminates duplicate values inside an array column without flattening the parent DataFrame.

🎯 WHAT IS IT USED FOR?

Deduplicating lists of user tags, device IDs, or product categories embedded in JSON documents.

💻 EXAMPLE

df_unique_tags = df.withColumn("tags", array_distinct(col("tags")))

Practice typing production-grade PySpark code for Deduplicating Nested Struct Sub-Records.