Reducing Partition Overhead with coalesce()

How do we reduce 200 small partitions down to 4 files before writing without triggering an expensive shuffle?

Expert⏱ 2 Minutes⭐ 960 XP

🤔 THE QUESTION

How do we reduce 200 small partitions down to 4 files before writing without triggering an expensive shuffle?

💡 WHAT IS IT?

coalesce(numPartitions) merges adjacent partitions on the same worker without performing a full shuffle.

🎯 WHAT IS IT USED FOR?

Preventing the 'small files problem' in data lakes when writing out filtered or aggregated DataFrames.

💻 EXAMPLE

df_coalesced = df.coalesce(4)

Practice typing production-grade PySpark code for Reducing Partition Overhead with coalesce().