Zero-Shuffle Partition Reduction with coalesce()

How can we reduce the number of output files before saving to Cloud Storage without causing an expensive network shuffle?

Expert⏱ 3 Minutes⭐ 840 XP

🤔 THE QUESTION

How can we reduce the number of output files before saving to Cloud Storage without causing an expensive network shuffle?

💡 WHAT IS IT?

coalesce(n) merges adjacent partitions locally on executors, reducing file counts with zero network shuffle overhead.

🎯 WHAT IS IT USED FOR?

Preventing the small file problem when saving filtered datasets to Amazon S3 or Google Cloud Storage.

💻 EXAMPLE

df_output = df_filtered.coalesce(10)
df_output.write.mode("overwrite").parquet("s3://bucket/output/")

Practice typing production-grade PySpark code for Zero-Shuffle Partition Reduction with coalesce().