Capstone Part 8: Partition Optimization and Compaction

How can we optimize output partition sizing using coalesce() to produce clean 256MB Parquet files?

Production⏱ 3 Minutes⭐ 870 XP

🤔 THE QUESTION

How can we optimize output partition sizing using coalesce() to produce clean 256MB Parquet files?

💡 WHAT IS IT?

coalesce(num_partitions) merges memory partitions to avoid the small file problem without network shuffling.

🎯 WHAT IS IT USED FOR?

Ensuring storage layout compliance before publishing tables to object storage.

💻 EXAMPLE

df_final_output = df_gold_mart.coalesce(20)

Practice typing production-grade PySpark code for Capstone Part 8: Partition Optimization and Compaction.