Reshuffling Partitions with repartition()

How do we rebalance a dataset into 200 uniform partitions hashed by customer_id to eliminate data skew?

Expert⏱ 2 Minutes⭐ 950 XP

🤔 THE QUESTION

How do we rebalance a dataset into 200 uniform partitions hashed by customer_id to eliminate data skew?

💡 WHAT IS IT?

repartition(numPartitions, *cols) performs a full cluster shuffle to evenly redistribute records.

🎯 WHAT IS IT USED FOR?

Eliminating data skew, increasing parallelism before heavy joins, and balancing executor workloads.

💻 EXAMPLE

df_repartitioned = df.repartition(200, "customer_id")

Practice typing production-grade PySpark code for Reshuffling Partitions with repartition().