Bucketing Data with bucketBy() and sortBy()

How can we pre-bucket and pre-sort a table on storage so future joins on that key require zero cluster shuffling?

Expert⏱ 3 Minutes⭐ 870 XP

🤔 THE QUESTION

How can we pre-bucket and pre-sort a table on storage so future joins on that key require zero cluster shuffling?

💡 WHAT IS IT?

bucketBy(numBuckets, col).sortBy(col) pre-shuffles data during write into fixed hash buckets for instant SortMergeJoins.

🎯 WHAT IS IT USED FOR?

Optimizing massive enterprise tables joined daily on customer_id or account_id in recurrent ETL pipelines.

💻 EXAMPLE

df.write.bucketBy(16, "customer_id").sortBy("order_date").mode("overwrite").saveAsTable("default.bucketed_orders")

Practice typing production-grade PySpark code for Bucketing Data with bucketBy() and sortBy().