Broadcast Hash Join Optimization

How can we eliminate expensive cluster shuffles when joining a massive fact table with a small dimension table?

Expert⏱ 3 Minutes⭐ 820 XP

🤔 THE QUESTION

How can we eliminate expensive cluster shuffles when joining a massive fact table with a small dimension table?

💡 WHAT IS IT?

broadcast(df_small) copies the small dataset to all executor nodes, turning a shuffle join into a fast local hash join.

🎯 WHAT IS IT USED FOR?

Accelerating star-schema warehouse joins by 5x-10x when dimension tables are under 100MB.

💻 EXAMPLE

df_joined = df_large_facts.join(broadcast(df_small_dim), "country_code")

Practice typing production-grade PySpark code for Broadcast Hash Join Optimization.