Skip to main content
DatabricksPySparkSnowflakeApache Airflow
Code Keys/PySpark Academy/Module 22/Lesson 174
PYSPARK • LESSON 174

Small-Table Broadcast Joins with broadcast()

How do we broadcast a small lookup dimension table to all executors to eliminate a costly SortMergeJoin shuffle?

Expert⏱ 2 Minutes⭐ 980 XP
🤔 THE QUESTION

How do we broadcast a small lookup dimension table to all executors to eliminate a costly SortMergeJoin shuffle?

💡 WHAT IS IT?

broadcast(smallDF) copies the small table to every executor node, transforming the join into a BroadcastHashJoin.

🎯 WHAT IS IT USED FOR?

Massively accelerating joins between massive fact tables (billions of rows) and small dimension tables.

💻 EXAMPLE
from pyspark.sql.functions import broadcast

optimized_join = large_fact.join(broadcast(small_dim), "country_code", "inner")

🎯 Mission Objectives

Practice typing production-grade PySpark code for Small-Table Broadcast Joins with broadcast().

  • ✓ Import broadcast function
  • ✓ Wrap small dimension DataFrame in broadcast()
  • ✓ Execute zero-shuffle BroadcastHashJoin
🚀 Start Mission
← Prev (Reducing Partition Overhead with coalesce())Next Lesson (Inspecting Physical Execution Plans with explain()) →