Fact Table Enrichment with Broadcast Lookups

How do we enrich transaction facts by broadcasting product and store dimensions without incurring cluster shuffles?

Production⏱ 3 Minutes⭐ 1700 XP

🤔 THE QUESTION

How do we enrich transaction facts by broadcasting product and store dimensions without incurring cluster shuffles?

💡 WHAT IS IT?

Joining fact transactions with broadcast product and store tables to compute line item revenue in a single pass.

🎯 WHAT IS IT USED FOR?

High-throughput sales enrichment in enterprise dimensional data warehouses.

💻 EXAMPLE

from pyspark.sql.functions import broadcast, col

enriched_sales = fact_sales \
    .join(broadcast(dim_products), "product_id", "inner") \
    .join(broadcast(dim_stores), "store_id", "left") \
    .select(
        fact_sales.transaction_id,
        fact_sales.transaction_date,
        dim_products.product_name,
        dim_products.category,
        dim_stores.region,
        fact_sales.quantity,
        (fact_sales.quantity * dim_products.unit_price).alias("revenue")
    )

🎯 Mission Objectives

Practice typing production-grade PySpark code for Fact Table Enrichment with Broadcast Lookups.

✓ Broadcast product and store dimensions
✓ Calculate line-item revenue projection
✓ Produce enriched sales dataset without shuffle

🚀 Start Mission

← Prev (High-Volume Natural Key Deduplication)Next Lesson (Multi-Level Regional KPI Aggregation) →