Capstone Part 7: Dimensional Aggregation and Gold Mart Creation

How can we aggregate the feature-enriched dataset into a regional daily summary gold data mart?

Production⏱ 3 Minutes⭐ 860 XP

🤔 THE QUESTION

How can we aggregate the feature-enriched dataset into a regional daily summary gold data mart?

💡 WHAT IS IT?

groupBy('region', 'event_date') with multi-metric agg() compiles business KPIs into gold analytics tables.

🎯 WHAT IS IT USED FOR?

Powering executive BI reporting tools (PowerBI, Tableau) with pre-aggregated dimensional summaries.

💻 EXAMPLE

df_gold_mart = df_features.withColumn("event_date", to_date(col("event_timestamp"))).groupBy("region", "event_date").agg(count("event_id").alias("total_transactions"), sum("amount").alias("total_revenue"), avg("amount").alias("avg_order_value"), count(when(col("risk_level") == "HIGH", 1)).alias("high_risk_count"))

🎯 Mission Objectives

Practice typing production-grade PySpark code for Capstone Part 7: Dimensional Aggregation and Gold Mart Creation.

✓ Gold data mart aggregation
✓ Multi-metric agg()
✓ Regional KPI computation

🚀 Start Mission

← Prev (Capstone Part 6: Window Feature Engineering for Analytics)Next Lesson (Capstone Part 8: Partition Optimization and Compaction) →