How do we build a complete multi-tier pipeline transforming Bronze raw logs to Silver clean orders to Gold customer marts?
A multi-stage Medallion architecture: Bronze deduplication, Silver dimension broadcast join, and Gold metric aggregation.
End-to-end lakehouse architectures powering enterprise BI dashboards and operational data stores.
from pyspark.sql.window import Window
from pyspark.sql.functions import broadcast, col, count, current_timestamp, row_number, sum
# Bronze to Silver: Deduplicate & Clean
silver_orders = spark.read.parquet("lakehouse/bronze/orders") \
.filter(col("order_id").isNotNull() & (col("amount") > 0)) \
.withColumn("rn", row_number().over(Window.partitionBy("order_id").orderBy(col("updated_at").desc()))) \
.filter(col("rn") == 1) \
.drop("rn")
# Silver to Gold: Enrich & Aggregate
dim_customers = spark.read.parquet("lakehouse/silver/dim_customers")
gold_customer_metrics = silver_orders \
.join(broadcast(dim_customers), "customer_id", "inner") \
.groupBy("customer_id", "customer_tier") \
.agg(count("order_id").alias("order_count"), sum("amount").alias("total_spend")) \
.withColumn("gold_generated_at", current_timestamp())
gold_customer_metrics.write.mode("overwrite").parquet("lakehouse/gold/customer_marts")Practice typing production-grade PySpark code for Scalable Bronze-to-Gold Warehouse Transformation.