Production Customer Behavior Window Pipeline

How can we assemble running totals, order frequencies, previous transaction dates, and tier ranks in a single analytics pipeline?

Advanced⏱ 3 Minutes⭐ 890 XP

🤔 THE QUESTION

How can we assemble running totals, order frequencies, previous transaction dates, and tier ranks in a single analytics pipeline?

💡 WHAT IS IT?

Sharing identical Window specifications across multiple column expressions maximizes Catalyst efficiency and minimizes shuffling.

🎯 WHAT IS IT USED FOR?

Feature engineering pipelines for customer churn and lifetime value machine learning models.

💻 EXAMPLE

w = Window.partitionBy("customer_id").orderBy("order_date")
w_cum = w.rowsBetween(Window.unboundedPreceding, Window.currentRow)
df_features = df.withColumn("lifetime_spend", sum("amount").over(w_cum)).withColumn("prev_order_date", lag("order_date", 1).over(w)).withColumn("order_sequence", row_number().over(w))

🎯 Mission Objectives

Practice typing production-grade PySpark code for Production Customer Behavior Window Pipeline.

✓ Window spec reuse
✓ Multi-metric feature engineering
✓ Production window pipeline

🚀 Start Mission

← Prev (Year-over-Year Growth with lag(12))Next Lesson (Understanding Lazy Evaluation & Actions) →