How can we group clickstream events into distinct browsing sessions based on a 30-minute inactivity threshold?
Using lag() to identify time gaps > 1800s, flagging new sessions, and calculating cumulative sums assigns unique session IDs.
Digital marketing attribution, user session analysis, and web traffic analytics.
w = Window.partitionBy("user_id").orderBy("timestamp")
df_flag = df.withColumn("is_new_session", when(col("timestamp").cast("long") - lag("timestamp", 1).over(w).cast("long") > 1800, 1).otherwise(0))
df_sessions = df_flag.withColumn("session_id", concat(col("user_id"), lit("_"), sum("is_new_session").over(w)))Practice typing production-grade PySpark code for Session Boundary Detection & Sessionization.