How can we compute the total active engagement time per user session by summing page view durations capped at 10 minutes?
Computing pageview intervals with lag(), capping outlier intervals with least(), and aggregating sums active dwell time.
Media publisher analytics calculating true content engagement and ad viewability metrics.
w = Window.partitionBy("session_id").orderBy("view_time")
df_dwell = df_views.withColumn("next_time", lead("view_time", 1).over(w)).withColumn("raw_duration", col("next_time").cast("long") - col("view_time").cast("long")).withColumn("duration_sec", when(col("raw_duration").isNull(), 30).otherwise(least(col("raw_duration"), lit(600))))
df_session_totals = df_dwell.groupBy("session_id").agg(sum("duration_sec").alias("total_session_seconds"))Practice typing production-grade PySpark code for Scenario: High-Scale Session Dwell Time Analytics.