Time-Between-Events with lag()

How can we measure the duration in seconds between a user's previous action and their current action?

Advanced⏱ 3 Minutes⭐ 830 XP

🤔 THE QUESTION

How can we measure the duration in seconds between a user's previous action and their current action?

💡 WHAT IS IT?

lag('event_timestamp', 1) retrieves the preceding row's timestamp to compute exact delta intervals.

🎯 WHAT IS IT USED FOR?

Calculating user session dwell times, detecting bot activity, and measuring transaction intervals.

💻 EXAMPLE

w = Window.partitionBy("session_id").orderBy("event_timestamp")
df_dwell = df.withColumn("prev_time", lag("event_timestamp", 1).over(w)).withColumn("dwell_seconds", col("event_timestamp").cast("long") - col("prev_time").cast("long"))

🎯 Mission Objectives

Practice typing production-grade PySpark code for Time-Between-Events with lag().

✓ lag() window function
✓ Inter-event duration calculation
✓ User behavior timing analytics

🚀 Start Mission

← Prev (Top-N Ranked Records per Category)Next Lesson (Forward-Looking Analysis with lead()) →