Stream-Stream Join with Watermarking and Interval Constraints

How can we join two independent real-time streams (ad clicks and purchase conversions) with watermarks and time boundaries?

Production⏱ 3 Minutes⭐ 870 XP

🤔 THE QUESTION

How can we join two independent real-time streams (ad clicks and purchase conversions) with watermarks and time boundaries?

💡 WHAT IS IT?

Stream-stream joins require watermarks on both streams and a temporal range condition (e.g. purchase within 2 hours of click).

🎯 WHAT IS IT USED FOR?

Real-time digital marketing attribution joining impression streams with checkout event streams.

💻 EXAMPLE

clicks_wm = clicks_stream.withWatermark("click_time", "1 hour")
orders_wm = orders_stream.withWatermark("order_time", "2 hours")
df_attributed = clicks_wm.join(orders_wm, expr("clicks_wm.user_id = orders_wm.user_id AND orders_wm.order_time >= clicks_wm.click_time AND orders_wm.order_time <= clicks_wm.click_time + INTERVAL 2 HOURS"))

🎯 Mission Objectives

Practice typing production-grade PySpark code for Stream-Stream Join with Watermarking and Interval Constraints.

✓ Stream-stream join syntax
✓ Dual watermarking
✓ Time interval constraint condition

🚀 Start Mission

← Prev (Stream-Static Join for Real-Time Dimension Enrichment)Next Lesson (Custom Micro-Batch Writing with foreachBatch) →