Event-Time Watermarking with withWatermark()

How can we instruct the streaming engine to discard late-arriving events older than 10 minutes and prune state memory?

Production⏱ 3 Minutes⭐ 830 XP

🤔 THE QUESTION

How can we instruct the streaming engine to discard late-arriving events older than 10 minutes and prune state memory?

💡 WHAT IS IT?

withWatermark('event_time', '10 minutes') defines how late data can arrive before being ignored and cleans up internal state store.

🎯 WHAT IS IT USED FOR?

Preventing streaming cluster state stores from running out of memory during long-running 24/7 streaming jobs.

💻 EXAMPLE

df_watermarked = streaming_df.withWatermark("event_timestamp", "10 minutes")

Practice typing production-grade PySpark code for Event-Time Watermarking with withWatermark().