Watermarking & Stateful Streaming Aggregations

How do we define a 10-minute watermark to drop late data and count events in 5-minute tumbling windows?

Production⏱ 3 Minutes⭐ 1250 XP

🤔 THE QUESTION

How do we define a 10-minute watermark to drop late data and count events in 5-minute tumbling windows?

💡 WHAT IS IT?

withWatermark() specifies how late data can arrive before being discarded from in-memory state stores.

🎯 WHAT IS IT USED FOR?

Real-time windowed metrics, DDoS detection, and rolling traffic volume monitoring.

💻 EXAMPLE

from pyspark.sql.functions import col, window

windowed_counts = streaming_df \
    .withWatermark("event_time", "10 minutes") \
    .groupBy(window(col("event_time"), "5 minutes"), col("device_type")) \
    .count()

🎯 Mission Objectives

Practice typing production-grade PySpark code for Watermarking & Stateful Streaming Aggregations.

✓ Define 10-minute watermark on event_time
✓ Group by 5-minute tumbling window
✓ Execute stateful streaming aggregation

🚀 Start Mission

← Prev (Controlling Micro-Batch Timing with trigger())Next Lesson (Stream-Static Joins for Real-Time Enrichment) →