Controlling Micro-Batch Timing with trigger()

How do we configure trigger(processingTime='10 seconds') to control micro-batch execution intervals?

Production⏱ 2 Minutes⭐ 1200 XP

🤔 THE QUESTION

How do we configure trigger(processingTime='10 seconds') to control micro-batch execution intervals?

💡 WHAT IS IT?

trigger() controls the frequency at which micro-batches are planned, processed, and committed.

🎯 WHAT IS IT USED FOR?

Balancing latency SLAs against cluster resource consumption in continuous data pipelines.

💻 EXAMPLE

query = streaming_df.writeStream \
    .format("parquet") \
    .trigger(processingTime="10 seconds") \
    .option("checkpointLocation", "checkpoints/orders") \
    .option("path", "lakehouse/silver_orders") \
    .start()

🎯 Mission Objectives

Practice typing production-grade PySpark code for Controlling Micro-Batch Timing with trigger().

✓ Set micro-batch processing trigger interval
✓ Configure 10-second processing cadence
✓ Balance latency with cluster efficiency

🚀 Start Mission

← Prev (Fault-Tolerant Checkpointing with checkpointLocation)Next Lesson (Watermarking & Stateful Streaming Aggregations) →