Managing Append, Update & Complete Output Modes

How do we configure append output mode with fault-tolerant checkpointing to write new events to Parquet?

Production⏱ 2 Minutes⭐ 1150 XP

🤔 THE QUESTION

How do we configure append output mode with fault-tolerant checkpointing to write new events to Parquet?

💡 WHAT IS IT?

outputMode('append') outputs only newly arrived records to the destination sink on every micro-batch.

🎯 WHAT IS IT USED FOR?

Writing immutable append-only event logs to data lakes and cloud object stores.

💻 EXAMPLE

query = streaming_df.writeStream \
    .outputMode("append") \
    .format("parquet") \
    .option("path", "sink/events") \
    .option("checkpointLocation", "checkpoints/events") \
    .start()

🎯 Mission Objectives

Practice typing production-grade PySpark code for Managing Append, Update & Complete Output Modes.

✓ Configure append output mode
✓ Set sink output path and checkpoint location
✓ Start fault-tolerant Parquet streaming sink

🚀 Start Mission

← Prev (Starting Streaming Queries with writeStream)Next Lesson (Fault-Tolerant Checkpointing with checkpointLocation) →