Skip to main content
DatabricksPySparkSnowflakeApache Airflow
Code Keys/PySpark Academy/Module 23/Lesson 180
PYSPARK • LESSON 180

Fault-Tolerant Checkpointing with checkpointLocation

Why and how do we configure checkpointLocation to guarantee exactly-once processing across cluster failures?

Production⏱ 2 Minutes⭐ 1180 XP
🤔 THE QUESTION

Why and how do we configure checkpointLocation to guarantee exactly-once processing across cluster failures?

💡 WHAT IS IT?

checkpointLocation persists stream read offsets, state stores, and commit logs to durable storage.

🎯 WHAT IS IT USED FOR?

Guaranteeing fault tolerance and exactly-once processing guarantees across cluster reboots and spot restarts.

💻 EXAMPLE
query = streaming_df.writeStream \
    .format("parquet") \
    .option("checkpointLocation", "s3://checkpoints/pipeline_01") \
    .option("path", "s3://silver/stream_output") \
    .start()

🎯 Mission Objectives

Practice typing production-grade PySpark code for Fault-Tolerant Checkpointing with checkpointLocation.

  • ✓ Configure durable S3 checkpointLocation
  • ✓ Enable exactly-once stream state recovery
  • ✓ Ensure fault-tolerant stream execution
🚀 Start Mission
← Prev (Managing Append, Update & Complete Output Modes)Next Lesson (Controlling Micro-Batch Timing with trigger()) →