Initializing Ingestion with readStream

How do we initialize a real-time streaming DataFrame from incoming Parquet file events with explicit schema?

Production⏱ 2 Minutes⭐ 1100 XP

🤔 THE QUESTION

How do we initialize a real-time streaming DataFrame from incoming Parquet file events with explicit schema?

💡 WHAT IS IT?

spark.readStream.format('parquet').schema(schema).load(path) creates an unbound streaming DataFrame.

🎯 WHAT IS IT USED FOR?

Real-time event processing, IoT telemetry ingestion, and continuous file landing pipelines.

💻 EXAMPLE

streaming_df = spark.readStream \
    .format("parquet") \
    .schema(event_schema) \
    .load("streaming_landing/events")

Practice typing production-grade PySpark code for Initializing Ingestion with readStream.