Reading with Schema Evolution & mergeSchema

How can we read a dataset composed of multiple Parquet files created with evolving schemas over time?

Expert⏱ 3 Minutes⭐ 860 XP

🤔 THE QUESTION

How can we read a dataset composed of multiple Parquet files created with evolving schemas over time?

💡 WHAT IS IT?

Setting option('mergeSchema', 'true') instructs Spark to compute the union schema across all partitioned Parquet files.

🎯 WHAT IS IT USED FOR?

Reading multi-year historical data lakes where new columns were added without breaking legacy queries.

💻 EXAMPLE

df_evolved = spark.read.option("mergeSchema", "true").parquet("s3://lakehouse/events_historical/")

Practice typing production-grade PySpark code for Reading with Schema Evolution & mergeSchema.