Writing Partitioned Parquet Directories

How do we write Parquet data partitioned into hierarchical year and month subdirectories on disk?

Expert⏱ 2 Minutes⭐ 900 XP

🤔 THE QUESTION

How do we write Parquet data partitioned into hierarchical year and month subdirectories on disk?

💡 WHAT IS IT?

partitionBy('year', 'month') writes data into directory paths like year=2026/month=08/ to enable partition pruning.

🎯 WHAT IS IT USED FOR?

Enabling downstream query engines to skip entire directories when querying specific time ranges.

💻 EXAMPLE

df.write \
  .mode("overwrite") \
  .partitionBy("year", "month") \
  .parquet("output/partitioned_sales")

Practice typing production-grade PySpark code for Writing Partitioned Parquet Directories.