Controlling File Sizing with maxRecordsPerFile

How can we prevent huge single output files by capping the maximum number of rows written to each file?

Expert⏱ 3 Minutes⭐ 830 XP

🤔 THE QUESTION

How can we prevent huge single output files by capping the maximum number of rows written to each file?

💡 WHAT IS IT?

The maxRecordsPerFile write option splits partition output into multiple manageable files when row count exceeds the threshold.

🎯 WHAT IS IT USED FOR?

Enforcing consistent 128MB-512MB file sizes on object storage for high-concurrency BI engine consumption.

💻 EXAMPLE

df.write.option("maxRecordsPerFile", 1000000).mode("overwrite").parquet("s3://lakehouse/bronze/transactions/")

Practice typing production-grade PySpark code for Controlling File Sizing with maxRecordsPerFile.