How can we read thousands of small streaming output files and compact them into a clean set of optimized 128MB files?
Reading a directory of micro-files and writing back with coalesce() or repartition() compacts file fragmentation.
Scheduled lakehouse maintenance jobs eliminating metastore overhead and slow S3 directory listing latencies.
df_fragmented = spark.read.parquet("s3://lakehouse/streaming_raw/")
df_compacted = df_fragmented.coalesce(10)
df_compacted.write.mode("overwrite").parquet("s3://lakehouse/compacted_raw/")Practice typing production-grade PySpark code for Solving the Small File Problem via Compaction.