Optimized Partitioning, Coalesce & Parquet Export

How do we derive partition columns, coalesce partitions to prevent small files, and export to cloud storage with Snappy?

Production⏱ 3 Minutes⭐ 1850 XP

🤔 THE QUESTION

How do we derive partition columns, coalesce partitions to prevent small files, and export to cloud storage with Snappy?

💡 WHAT IS IT?

Preparing gold-layer datasets for cloud export with date partitioning, file count optimization, and Snappy compression.

🎯 WHAT IS IT USED FOR?

Exporting analytical data marts to AWS S3, Google Cloud Storage, and Snowflake external stages.

💻 EXAMPLE

from pyspark.sql.functions import col, month, year

mart_df = customer_360 \
    .withColumn("order_year", year(col("order_date"))) \
    .withColumn("order_month", month(col("order_date"))) \
    .coalesce(16)

mart_df.write \
    .mode("overwrite") \
    .partitionBy("order_year", "order_month") \
    .option("compression", "snappy") \
    .parquet("s3://warehouse/gold/customer_360")

🎯 Mission Objectives

Practice typing production-grade PySpark code for Optimized Partitioning, Coalesce & Parquet Export.

✓ Derive year and month partition keys
✓ Coalesce partition count to 16 files
✓ Export partitioned Snappy Parquet to cloud storage

🚀 Start Mission

← Prev (Customer 360 Analytics & Rolling Window Metrics)Next Lesson (Scalable Bronze-to-Gold Warehouse Transformation) →