Capstone Part 2: Automated Null Imputation and Cleaning

How can we sanitize text strings, standardize dates, and impute missing monetary values with median defaults?

Production⏱ 3 Minutes⭐ 810 XP

🤔 THE QUESTION

How can we sanitize text strings, standardize dates, and impute missing monetary values with median defaults?

💡 WHAT IS IT?

Combining trim(), to_timestamp(), and fillna() cleans all fields in preparation for business transformations.

🎯 WHAT IS IT USED FOR?

Data preparation and cleansing in enterprise Silver ETL layers.

💻 EXAMPLE

df_clean = df_parsed.withColumn("event_id", trim(col("event_id"))).withColumn("event_timestamp", to_timestamp(col("timestamp"))).fillna({"amount": 0.0, "user_id": "UNKNOWN"})

🎯 Mission Objectives

Practice typing production-grade PySpark code for Capstone Part 2: Automated Null Imputation and Cleaning.

✓ String and date cleansing
✓ fillna() value imputation
✓ Silver layer preparation

🚀 Start Mission

← Prev (Capstone Part 1: Schema Enforcement on Dirty JSON Payloads)Next Lesson (Capstone Part 3: CDC Natural Key Deduplication) →