Automated Data Quality Rule Assertions

How do we execute programmatic data quality assertions to verify zero null IDs and zero negative amounts before publishing?

Production⏱ 3 Minutes⭐ 1420 XP

🤔 THE QUESTION

How do we execute programmatic data quality assertions to verify zero null IDs and zero negative amounts before publishing?

💡 WHAT IS IT?

Evaluating aggregate validation metrics and executing Python assertions to block invalid data releases.

🎯 WHAT IS IT USED FOR?

Automated data testing in CI/CD pipelines, Great Expectations integration, and data contract enforcement.

💻 EXAMPLE

from pyspark.sql.functions import col, count, when

dq_metrics = df.select(
    count(when(col("customer_id").isNull(), 1)).alias("null_cust_ids"),
    count(when(col("total_amount") < 0, 1)).alias("negative_amounts")
).collect()[0]

assert dq_metrics["null_cust_ids"] == 0, "Data Quality Assertion Failed: Null customer IDs"
assert dq_metrics["negative_amounts"] == 0, "Data Quality Assertion Failed: Negative amounts"

🎯 Mission Objectives

Practice typing production-grade PySpark code for Automated Data Quality Rule Assertions.

✓ Calculate data quality violation metrics
✓ Collect metrics row to driver
✓ Execute programmatic assertions to fail bad runs

🚀 Start Mission

← Prev (Handling Late-Arriving & Out-of-Order Events)Next Lesson (Partition-Pruned Incremental Daily Ingestion) →