Automated Data Quality Null Audit

How do we dynamically compute the count of null values across every column in a DataFrame?

Intermediate⏱ 3 Minutes⭐ 400 XP

🤔 THE QUESTION

How do we dynamically compute the count of null values across every column in a DataFrame?

💡 WHAT IS IT?

Using a Python list comprehension to count null values for each column dynamically in a single select statement.

🎯 WHAT IS IT USED FOR?

Automated pipeline data profiling, ingress data validation gates, and data quality SLA alerts.

💻 EXAMPLE

from pyspark.sql.functions import col, count, when

df.select([
    count(when(col(c).isNull(), c)).alias(c)
    for c in df.columns
]).show()

Practice typing production-grade PySpark code for Automated Data Quality Null Audit.