How can we split an ingestion stream into clean valid records for bronze tables and invalid records for triage?
A master is_valid boolean condition enables filtering the DataFrame into two distinct destination streams.
Ensuring bronze-to-silver ETL pipelines never halt while capturing flawed records for analysis.
is_valid_expr = col("id").isNotNull() & (col("amount") > 0) & col("email").isNotNull()
df_clean = df.filter(is_valid_expr)
df_invalid = df.filter(~is_valid_expr)Practice typing production-grade PySpark code for Splitting Clean and Invalid Datasets.