Quarantine Routing with Error Reasons

How can we assemble an array of specific error reason codes for failed records and route them to quarantine?

Advanced⏱ 3 Minutes⭐ 860 XP

🤔 THE QUESTION

How can we assemble an array of specific error reason codes for failed records and route them to quarantine?

💡 WHAT IS IT?

array_compact() and when() combine triggered error codes into a list of specific failure reasons.

🎯 WHAT IS IT USED FOR?

Dead-letter queue (DLQ) tables allowing upstream data providers to remediate flawed payloads.

💻 EXAMPLE

df_errors = df.withColumn("errors", array_compact(array(when(col("id").isNull(), "ERR_NULL_ID"), when(col("amount") <= 0, "ERR_INVALID_AMOUNT"))))
df_quarantine = df_errors.filter(size(col("errors")) > 0)

🎯 Mission Objectives

Practice typing production-grade PySpark code for Quarantine Routing with Error Reasons.

✓ array_compact() function
✓ Error reason aggregation
✓ Dead-letter queue architecture

🚀 Start Mission

← Prev (Data Quality Flag Matrix)Next Lesson (Splitting Clean and Invalid Datasets) →