Skip to main content
DatabricksPySparkSnowflakeApache Airflow
Code Keys/PySpark Academy/Module 27/Lesson 217
PYSPARK • LESSON 217

Quarantine Routing with Error Reasons

How can we assemble an array of specific error reason codes for failed records and route them to quarantine?

Advanced⏱ 3 Minutes⭐ 860 XP
🤔 THE QUESTION

How can we assemble an array of specific error reason codes for failed records and route them to quarantine?

💡 WHAT IS IT?

array_compact() and when() combine triggered error codes into a list of specific failure reasons.

🎯 WHAT IS IT USED FOR?

Dead-letter queue (DLQ) tables allowing upstream data providers to remediate flawed payloads.

💻 EXAMPLE
df_errors = df.withColumn("errors", array_compact(array(when(col("id").isNull(), "ERR_NULL_ID"), when(col("amount") <= 0, "ERR_INVALID_AMOUNT"))))
df_quarantine = df_errors.filter(size(col("errors")) > 0)

🎯 Mission Objectives

Practice typing production-grade PySpark code for Quarantine Routing with Error Reasons.

  • ✓ array_compact() function
  • ✓ Error reason aggregation
  • ✓ Dead-letter queue architecture
🚀 Start Mission
← Prev (Data Quality Flag Matrix)Next Lesson (Splitting Clean and Invalid Datasets) →