Array Filtering with filter_array()

How can we filter out negative numbers or invalid values from an array column directly?

Advanced⏱ 3 Minutes⭐ 860 XP

🤔 THE QUESTION

How can we filter out negative numbers or invalid values from an array column directly?

💡 WHAT IS IT?

The array_filter() function filters elements in an array column based on a boolean lambda predicate.

🎯 WHAT IS IT USED FOR?

Sanitizing embedded sensor logs, removing zero or negative price entries from order arrays.

💻 EXAMPLE

df_valid_scores = df.withColumn("positive_scores", array_filter(col("scores"), lambda x: x > 0))

Practice typing production-grade PySpark code for Array Filtering with filter_array().