Array Transformations with transform()

How can we transform every element inside an array column without exploding the DataFrame rows?

Advanced⏱ 3 Minutes⭐ 850 XP

🤔 THE QUESTION

How can we transform every element inside an array column without exploding the DataFrame rows?

💡 WHAT IS IT?

The higher-order transform() function applies a lambda expression to each item in an array natively in Catalyst.

🎯 WHAT IS IT USED FOR?

Normalizing, discounting, or formatting nested lists of items without multiplying row volume.

💻 EXAMPLE

df_discounted = df.withColumn("discounted_prices", transform(col("prices"), lambda x: x * 0.9))

Practice typing production-grade PySpark code for Array Transformations with transform().