Higher-Order aggregate() on Arrays

How can we calculate the sum or product of all elements inside an array column without explode()?

Advanced⏱ 3 Minutes⭐ 880 XP

🤔 THE QUESTION

How can we calculate the sum or product of all elements inside an array column without explode()?

💡 WHAT IS IT?

The aggregate() higher-order function reduces an array to a scalar value using an accumulator and lambda functions.

🎯 WHAT IS IT USED FOR?

Calculating row-level totals from embedded line items in document-oriented pipelines.

💻 EXAMPLE

df_total = df.withColumn("items_total", aggregate(col("item_prices"), lit(0.0), lambda acc, x: acc + x))

Practice typing production-grade PySpark code for Higher-Order aggregate() on Arrays.