Custom Multi-Metric Aggregations with agg()

How do we compute headcount, total spend, average comp, and maximum salary per department in one pass?

Intermediate⏱ 2 Minutes⭐ 440 XP

🤔 THE QUESTION

How do we compute headcount, total spend, average comp, and maximum salary per department in one pass?

💡 WHAT IS IT?

Passing multiple aggregate functions into agg() to calculate comprehensive group statistics in a single shuffle.

🎯 WHAT IS IT USED FOR?

Departmental budget audits, organizational compensation analysis, and executive HR reports.

💻 EXAMPLE

from pyspark.sql.functions import avg, col, count, max, sum

df.groupBy("department").agg(
    count(col("emp_id")).alias("headcount"),
    sum(col("salary")).alias("dept_spend"),
    avg(col("salary")).alias("avg_comp"),
    max(col("salary")).alias("top_salary")
).show()

🎯 Mission Objectives

Practice typing production-grade PySpark code for Custom Multi-Metric Aggregations with agg().

✓ Compute count, sum, avg, and max in single agg()
✓ Alias all departmental metrics
✓ Produce executive HR summary

🚀 Start Mission

← Prev (Multi-Dimensional Group Aggregation)Next Lesson (Conditional Aggregations per Group) →