Skip to main content
PYSPARK • LESSON 138

Navigating Nested Struct Fields with Dot Notation

How do we project individual sub-fields (like address.city) from a nested struct column using dot notation?

Advanced2 Minutes690 XP
🤔 THE QUESTION

How do we project individual sub-fields (like address.city) from a nested struct column using dot notation?

💡 WHAT IS IT?

Referencing sub-fields with col('struct_name.field_name') to query nested struct hierarchies directly.

🎯 WHAT IS IT USED FOR?

Flattening nested records, accessing sub-properties, and extracting specific fields from JSON/Avro payloads.

💻 EXAMPLE
from pyspark.sql.functions import col

df.select(col("user_id"), col("address.city"), col("address.zipcode")).show()

🎯 Mission Objectives

Practice typing production-grade PySpark code for Navigating Nested Struct Fields with Dot Notation.

  • Use dot notation to reference nested fields
  • Select specific sub-properties
  • Display unpacked address attributes