Automatic Schema Inference with inferSchema

How can we instruct PySpark to inspect CSV values and automatically infer data types during exploration?

Beginner⏱ 2 Minutes⭐ 170 XP

🤔 THE QUESTION

How can we instruct PySpark to inspect CSV values and automatically infer data types during exploration?

💡 WHAT IS IT?

inferSchema triggers an extra pass over the data to automatically cast integers, floats, and booleans from strings.

🎯 WHAT IS IT USED FOR?

Rapid exploratory data analysis and prototyping in development environments.

💻 EXAMPLE

df = spark.read \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .csv("data/logs.csv")

Practice typing production-grade PySpark code for Automatic Schema Inference with inferSchema.