Create DataFrame from In-Memory Data

How can we create a distributed DataFrame from in-memory Python collections for testing?

Beginner⏱ 2 Minutes⭐ 140 XP

🤔 THE QUESTION

How can we create a distributed DataFrame from in-memory Python collections for testing?

💡 WHAT IS IT?

createDataFrame converts local Python lists or tuples into a distributed, partitioned Spark DataFrame.

🎯 WHAT IS IT USED FOR?

Mocking test datasets in automated CI/CD pipelines and testing transformation logic without external file dependencies.

💻 EXAMPLE

data = [("Alice", 34), ("Bob", 45)]
df = spark.createDataFrame(data, ["name", "age"])

Practice typing production-grade PySpark code for Create DataFrame from In-Memory Data.