In-Memory Ingestion Caching with cache()

How and when do we cache a DataFrame in executor memory when it is accessed multiple times across queries?

Expert⏱ 2 Minutes⭐ 900 XP

🤔 THE QUESTION

How and when do we cache a DataFrame in executor memory when it is accessed multiple times across queries?

💡 WHAT IS IT?

cache() sets StorageLevel.MEMORY_AND_DISK, storing deserialized partitions in memory after the first action.

🎯 WHAT IS IT USED FOR?

Iterative machine learning, multi-query dashboard backends, and avoiding re-reading large datasets from disk.

💻 EXAMPLE

df = spark.read.parquet("data/large_dataset.parquet").cache()
df.count()

Practice typing production-grade PySpark code for In-Memory Ingestion Caching with cache().