Selective Memory Caching with persist()

How can we cache a repeatedly accessed intermediate DataFrame across Memory and Disk with serialization?

Expert⏱ 3 Minutes⭐ 850 XP

🤔 THE QUESTION

How can we cache a repeatedly accessed intermediate DataFrame across Memory and Disk with serialization?

💡 WHAT IS IT?

persist(StorageLevel.MEMORY_AND_DISK_SER) saves computed partitions to RAM with serialized disk overflow.

🎯 WHAT IS IT USED FOR?

Iterative machine learning algorithms and pipelines feeding multiple downstream gold aggregation tables.

💻 EXAMPLE

from pyspark import StorageLevel
df_cached = df.filter(col("is_active") == True).persist(StorageLevel.MEMORY_AND_DISK_SER)

Practice typing production-grade PySpark code for Selective Memory Caching with persist().