Skip to main content
DatabricksPySparkSnowflakeApache Airflow
Code Keys/PySpark Academy/Module 22/Lesson 170
PYSPARK • LESSON 170

Storage Level Tuning with persist()

How do we persist a dataset with serialized in-memory and disk spillover (MEMORY_AND_DISK_SER)?

Expert⏱ 2 Minutes⭐ 920 XP
🤔 THE QUESTION

How do we persist a dataset with serialized in-memory and disk spillover (MEMORY_AND_DISK_SER)?

💡 WHAT IS IT?

persist(StorageLevel) enables fine-grained control over serialization, disk spill, and replication factors.

🎯 WHAT IS IT USED FOR?

Conserving executor RAM by storing compact serialized bytes while preventing job crashes via disk spill.

💻 EXAMPLE
from pyspark import StorageLevel

df.persist(StorageLevel.MEMORY_AND_DISK_SER)

🎯 Mission Objectives

Practice typing production-grade PySpark code for Storage Level Tuning with persist().

  • ✓ Import StorageLevel class
  • ✓ Persist with serialized in-memory storage
  • ✓ Prevent executor memory pressure
🚀 Start Mission
← Prev (In-Memory Ingestion Caching with cache())Next Lesson (Freeing Cluster Memory with unpersist()) →