Workload Profiles:

Physical Cluster Specifications

Calculated Allocations (The 5-Core Best Practice Rule)
29
Total Executors
5
Cores / Executor
18 GB
Executor Heap

JVM Heap & Memory Fraction Breakdown

Execution Memory (Joins/Sorts)
5.31 GB
Active shuffle shuffle buffers
Storage Memory (Cache/Broadcast)
5.31 GB
DataFrame .cache() / persist()
User Memory
7.08 GB
Custom objects & UDF structures
Memory Overhead
1843 MB
Off-heap, PySpark Python daemons

Generated spark-submit Production Configuration

spark-submit \
  --class com.insightfulsaga.pipeline.MainApp \
  --master yarn \
  --deploy-mode cluster \
  --num-executors 29 \
  --executor-cores 5 \
  --executor-memory 18g \
  --driver-cores 5 \
  --driver-memory 18g \
  --conf spark.executor.memoryOverhead=1843m \
  \
  --conf spark.memory.fraction=0.6 \
  --conf spark.memory.storageFraction=0.5 \
  --conf spark.sql.shuffle.partitions=435 \
  hdfs:///apps/pipeline.jar