Customer reporting dashboard is delayed and SLA has been missed.
Data Engineering Lead
Spark job runtime increased dramatically during join processing.
Platform Team
Cluster health looks normal but one executor is heavily overloaded.
Analytics Team
Downstream dashboards are waiting for today's aggregation load.
Data Skew Runtime Timeline
A highly skewed join key caused one executor to process most of the shuffle data, dramatically increasing Spark job runtime.
Data Skew Failure Risk
Data Load Started: 08:00 AM (SUCCESS) Shuffle Stage: 08:25 AM (SLOW RUNNING) Executor Analysis: 08:40 AM (SKEW DETECTED) Impact: Aggregation job exceeded SLA
🔴 HIGH PRIORITY INCIDENT
Data Skew Incident
Production Spark aggregation workflow experienced severe runtime degradation due to data skew, causing one executor to process a disproportionate amount of data and resulting in SLA breach.
Pipeline
daily_customer_aggregation
Severity
HIGH
Production ETL workflow suffered from data skew where heavily concentrated business keys created uneven partition distribution, leading to executor delays and SLA breach risk.