Skip to main content
DatabricksPySparkSnowflakeApache Airflow
Code Keys/PySpark Academy/Module 34/Lesson 286
PYSPARK • LESSON 286

Scenario: IoT Sensor Telemetry Anomaly Detection

How can we detect factory machine sensor spikes where vibration readings exceed 3 standard deviations from the rolling mean?

Production⏱ 3 Minutes⭐ 850 XP
🤔 THE QUESTION

How can we detect factory machine sensor spikes where vibration readings exceed 3 standard deviations from the rolling mean?

💡 WHAT IS IT?

Calculating rolling average and standard deviation with Window functions flags z-score statistical anomalies.

🎯 WHAT IS IT USED FOR?

Predictive maintenance pipelines alerting engineers before industrial equipment experiences catastrophic failure.

💻 EXAMPLE
w = Window.partitionBy("sensor_id").orderBy("timestamp").rowsBetween(-29, Window.currentRow)
df_anomalies = df_sensor.withColumn("rolling_avg", avg("vibration").over(w)).withColumn("rolling_std", stddev("vibration").over(w)).withColumn("z_score", abs((col("vibration") - col("rolling_avg")) / col("rolling_std"))).filter(col("z_score") > 3.0)

🎯 Mission Objectives

Practice typing production-grade PySpark code for Scenario: IoT Sensor Telemetry Anomaly Detection.

  • ✓ Rolling z-score calculation
  • ✓ stddev() over window
  • ✓ Industrial IoT anomaly detection
🚀 Start Mission
← Prev (Scenario: Slowly Changing Dimension (SCD Type 2) Ingestion)Next Lesson (Scenario: User Retention Cohort Analysis) →