PySpark Certification Practice Test 10
Debug The ETL Pipeline
PySpark Certification Practice Test 10
Debug The ETL Pipeline
Insightful Saga — Modern Data Engineering Certification Preparation
Question 181
Problem: ```python df.filter(col("salary") > 10000) ``` Expected Output: 100 rows Actual Output: 1000 rows Most likely issue?
### Fix ```python df = df.filter(col("salary") > 10000) ```
Question 182
Problem: ```python df.join(df2) ``` Output row count is extremely large. Most likely fix?
Question 183
Problem: ```python df.collect() ``` Job fails with Driver OOM. Most likely fix?
Question 184
Problem: ```python df.write.mode("append") ``` Duplicates appear every day. Best fix?
Question 185
Problem: A join takes 2 hours. Tables: ```text Transactions = 5 TB Reference = 20 MB ``` Fix?
Question 186
Problem: ```python df.repartition(1) ``` Output size: ```text 600 GB ``` Best fix?
Question 187
Problem: Pipeline runtime increased suddenly. Logs show: ```python Python UDF ``` processing billions of rows. Best recommendation?
Question 188
Problem: Pipeline loads customer data. Business finds outdated records. Most likely fix?
Question 189
Problem: ```python dropDuplicates() ``` still leaves duplicates. Most likely issue?
Question 190
Problem: ```python orderBy() ``` takes very long. Dataset: ```text 3 Billion Rows ``` Most likely explanation?
Question 191
Problem: Job succeeds. Gold table row count: ```text 0 ``` Most likely next step?
Question 192
Problem: SCD Type 2 table has multiple active rows. Expected: ```text One active record ``` Most likely fix?
Question 193
Problem: Incremental load misses records. Watermark: ```python last_load_date ``` Most likely area to review?
Question 194
Problem: ```python df.count() ``` used repeatedly for debugging. Pipeline becomes slow. Best recommendation?
Question 195
Problem: Customer table contains: ```text 1 customer 5 duplicate orders ``` Join multiplies rows. Best fix?
Question 196
Problem: Source schema changed. Pipeline starts failing. Best engineering solution?
Question 197
Problem: Driver CPU constantly at 100%. Logs show: ```python collect() foreach() local loops ``` Most likely root cause?
Question 198
Problem: A skewed join causes one task to run much longer than others. Most likely technique?
Question 199
Problem: Business reports inconsistent revenue numbers. Two ETL pipelines calculate revenue differently. Best correction?
Question 200
Problem: A PySpark ETL solution works perfectly in QA but fails at production scale. What is the MOST likely missing consideration?
### Explanation Many ETL solutions work on millions of rows but fail on billions because scalability, skew, shuffles, memory usage, and execution plans were not evaluated properly. --- --- title: PySpark Certification Practice Test 11 description: Spark Optimization Scenarios --- # PySpark Certification Practice Test 11