PySpark Fundamentals Assessment: Architecture, DataFrames & RDDs

Apache Spark is the standard for distributed big data processing. Writing scalable PySpark pipelines requires understanding the relationship between the driver and executor processes, the lazy evaluation of DAGs, and the architectural differences between low-level RDDs and structured DataFrames. This assessment validates core foundational knowledge.

1. Assessment Overview & Prerequisites

This beginner assessment evaluates foundational proficiency in PySpark. It covers the distributed execution engine, SparkSession creation, DataFrame reading mechanics, selection and filtering syntax, and the distinction between lazy transformations and eager actions.

Recommended Prerequisites:

Basic Python programming knowledge.
Familiarity with SQL SELECT, WHERE, and DISTINCT concepts.
Review of our foundational guide on Introduction to PySpark.

2. What You Will Practice & Validate

Distributed Abstractions: Understanding RDDs, DataFrames, and Datasets in the Spark ecosystem.
Session Initialization: Constructing and configuring the SparkSession entrypoint.
Transformations vs Actions: Identifying which methods build execution graphs and which trigger computation.
DataFrame Operations: Applying select, filter, dropDuplicates, and show syntax.

3. How to Use This Assessment & Practice Guide

This assessment features 10 foundational questions:

Immediate Feedback: Select your answer to view instant visual confirmation and distractor analysis.
Review Explanations: Understand the Catalyst optimizer's role in making DataFrames significantly faster than raw RDDs.
Passing Standard: Aim for 80% (8/10) before advancing to intermediate joins and aggregations.

4. Target Technical Concept Matrix

Architectural Area	Operational Purpose	Production Significance
DataFrame Abstraction	Distributed table with named columns and schema	Optimized by Catalyst query planner and Tungsten in-memory execution.
SparkSession	Unified entrypoint for Spark applications since Spark 2.0	Consolidates SparkContext, SQLContext, and HiveContext into a single builder.
Lazy Evaluation	Builds lineage graphs without immediate computation	Enables global query optimization, filter pushdown, and projection pruning.
Actions vs Transformations	Actions trigger job execution; transformations return new DataFrames	Knowing the boundary prevents accidental cluster stalls and driver OOMs.

5. Interactive Knowledge Assessment

1. What is the main abstraction in PySpark for distributed data processing?

2. How do you create a SparkSession in PySpark?

3. Which PySpark transformation is lazy and returns a new RDD/DataFrame?

4. Which PySpark action triggers execution and returns results to the driver?

5. How do you read a CSV file into a PySpark DataFrame?

6. What is the difference between RDD and DataFrame?

7. How do you select a column named 'age' from a DataFrame df?

8. Which PySpark function removes duplicate rows?

9. How do you filter rows where age > 30?

10. How do you show the top 5 rows of a DataFrame?

6. Deep Dive & Architectural Analysis

The Spark Catalyst Optimizer & Tungsten Engine

The fundamental reason DataFrames outperform raw RDDs is the Catalyst Optimizer:

Analysis: Resolves table and column names against the Catalog schema.
Logical Optimization: Applies deterministic rule-based transformations:
- Predicate Pushdown: Pushes filter() conditions directly to the Parquet/Delta storage reader to avoid loading unwanted rows into memory.
- Projection Pruning: Reads only the columns requested in select(), ignoring all unused columns on disk.
Physical Planning: Generates multiple physical plans and uses cost-based models (CBO) to choose between Broadcast Hash Join and Sort-Merge Join.
Tungsten Code Generation: Generates compact JVM bytecode on the fly and stores data in raw off-heap binary memory without Java object serialization overhead.

Lazy Evaluation & The Lineage Graph

In PySpark, transformations do not execute immediately; they construct an internal dependency graph called the Lineage Graph (or DAG of RDDs). When an action (e.g., count(), write.save()) is invoked:

Spark submits the DAG to the DAGScheduler.
The DAG is split into Stages based on shuffle boundaries (narrow vs wide dependencies).
Stages are divided into individual Tasks sent to executor slots across the cluster. If any worker node crashes, Spark recomputes only the missing partitions from the lineage graph, providing automatic fault tolerance without manual checkpoints.

7. Real-World Production Scenarios & Pitfalls

⚠️ Calling collect() on Large Production Datasets

Failure Mode: Executing all_rows = df.collect() on a 50GB distributed DataFrame to iterate over records in Python.
Impact: Executors send 50GB of data over the network to the driver VM, exhausting driver JVM heap memory and crashing the entire Spark application with java.lang.OutOfMemoryError.
Production Remedy: Process data transformationally in Spark or write output directly to storage (Parquet/Delta). If inspection is needed, use df.take(10) or df.show(10).

⚠️ Reading CSVs with Default String Typing in Production

Failure Mode: Loading CSV without defining an explicit schema or inferSchema=true.
Impact: All numeric and timestamp columns default to StringType. Subsequent numerical calculations or aggregations fail or yield incorrect alphabetical sorting.
Production Remedy: Define an explicit StructType schema with correct IntegerType, DoubleType, and TimestampType definitions.

8. Canonical Tutorial Reference

Review the canonical tutorials covering these exact architectures before or after taking this assessment:

PySpark Introduction & SparkSession — Understanding Spark architecture, driver/worker topology, and session creation.
RDD vs DataFrame Architectural Deep Dive — Detailed comparison of performance, Catalyst optimization, and memory models.
DataFrame Creation & File Formats — Reading and writing CSV, Parquet, JSON, and ORC with explicit schemas.

9. Next Steps in Your Learning Progression

Advance to the PySpark Intermediate Quiz covering groupBy aggregations, conditional column creation with when/otherwise, custom UDFs, and join types.

10. Key Engineering Takeaways

Prefer DataFrames over RDDs: DataFrames utilize Catalyst and Tungsten for near-native C-speed execution regardless of Python.
Understand Lazy Evaluation: Transformations build execution plans; only actions trigger distributed computation.
Never collect() Production Big Data: Keep computation distributed; use show(), take(), or write output to Delta tables.

1. Assessment Overview & Prerequisites​

2. What You Will Practice & Validate​

3. How to Use This Assessment & Practice Guide​

4. Target Technical Concept Matrix​

5. Interactive Knowledge Assessment​

6. Deep Dive & Architectural Analysis​

The Spark Catalyst Optimizer & Tungsten Engine​

Lazy Evaluation & The Lineage Graph​

7. Real-World Production Scenarios & Pitfalls​

⚠️ Calling collect() on Large Production Datasets​

⚠️ Reading CSVs with Default String Typing in Production​

8. Canonical Tutorial Reference​

9. Next Steps in Your Learning Progression​

10. Key Engineering Takeaways​