Skip to main content
DatabricksPySparkSnowflakeApache Airflow

Creating DataFrames from CSV, JSON, Parquet & Hive Tables

Every analytics pipeline at NeoMart, our growing e-commerce platform, starts with one step: loading data into Spark.
Whether it comes from mobile apps, warehouses, partners, or machine logs, your first job as a data engineer is to convert this raw data into a DataFrame — Spark’s most widely used data structure.

DataFrames provide schema, structure, column-level operations, and optimization through Catalyst.
But how you create a DataFrame depends on the file format you’re working with.

Let’s explore the four most common formats: CSV, JSON, Parquet, and Hive tables.

Why File Formats Matter​

Not all file formats behave the same.
Some are slow but simple (CSV), others lightning fast (Parquet), and some ideal for semi-structured workloads (JSON).

Choosing the right format can easily save minutes or even hours in large-scale ETL jobs.

1. Creating DataFrames from CSV Files​

CSV files are widely used but come with limitations — no schema, no compression, and slow parsing.

df = spark.read \
    .option("header", True) \
    .option("inferSchema", True) \
    .csv("/mnt/data/sales.csv")

✔ When to Use CSV​

  • During initial ingestion
  • When partners/vendors deliver small datasets
  • For debugging and quick data inspection

❌ Avoid for big data​

CSV parsing becomes slow as data volume increases.

2. Creating DataFrames from JSON Files​

JSON is perfect for logs, nested attributes, and NoSQL-like structures.

df = spark.read \
    .option("multiline", True) \
    .json("/mnt/data/events.json")

✔ Best for​

  • Clickstream logs
  • IoT events
  • User activity streams

Story Example​

NeoMart’s mobile app sends events like:

{
  "user": "123",
  "actions": ["view", "add_to_cart"]
}

JSON allows nested data, which Spark can parse easily.

3. Creating DataFrames from Parquet Files (Best Practice)​

Parquet is the default format for big data because of:

  • Columnar storage
  • Built-in compression
  • Predicate pushdown
  • Fast read/write
df = spark.read.parquet("/mnt/data/transactions/")

✔ Best Format For​

  • Analytics
  • Large-scale ETL
  • Machine learning pipelines
  • Databricks Delta workflows

This is NeoMart’s recommended storage format for raw, clean, and analytics layers.

4. Creating DataFrames from Hive Tables​

Hive tables allow you to store structured datasets with metadata (schema, partitions).

df = spark.table("analytics.daily_orders")

or using SQL:

df = spark.sql("SELECT * FROM analytics.daily_orders")

✔ Helpful When​

  • Working with enterprise data warehouses
  • Using Databricks metastore
  • Structuring data by partitions (date, region, etc.)

5. Summary​

  • CSV → simple, human-readable, but slow
  • JSON → perfect for nested & semi-structured data
  • Parquet → fastest & most efficient (recommended for big data)
  • Hive Tables → ideal for enterprise-scale structured storage

Proper DataFrame creation lays the foundation for the entire transformation pipeline — ensuring performance, accuracy, and scalability.

Next up, we’ll master the DataFrame API — Select, Filter, WithColumn, Drop, the core tools used to transform raw data into analytics-ready datasets.