Skip to main content
DatabricksPySparkSnowflakeApache Airflow
Code Keys/PySpark Academy/Module 31/Lesson 259
PYSPARK • LESSON 259

Creating Hive/Spark Managed and External Tables via DDL

How can we create persistent partitioned Parquet tables in the metastore catalog using CREATE TABLE DDL in Spark SQL?

Advanced⏱ 3 Minutes⭐ 880 XP
🤔 THE QUESTION

How can we create persistent partitioned Parquet tables in the metastore catalog using CREATE TABLE DDL in Spark SQL?

💡 WHAT IS IT?

spark.sql('CREATE TABLE ... USING PARQUET PARTITIONED BY (...)') registers table schemas and partition schemes in the metastore.

🎯 WHAT IS IT USED FOR?

Setting up data lakehouse warehouse schemas for enterprise BI and reporting consumption.

💻 EXAMPLE
spark.sql("""CREATE TABLE IF NOT EXISTS default.analytics_sales (order_id STRING, customer_id STRING, amount DOUBLE, sale_date DATE) USING PARQUET PARTITIONED BY (sale_date)""")

🎯 Mission Objectives

Practice typing production-grade PySpark code for Creating Hive/Spark Managed and External Tables via DDL.

  • ✓ CREATE TABLE DDL in Spark SQL
  • ✓ USING PARQUET storage format
  • ✓ PARTITIONED BY registration
🚀 Start Mission
← Prev (Array and Struct Extraction with Lateral Views in SQL)Next Lesson (Production Spark SQL Analytical Pipeline) →