Creating Hive/Spark Managed and External Tables via DDL

How can we create persistent partitioned Parquet tables in the metastore catalog using CREATE TABLE DDL in Spark SQL?

Advanced⏱ 3 Minutes⭐ 880 XP

🤔 THE QUESTION

How can we create persistent partitioned Parquet tables in the metastore catalog using CREATE TABLE DDL in Spark SQL?

💡 WHAT IS IT?

spark.sql('CREATE TABLE ... USING PARQUET PARTITIONED BY (...)') registers table schemas and partition schemes in the metastore.

🎯 WHAT IS IT USED FOR?

Setting up data lakehouse warehouse schemas for enterprise BI and reporting consumption.

💻 EXAMPLE

spark.sql("""CREATE TABLE IF NOT EXISTS default.analytics_sales (order_id STRING, customer_id STRING, amount DOUBLE, sale_date DATE) USING PARQUET PARTITIONED BY (sale_date)""")

🎯 Mission Objectives

Practice typing production-grade PySpark code for Creating Hive/Spark Managed and External Tables via DDL.

✓ CREATE TABLE DDL in Spark SQL
✓ USING PARQUET storage format
✓ PARTITIONED BY registration

🚀 Start Mission

← Prev (Array and Struct Extraction with Lateral Views in SQL)Next Lesson (Production Spark SQL Analytical Pipeline) →