Executing SQL Queries with spark.sql()

How do we execute an ANSI SQL SELECT query against a registered view and receive a DataFrame back?

Advanced⏱ 2 Minutes⭐ 630 XP

🤔 THE QUESTION

How do we execute an ANSI SQL SELECT query against a registered view and receive a DataFrame back?

💡 WHAT IS IT?

spark.sql(sqlQuery) executes an ANSI SQL string against catalog tables and returns the result as a DataFrame.

🎯 WHAT IS IT USED FOR?

Running complex SQL scripts, integrating legacy SQL codebases, and ad-hoc data analysis.

💻 EXAMPLE

result_df = spark.sql("SELECT * FROM v_customers WHERE active = true")

Practice typing production-grade PySpark code for Executing SQL Queries with spark.sql().