Reading Cloud Storage with Recursive File Lookup

How can we read files from deeply nested non-standard directory trees recursively using wildcards?

Expert⏱ 3 Minutes⭐ 880 XP

🤔 THE QUESTION

How can we read files from deeply nested non-standard directory trees recursively using wildcards?

💡 WHAT IS IT?

Using recursiveFileLookup option traverses arbitrary nested directory hierarchies without requiring Hive partition syntax.

🎯 WHAT IS IT USED FOR?

Ingesting legacy FTP dumps, timestamped bucket folders, and third-party vendor landing zones.

💻 EXAMPLE

df_recursive = spark.read.option("recursiveFileLookup", "true").parquet("s3://lakehouse/landing_zone/legacy_data/")

Practice typing production-grade PySpark code for Reading Cloud Storage with Recursive File Lookup.