Development Arena
Build production data engineering pipelines from the ground up across modern lakehouse and warehouse architectures. Architect Bronze, Silver, and Gold layers, configure Delta tables, integrate external APIs, design dimensional data marts, implement CDC pipelines, and tackle interactive Hands-On Engineering Labs.
Build Customer Lakehouse Pipeline
Build a production-ready customer data pipeline from source ingestion through Bronze, Silver, and Gold layers for analytics consumption.
Incremental Sales Pipeline
Build a reliable incremental sales pipeline that processes cloud storage transaction files, handles duplicates and late data, and delivers trusted analytics datasets.
REST API Integration Pipeline
Build an ingestion pipeline consuming third-party REST API customer activity with pagination, transient error handling, rate limiting, and Silver/Gold curation.
Enterprise Sales Reporting Data Mart
Design and implement a dimensional reporting data mart with explicit grain definition, fact/dimension relationships, SCD handling, and Snowflake reconciliation.
CDC-Based Customer Data Pipeline
Design a Change Data Capture (CDC) processing pipeline handling INSERT, UPDATE, and DELETE event streams with out-of-order sequencing, replay, and current-state materialization.
Real-Time Sales Streaming Pipeline
Build a low-latency streaming pipeline continuously ingesting real-time sales events, handling watermarking, late data, stateful micro-batch aggregations, and Snowflake synchronization.
Customer ML Feature Engineering Pipeline
Design a production-grade ML feature engineering pipeline transforming customer and transaction history into reusable feature stores with strict point-in-time leakage prevention.
Production CI/CD Data Platform
Implement automated GitHub Actions CI/CD workflows for multi-environment data pipelines with linting, unit/integration testing, secret management, and rollback mechanisms.
Multi-Source Enterprise Data Platform
Build a unified enterprise platform integrating Customer DB, Sales Files, Product REST API, and Store Reference Data with entity resolution, SLA freshness, and cross-source reconciliation.
Enterprise Data Engineering Capstone
Architect and build an end-to-end enterprise lakehouse platform combining batch ingestion, real-time streaming, CDC, REST APIs, ML feature stores, CI/CD, and Snowflake data marts.
Build a Production-Ready Sales Pipeline
Hands-On Lab: Ingest raw retail sales CSV files, clean corrupt/duplicate data, construct Bronze, Silver, and Gold Delta layers, and calculate executive business metrics.
Production Pipeline Debugging Challenge
Hands-On Lab: Receive a broken production PySpark pipeline producing corrupted data, diagnose Cartesian joins and null handling bugs, fix the implementation, and validate reconciliation.
Optimize a Slow Spark Pipeline
Hands-On Lab: Profile a deliberately inefficient 48-minute Spark workload, eliminate shuffle bottlenecks, handle key skew, tune partitions, and benchmark measurable runtime improvement.
Production Data Pipeline Incident Recovery
Hands-On Lab: Respond to an urgent 06:30 AM production pipeline failure, inspect error logs, quarantine corrupt files, execute Delta time-travel recovery, and author a Post-Incident Review.
Build an End-to-End Data Engineering Platform
Hands-On Master Challenge: Given a 5-source enterprise starter dataset, build the complete platform code: ingestion, quarantine, Silver conformed layer, Gold star schema, testing, and telemetry.