Difficulty

Expert / Capstone

Reward

1,000 XP · 🏆 Enterprise Data Engineer

Prerequisites

Batch · Streaming · CDC · ML Features · CI/CD

Primary Stack

Databricks · PySpark · Snowflake

Target

Unified Enterprise Platform

Capstone Challenge

DEV-010

Enterprise Data Engineering Capstone

The ultimate Data Engineering challenge. Architect, build, and deploy an enterprise-grade lakehouse and warehouse platform combining batch ETL, real-time streaming, CDC replication, REST API integration, ML feature stores, CI/CD automation, and Snowflake serving with rigorous data governance.

Project Brief

Synthesize foundational, intermediate, advanced, and expert engineering capabilities into a production platform.

Enterprise Business Scenario

A multinational omnichannel retail conglomerate operates hundreds of physical department stores, high-throughput e-commerce platforms, and a direct-to-consumer mobile application. The enterprise is undergoing a comprehensive digital transformation to unify its fragmented data landscape into a unified, modern analytics and AI platform.

The company currently receives data across five distinct ingestion channels: continuous customer profile CDC streams from transactional databases, high-volume batch sales files from store servers, real-time event streams from online checkouts, external product catalog feeds via REST APIs, and store reference hierarchies from ERP systems.

Capstone Development Objective

As the Lead Data Platform Architect, you must design, build, test, and deploy the complete end-to-end Enterprise Data Platform. The solution must ingest all five data sources across batch, streaming, CDC, and REST protocols; standardize and curate records through an immutable Medallion Lakehouse (Bronze, Silver, Gold); generate leakage-free ML feature stores; publish an enterprise dimensional data mart to Snowflake; orchestrate dependencies with automated retries; and implement automated GitHub Actions CI/CD workflows with automated rollback and comprehensive observability.

Expected Outcome

5-Way Ingestion Engine

Unify Batch files, Real-Time Streaming, Database CDC, REST APIs, and Reference data.

Medallion Lakehouse

Implement Bronze raw ledgers, Silver entity-resolved tables, and Gold analytical marts.

Real-Time Streaming

Process low-latency sales streams with event-time watermarking and sliding window KPIs.

Operational CDC Engine

Replicate INSERT, UPDATE, and DELETE database changes with deterministic state resolution.

ML Feature Store

Serve point-in-time accurate, leakage-free feature vectors for predictive machine learning.

Production CI/CD & Governance

Automate testing, multi-environment deployment, secret vaults, and disaster recovery.

Capstone Standard

This capstone challenge evaluates complete data engineering competency: architectural design, distributed compute optimization, schema evolution, streaming state management, DataOps automation, and financial reconciliation.

Source System

Inspect the five enterprise data feeds, ingestion protocols, update frequencies, and schema contracts.

Five Ingestion Feeds Overview

The capstone platform integrates five heterogeneous enterprise sources:

Feed 1 — Customer CDC

Real-time Debezium / Kafka CDC Stream

Feed 2 — Sales Batch Files

Cloud Storage CSV / Parquet (Hourly)

Feed 3 — Real-Time Streaming

POS Event Stream / Structured Streaming

Feed 4 — Product REST API

External Partner Catalog API (JSON / 6h)

Feed 5 — Store Reference

ERP Master Data Reference Tables

Target Warehouse

Snowflake Enterprise Dimensional Star Schema

Source Feeds Contract Matrix

Feed 1: customer_cdc*

CDC STREAM

event_id, customer_id, operation (INS/UPD/DEL), event_timestamp, lsn, profile_payload.

Feed 2: sales_batch*

BATCH FILES

batch_id, txn_id, store_id, customer_id, sku, timestamp, qty, price, discount, net_total.

Feed 3: streaming_events*

REAL-TIME

event_id, txn_id, terminal_id, event_timestamp, item_array, gross_amount, payment_status.

Feed 4: product_api*

REST API

sku_code, product_name, category, brand, wholesale_cost, msrp, is_active, updated_at.

Feed 5: store_reference*

ERP MASTER

store_id, store_name, region, territory, store_format, sq_footage, opening_date, status.

Known Complex Edge Cases

Cross-Protocol Identifier Alignment

CDC, batch, and streaming feeds generate varying customer and SKU formats requiring unified surrogate mapping.

Concurrent Batch & Streaming Ingestion

Hourly batch transactions and real-time streaming events write to complementary analytical facts without duplication.

Out-of-Order & Late CDC Mutations

Delayed customer address updates must not overwrite newer customer state.

External API Throttling & Rate Limits

Product API ingestion must obey 429 backoff headers without blocking core transaction processing.

Architecture Standard

The platform must achieve complete decoupling. A network failure in the product catalog API or a delayed batch file must never compromise the real-time sales stream or corrupt historical customer master state.

Expected Architecture

Unified Medallion Lakehouse & Warehouse architecture combining Batch, Streaming, CDC, ML, and CI/CD.

Capstone End-to-End Flow

The architecture implements a multi-protocol ingestion layer, an immutable Medallion Lakehouse in Databricks/Delta Lake, an automated ML Feature Store, and a dimensional serving layer in Snowflake.

Multi-Source Ingestion Layer

CDC / Batch / Streaming / REST / ERP

Ingest all 5 feeds via dedicated, fault-isolated pipelines with rate limits and retry policies.

↓

Bronze — Immutable Raw Storage

Delta Lake (Append-Only)

Store unmodified raw records with ingestion metadata, LSNs, file paths, and batch tracking.

↓

Silver — Standardized Master State

PySpark / Delta MERGE / CDF

Execute schema normalization, CDC state resolution, point-in-time watermarking, and dead-letter quarantine.

↓

Gold — Multi-Disciplinary Serving

Batch Marts / Streaming KPIs / Features

Assemble enterprise dimensional models, 5-min tumbling streaming KPIs, and point-in-time ML feature stores.

↓

Snowflake Enterprise Serving

Snowflake Tables & Semantic Views

Host physical DIM_*, FACT_*, ML_*, and executive reporting views with clustering and security governance.

↓

DataOps, CI/CD & Governance

GitHub Actions / OIDC / PagerDuty

Automate multi-environment promotion (Dev → QA → Prod), secret vaults, automated rollback, and SLA alerting.

Architecture Expectations

Full End-to-End Replayability

Truncating Silver and replaying Bronze changelogs and batch files must reconstruct 100% accurate current state.

Zero Future ML Data Leakage

Feature store calculation must enforce strict transaction_timestamp <= as_of_date boundary conditions.

100% Financial Reconciliation

Total revenue in Snowflake enterprise facts must reconcile to the penny against raw transaction inputs.

Automated Zero-Downtime DataOps

GitHub Actions deploys all Databricks workflows and Snowflake DDLs with single-click rollback capability.

Architecture Note

The capstone represents a cohesive enterprise platform. Every component—from ingestion to ML features and CI/CD—must work in harmony, demonstrating complete production readiness.

Development Requirements

Comprehensive engineering requirements across architecture, ingestion, streaming, CDC, ML, CI/CD, and governance.

Developer Responsibilities

The capstone implementation must fulfill requirements across 8 core engineering disciplines:

Multi-Protocol Ingestion Pipelines

Ingest all five heterogeneous enterprise feeds.

Required Checklist

✓Implement batch cloud storage reader with incremental partition tracking.

✓Implement Structured Streaming reader for POS events with maxOffsetsPerTrigger rate limits.

✓Implement CDC micro-batch reader handling INSERT, UPDATE, and DELETE event streams.

✓Implement Python REST client with pagination, auth tokens, and backoff retries.

✓Implement ERP store reference table ingestor.

Expected Outcome

Complete ingestion coverage across all enterprise data sources.

Medallion Lakehouse Architecture

Deploy structured Bronze, Silver, and Gold Delta Lake layers.

Required Checklist

✓Persist append-only Bronze tables with ingestion timestamps, batch IDs, and source provenance.

✓Build Silver tables with typed schemas, entity resolution, and duplicate filtering.

✓Build Gold tables optimized with Z-ORDER and liquid clustering.

Expected Outcome

Clean, scalable Medallion storage architecture.

CDC State Resolution & Deduplication

Materialize current customer master state from change streams.

Required Checklist

✓Apply windowing (row_number by LSN and timestamp desc) to pick latest state within micro-batches.

✓Execute idempotent Delta MERGE updating attributes and handling soft/hard deletes.

✓Maintain immutable historical changelog for compliance and disaster replay.

Expected Outcome

Low-latency, accurate customer master dataset.

Real-Time Streaming & Watermarking

Process streaming transactions and compute operational windowed KPIs.

Required Checklist

✓Enforce event-time watermarking (e.g. 15-minute watermark) to bound state store size.

✓Calculate 5-minute tumbling and 1-hour sliding window operational revenue metrics.

✓Publish streaming updates to Snowflake and Delta via foreachBatch sinks.

Expected Outcome

Low-latency operational visibility for live monitoring.

Leakage-Free ML Feature Store

Engineer behavioral customer features for machine learning.

Required Checklist

✓Build point-in-time observation spine (customer_id × as_of_date).

✓Enforce strict transaction_timestamp <= as_of_date join condition to eliminate data leakage.

✓Compute rolling 30d/90d/180d RFM, category diversity, and velocity features.

✓Support daily incremental feature refresh and persist in Delta Feature Store.

Expected Outcome

Governed, reproducible feature store for data science.

Enterprise Dimensional Modeling

Build and deploy a conformed Star Schema in Snowflake.

Required Checklist

✓Deploy DIM_CUSTOMER (SCD 1/2), DIM_PRODUCT, DIM_STORE, DIM_DATE, and FACT_ENTERPRISE_SALES.

✓Generate durable integer surrogate keys and map unmatched lookups to -1 (Unknown) keys.

✓Create curated semantic presentation views with analyst-friendly business aliases.

Expected Outcome

High-performance Snowflake enterprise data mart.

Automated GitHub Actions CI/CD

Automate testing, packaging, deployment, and rollback.

Required Checklist

✓Configure PR validation workflow running Flake8, Black, TruffleHog, and pytest unit tests.

✓Configure Staging deployment workflow with automated integration test execution.

✓Configure gated Production release workflow with environment approval rules.

✓Implement automated one-click rollback workflow restoring previous Git tags and table versions.

Expected Outcome

Production DataOps deployment automation.

Data Quality, Quarantine & Observability

Enforce zero silent data loss and complete operational monitoring.

Required Checklist

✓Route malformed JSON, corrupt CSV rows, and invalid operation codes to quarantine storage.

✓Monitor source freshness lag and emit alerts upon SLA breach.

✓Automate financial reconciliation queries verifying zero revenue variance between source and target.

Expected Outcome

Full operational observability, governance, and auditability.

Engineering Autonomy

You must make all underlying engineering decisions regarding table partitioning, watermark durations, surrogate key algorithms, and cluster configurations. Document your architectural choices and justify them.

Testing Requirements

Execute comprehensive test suites across 24 rigorous enterprise validation scenarios.

24 Required Verification Scenarios

The capstone platform must be validated against all 24 required test scenarios:

Successful Batch File Ingestion

Scenario

Ingest 100,000 sales transaction records across multiple cloud storage files.

Expected Result

100% of records land in Bronze and Silver with matching row counts and metadata.

Duplicate Batch File Handling

Scenario

Re-deliver identical sales batch files to cloud storage.

Expected Result

Idempotent pipeline logic avoids duplicate fact records and metric inflation.

Late-Arriving Batch File Ingestion

Scenario

Deliver batch file containing transactions from 3 days prior.

Expected Result

File is processed and historical date partitions in Silver/Gold are updated cleanly.

Product REST API Transient Outage

Scenario

Simulate HTTP 503 error on Product API during transaction ingestion.

Expected Result

Sales processing continues; missing product lookups default to -1 without crashing.

Product REST API Rate-Limit Throttling

Scenario

API returns HTTP 429 with Retry-After: 3 header.

Expected Result

Client detects 429, sleeps for 3 seconds, retries, and completes pagination.

Customer CDC INSERT Operation

Scenario

Stream INSERT change event for new customer.

Expected Result

Silver master creates new active customer record with correct profile attributes.

Customer CDC UPDATE Operation

Scenario

Stream UPDATE change event modifying customer email and city.

Expected Result

Silver master updates record in-place with latest attributes and updated_at timestamp.

Customer CDC DELETE Operation

Scenario

Stream DELETE change event for terminated account.

Expected Result

Record is flagged is_deleted=true / status='DELETED' and excluded from active views.

Duplicate CDC Event Suppression

Scenario

Re-stream identical CDC messages with same event_id and LSN.

Expected Result

Pipeline suppresses duplicate messages idempotently without state corruption.

Out-of-Order CDC Event Protection

Scenario

Deliver older UPDATE event after a newer UPDATE has already been processed.

Expected Result

Pipeline detects stale LSN/timestamp and ignores older event, preserving latest state.

Real-Time Streaming Cluster Failure

Scenario

Simulate driver termination mid-stream and re-launch query.

Expected Result

Stream resumes from exact checkpoint offset without missing or duplicating events.

Streaming Burst Traffic Surge

Scenario

Inject 50,000 events in a 5-second burst into streaming topic.

Expected Result

Structured Streaming processes backlog across micro-batches via rate-limits stably.

Malformed Streaming Payload Quarantine

Scenario

Deliver corrupted JSON strings to the streaming event source.

Expected Result

Corrupt records are diverted to bad_records sink; valid transactions proceed normally.

Schema Drift & Evolution

Scenario

Introduce a new optional field into incoming sales JSON files.

Expected Result

Delta Lake schema evolution captures new column cleanly without pipeline crash.

Data Quality & Integrity Failure

Scenario

Inject records with negative monetary amounts or null transaction_ids.

Expected Result

Quarantine framework intercepts invalid records with descriptive error tags.

Downstream Snowflake Connection Failure

Scenario

Simulate Snowflake staging connection timeout during foreachBatch.

Expected Result

Micro-batch retries automatically, succeeds upon reconnect, and maintains state.

Orchestration Task Failure & Retry

Scenario

Simulate transient cluster startup timeout in orchestration DAG.

Expected Result

DAG executes configured exponential retry and completes subsequent tasks.

Full Historical Reprocessing / Backfill

Scenario

Re-run historical pipeline for a past calendar month.

Expected Result

Historical partitions are overwritten atomically without duplicate row inflation.

ML Feature Point-in-Time Leakage Test

Scenario

Inject future transaction at T+10 and compute features for observation date T.

Expected Result

Future transaction has 0 impact on feature vector; zero leakage confirmed.

CI/CD Quality Gate PR Block

Scenario

Submit pull request with unit test failure and Flake8 lint violation.

Expected Result

GitHub Actions workflow blocks PR merge with detailed test failure report.

Automated Rollback Verification

Scenario

Simulate failed production release and trigger rollback workflow.

Expected Result

Jobs and schemas revert to previous Git tag; production remains healthy.

End-to-End Financial Reconciliation

Scenario

Compare total net revenue across all Silver/Gold facts against raw input files.

Expected Result

Variance equals exactly 0.00 across all stores, channels, and date partitions.

Source Freshness SLA Alerting

Scenario

Simulate 4-hour delay on sales file delivery.

Expected Result

Freshness monitoring service flags SLA breach and logs alerting telemetry.

Security & Secret Governance Scan

Scenario

Run TruffleHog secret scan across repository and workflow configurations.

Expected Result

Zero hard-coded credentials, private keys, or passwords detected in source control.

Verification Rule

Passing all 24 verification scenarios with documented test outputs proves your platform is resilient against every category of real-world production failure.

Acceptance Criteria

Verify that the capstone platform meets all production Definition of Done criteria.

Definition of Done

The capstone platform is accepted when all 12 master criteria are satisfied:

All 5 Enterprise Sources Integrated

Batch, Streaming, CDC, REST API, and ERP master feeds ingest reliably into isolated Bronze storage.

Medallion Lakehouse Fully Operational

Bronze raw ledgers, Silver normalized master tables, and Gold analytical marts are fully deployed.

CDC State Materialization Proven

INSERT, UPDATE, and DELETE operations materialize current customer master state with zero corruption.

Real-Time Streaming & Watermarking Active

Low-latency streaming queries process transactions with event-time watermarking and sliding window KPIs.

Leakage-Free ML Feature Store Deployed

Point-in-time behavioral features are computed, validated, and published for data science consumption.

Snowflake Dimensional Star Schema Synced

Conformed dimensions, facts, and executive presentation views are deployed to Snowflake.

100% Financial Reconciliation Proven

Total net revenue in Snowflake matches raw transaction inputs to the penny across all dates.

Automated CI/CD Workflows Active

GitHub Actions automates PR validation, staging integration tests, gated prod release, and rollback.

Secret Governance & Zero Hard-Coded Keys

All credentials, connection strings, and tokens are externalized in secure secret scopes.

Freshness SLAs & Dead-Letter Quarantine

Source arrival lag is monitored continuously and corrupted records are quarantined safely.

All 24 Test Scenarios Pass

Comprehensive test suite passes with documented execution logs and validation proof.

Production Architecture Runbook Published

Complete architectural design documentation, data dictionaries, and operational runbooks are published.

Capstone Sign-Off

Sign-off requires evidence demonstrating complete end-to-end integration, passing all 24 test scenarios, and proving 100% financial reconciliation in Snowflake.

Developer Deliverables

Submit all source modules, Medallion pipelines, feature stores, CI/CD workflows, and test evidence.

Required Deliverables

The capstone submission must include the following 12 artifacts:

Five Multi-Protocol Ingestion Modules

PySpark / Python extractors for Batch, Real-Time Streaming, CDC, REST API, and ERP master data.

Bronze & Dead-Letter Quarantine Framework

Append-only raw persistence writers and dead-letter quarantine handlers for corrupted payloads.

Silver Curation & CDC Materialization Modules

Schema normalization, entity resolution, and idempotent Delta MERGE handlers for customer CDC.

Real-Time Streaming & Windowing Pipeline

Structured Streaming queries computing 5-min tumbling and 1-hr sliding operational KPIs.

Leakage-Free ML Feature Store Pipeline

Point-in-time observation spine, range join engine, and rolling RFM feature transformers.

Snowflake Dimensional DDL & Loader Scripts

Production SQL DDL defining Snowflake Star Schema tables, clustering keys, and presentation views.

Financial Reconciliation & Freshness Suite

Automated variance checkers and freshness SLA monitoring services logging operational telemetry.

GitHub Actions CI/CD Workflow Suite

Workflows in .github/workflows/ for CI quality gates, staging deploy, gated prod release, and rollback.

Comprehensive 24-Scenario Test Suite

Automated unit, integration, and mock test suites covering all 24 required verification scenarios.

Environment Configuration Templates

dev.yaml, staging.yaml, and prod.yaml externalizing cluster parameters, database names, and paths.

Execution & Reconciliation Evidence Log

Logs, telemetry screenshots, and query output tables proving zero variance and successful CI/CD runs.

Enterprise Architecture Guide & Runbook

README covering logical/physical data models, state machines, and operational disaster recovery runbooks.

Submission Principle

Organize your capstone repository cleanly so that peer architects can inspect every tier—from raw ingestion to CI/CD workflows and analytical marts—without ambiguity.

Engineering Constraints

Adhere to strict operational, security, and architectural boundaries for the enterprise capstone.

Required Boundaries

The solution must strictly comply with the following 10 constraints:

Zero Hard-Coded Credentials

All database passwords, API tokens, cloud keys, and certificates must be stored in secure secret vaults.

Zero Silent Data Loss

Every invalid, corrupt, or unparseable record across all 5 feeds must be routed to quarantine storage.

Mandatory Watermarking

All stateful streaming aggregations must specify event-time watermarks to prevent unbounded memory growth.

Zero Future ML Leakage

Feature calculations must strictly enforce transaction_timestamp <= as_of_date boundary conditions.

Strict Idempotency

Re-running any batch, stream micro-batch, or CDC replay must yield identical, deterministic target state.

Fault-Isolated Ingestion

Upstream outages on any single feed must never crash or block processing on other data channels.

Immutable Raw Storage

Bronze landing tables and raw files must remain append-only and never be updated or deleted in place.

Observable Operational Telemetry

All pipelines must emit throughput, latency lag, record counts, and SLA compliance metrics continuously.

Automated DataOps Deployments

Manual production deployments are prohibited; all releases must flow through versioned CI/CD workflows.

Production Code Quality

All modules must be typed, documented, tested, and formatted according to PEP-8 standards.

Constraint Notice

Failing to handle late-arriving CDC events, leaking future data into ML features, or skipping automated rollback will result in capstone review rejection.

Suggested Project Structure

Recommended enterprise repository layout for the capstone platform.

Recommended Project Layout

Structure your capstone repository to separate ingestion protocols, lakehouse tiers, ML feature store, CI/CD, and tests:

DEV-010-enterprise-data-engineering-capstone/
│
├── README.md
├── pyproject.toml
│
├── .github/
│   └── workflows/
│       ├── ci.yml                 # PR quality gates: lint, secret scan, pytest
│       ├── deploy-staging.yml     # Staging deploy & 24-scenario test run
│       ├── deploy-prod.yml        # Gated production promotion
│       └── rollback.yml           # Automated emergency rollback
│
├── config/
│   ├── dev.yaml
│   ├── staging.yaml
│   └── prod.yaml
│
├── src/
│   ├── ingestion/
│   │   ├── batch_sales.py         # Cloud Storage batch file reader
│   │   ├── stream_sales.py        # Structured Streaming POS reader
│   │   ├── cdc_customer.py        # Debezium / Kafka CDC reader
│   │   ├── api_product.py         # REST API product catalog extractor
│   │   └── erp_store.py           # ERP store reference extractor
│   │
│   ├── bronze/
│   │   └── bronze_writer.py       # Append-only raw landing & metadata
│   │
│   ├── silver/
│   │   ├── normalize.py           # Schema normalization & typing
│   │   ├── cdc_merger.py          # LSN ordering & Delta MERGE
│   │   ├── stream_watermark.py    # Event-time watermarking & deduplication
│   │   └── quarantine.py          # Bad-records quarantine handler
│   │
│   ├── gold/
│   │   ├── dimensions/            # Conformed DIM_CUSTOMER, DIM_PRODUCT, DIM_STORE
│   │   ├── facts/                 # FACT_ENTERPRISE_SALES & aggregate marts
│   │   └── streaming_kpis/        # 5-min tumbling & 1-hr sliding KPI queries
│   │
│   ├── ml_features/
│   │   ├── observation_spine.py   # Point-in-time snapshot grid
│   │   ├── rfm_transformers.py    # Rolling 30d/90d/180d RFM features
│   │   └── feature_store_sync.py  # Delta Feature Store & Snowflake sync
│   │
│   └── monitoring/
│       ├── freshness_sla.py       # Source arrival lag & SLA tracker
│       └── reconciliation.py      # Automated financial variance auditor
│
├── ddl/
│   ├── 01_dimensions.sql
│   ├── 02_facts.sql
│   ├── 03_ml_features.sql
│   └── 04_semantic_views.sql
│
├── tests/
│   ├── unit/                      # Pytest transformation unit tests
│   ├── integration/               # End-to-end 24-scenario test suite
│   └── mocks/                     # Mock data generators for all 5 feeds
│
└── docs/
    ├── enterprise_architecture.md # Logical & physical architecture diagrams
    ├── data_dictionary.md         # Schema definitions & metric formulas
    └── disaster_recovery.md       # Full replay & incident recovery runbook

Module Responsibilities

src/ingestion/

Decoupled extractors for Batch, Real-Time Streaming, CDC, REST API, and ERP master data.

src/silver/

Contains schema normalization, CDC state resolution, stream watermarking, and dead-letter quarantine.

src/gold/

Builds enterprise conformed dimensions, central sales fact tables, and sliding window streaming KPIs.

src/ml_features/

Point-in-time observation spines, leakage-free range joins, and rolling RFM feature store builders.

src/monitoring/

Automated freshness lag monitoring, SLA breach alerting, and financial variance checkers.

.github/workflows/

Automated GitHub Actions CI/CD workflows for quality gates, staging tests, prod release, and rollback.

tests/

Automated test harness validating all 24 required verification scenarios.

docs/

Enterprise architecture diagrams, data dictionaries, and disaster recovery playbooks.

Design Rationale

Structuring the repository with clear boundaries between ingestion protocols, Medallion storage tiers, ML feature store, and DataOps automation ensures individual components can be scaled and maintained independently.

Submission Checklist

Final engineering quality checklist before submitting the DEV-010 Capstone.

Final Review Checklist

Verify every checklist item before submitting your capstone challenge:

✓

All 5 Ingestion Feeds Operational

Batch, Streaming, CDC, REST API, and ERP master feeds ingest reliably into Bronze.

✓

Medallion Lakehouse Tiers Verified

Bronze raw ledgers, Silver normalized master tables, and Gold analytical marts are fully deployed.

✓

CDC State Materialization Tested

INSERT, UPDATE, and DELETE operations materialize current customer master state with zero corruption.

✓

Real-Time Streaming & Watermarking Active

Low-latency streaming queries process transactions with event-time watermarking and sliding window KPIs.

✓

Zero Future Data Leakage in ML Features

Point-in-time tests verify that no transaction data occurring after observation timestamp enters features.

✓

Snowflake Dimensional Star Schema Synced

Conformed dimensions, facts, and executive presentation views are deployed to Snowflake.

✓

Financial Reconciliation Proven

Total net revenue in Snowflake matches raw transaction inputs to the penny across all dates.

✓

Automated CI/CD Workflows Active

GitHub Actions automates PR validation, staging integration tests, gated prod release, and rollback.

✓

Zero Hard-Coded Credentials

All credentials, connection strings, and tokens are externalized in secure secret scopes.

✓

Freshness SLAs & Quarantine Active

Source arrival lag is monitored continuously and corrupted records are quarantined safely.

✓

All 24 Test Scenarios Pass

Comprehensive test suite passes with documented execution logs and validation proof.

✓

Architecture Guide & Runbook Published

README contains logical/physical architecture diagrams, data dictionaries, and disaster recovery runbooks.

Ready for Review

Submit the DEV-010 Capstone only after all five ingestion pipelines, Medallion tiers, CDC handlers, real-time streaming queries, ML feature store, Snowflake serving mart, CI/CD automation, and 24 test scenarios have been thoroughly verified.