Difficulty

Advanced

Reward

500 XP · 🔄 Change Data Engineer

Prerequisites

Structured Streaming · Delta Lake · CDC Concepts

Primary Stack

Databricks · PySpark · Delta Lake

Target

Snowflake

Development Task

DEV-005

CDC-Based Customer Data Pipeline

Design and implement a low-latency Change Data Capture (CDC) processing pipeline handling INSERT, UPDATE, and DELETE event streams with out-of-order sequencing, duplicate suppression, replayability, and current-state materialization in Delta Lake and Snowflake.

Project Brief

Understand real-time Change Data Capture (CDC), event stream processing, and current-state replication.

Business Scenario

The core customer operations database processes hundreds of thousands of customer profile updates, new registrations, address modifications, and GDPR account deletions every day. Currently, downstream analytics teams rely on heavy, expensive nightly full-table batch extractions that lock operational databases and introduce a 24-hour latency.

The business requires a low-latency Change Data Capture (CDC) pipeline that continuously ingests operational database change logs (INSERT, UPDATE, DELETE) and maintains an up-to-date, trusted customer master table in Delta Lake and Snowflake without performing full historical reloads.

Development Objective

Design and implement a robust, production-grade CDC pipeline in Databricks and PySpark. The pipeline must ingest CDC event streams, handle out-of-order and duplicate events, determine the true latest state of each customer, apply idempotent MERGE operations into a Silver current-state Delta table, maintain an immutable historical audit log, and publish synchronized customer datasets to Snowflake.

Expected Outcome

CDC Stream Ingestion

Capture continuous INSERT, UPDATE, and DELETE event streams with full metadata fidelity.

Event Sequencing & Windowing

Order multiple changes per customer within and across batches using log sequence numbers (LSN) and event timestamps.

Duplicate Event Suppression

Filter redundant or replayed CDC messages to prevent state corruption.

Current-State Materialization

Maintain an active, query-ready Silver Customer Delta table using idempotent MERGE / delete logic.

Immutable Audit History

Preserve the complete historical timeline of customer modifications for compliance and debugging.

Snowflake Synchronization

Synchronize latest customer state and operational metrics to Snowflake with zero latency lag.

Developer Focus

The primary engineering challenges in CDC are event ordering, handling multiple updates to the same customer within a single micro-batch, processing hard/soft deletes, and ensuring exactly-once business outcomes upon stream replay.

Source System

Inspect CDC event stream format, operation codes, log sequence numbers, and payload structures.

Source Event Stream Overview

The source system is an operational database CDC stream (e.g., Debezium, AWS DMS, or Kafka log stream) emitting row-level change events. For development and testing, CDC streams can be simulated using streaming JSON event files or test event generators.

Source Type

CDC Event Stream (JSON / Kafka)

Operations Supported

INSERT, UPDATE, DELETE

Processing Mode

Structured Streaming / Micro-batch

Ordering Key

event_timestamp + lsn / sequence_id

Primary Business Key

customer_id

Target State

Current Customer Master

Example CDC Event Payloads

Each event represents an atomic database operation on a customer record:

// 1. INSERT EVENT
{
  "event_id": "EVT-1001",
  "customer_id": "C-9021",
  "operation": "INSERT",
  "event_timestamp": "2026-08-30T10:00:00Z",
  "customer_name": "Aarav Sharma",
  "email": "aarav.sharma@example.com",
  "phone": "+91-9876543210",
  "city": "Bengaluru",
  "country": "India",
  "status": "ACTIVE",
  "metadata": {
    "source_table": "customers",
    "lsn": "00000028:000001bc:0001",
    "tx_id": "TX-49201"
  }
}

// 2. UPDATE EVENT
{
  "event_id": "EVT-1002",
  "customer_id": "C-9021",
  "operation": "UPDATE",
  "event_timestamp": "2026-08-30T11:15:00Z",
  "customer_name": "Aarav Sharma",
  "email": "aarav.sharma@newdomain.com",
  "phone": "+91-9876543210",
  "city": "Mumbai",
  "country": "India",
  "status": "ACTIVE",
  "metadata": {
    "source_table": "customers",
    "lsn": "00000028:000001bc:0002",
    "tx_id": "TX-49280"
  }
}

// 3. DELETE EVENT
{
  "event_id": "EVT-1003",
  "customer_id": "C-9021",
  "operation": "DELETE",
  "event_timestamp": "2026-08-30T14:30:00Z",
  "customer_name": null,
  "email": null,
  "phone": null,
  "city": null,
  "country": null,
  "status": "DELETED",
  "metadata": {
    "source_table": "customers",
    "lsn": "00000028:000001bc:0003",
    "tx_id": "TX-49350"
  }
}

CDC Event Schema Fields

event_id*

STRING

Unique identifier for the CDC event message.

customer_id*

STRING

Primary business identifier of the customer record.

operation*

STRING

CDC action: 'INSERT', 'UPDATE', or 'DELETE'.

event_timestamp*

TIMESTAMP

ISO UTC timestamp when the database transaction occurred.

customer_name

STRING

Customer full name (null for DELETE events).

email

STRING

Customer email address.

city / country

STRING

Geographic attributes of the customer.

status*

STRING

Lifecycle status ('ACTIVE', 'INACTIVE', 'SUSPENDED', 'DELETED').

metadata.lsn*

STRING

Log Sequence Number strictly guaranteeing source commit ordering.

Known CDC Source Constraints

Multiple Changes per Micro-Batch

A single ingestion batch may contain an INSERT and two subsequent UPDATEs for the same customer_id.

Out-of-Order Event Arrival

Network latency may cause an older UPDATE to arrive after a newer UPDATE event.

Duplicate CDC Events

At-least-once message delivery yields duplicate event_ids during network reconnects.

Hard vs Soft Deletes

Deletes may require soft-deletion flags (is_deleted=true) or physical record removal.

Late-Arriving Events

Historical change events from hours or days prior may arrive during catch-up periods.

CDC Engineering Contract

The pipeline must achieve deterministic, exactly-once business state regardless of network re-deliveries, batch sizes, or out-of-order message arrivals.

Expected Architecture

Layered CDC lakehouse architecture: Bronze changelog, Silver current-state materialization, and Gold analytics.

Target Architecture Flow

The CDC architecture separates append-only changelog storage (Bronze) from stateful customer materialization (Silver) and analytical aggregations (Gold).

Operational Database CDC Stream

Kafka / Debezium / Cloud Queue

Emits row-level change events (INSERT, UPDATE, DELETE) with transaction metadata.

↓

CDC Ingestion Engine

Structured Streaming / PySpark

Consumes streaming event batches, validates JSON schemas, and coordinates checkpoints.

↓

Bronze — Immutable Event Changelog

Delta Lake (Append-Only)

Stores all raw change events with ingestion metadata, enabling full point-in-time replay.

↓

Change Ordering & Windowing

PySpark Windowing

Deduplicates events and selects the highest LSN / latest timestamp per customer within each micro-batch.

↓

Silver — Current Customer State

Delta Lake (MERGE / CDF)

Maintains the current single-version-of-truth customer master using idempotent Delta MERGE logic.

↓

Gold — Customer Analytics & Marts

Databricks / PySpark

Computes customer lifecycle metrics, churn indicators, and daily registration summaries.

↓

Snowflake Replication Target

Snowflake

Synchronizes current customer records and analytical KPIs for enterprise BI tools.

Layer Responsibilities

Bronze Changelog· Immutable Audit Trail

Append-only Delta table preserving every change event in sequence. Never mutated, enabling complete disaster recovery replay.

Silver Current State· Single Version of Truth

Keyed on customer_id. Applies latest INSERT/UPDATE values and marks or deletes terminated customer records.

Gold Analytics· Downstream Metrics

Calculates customer demographic distributions, active customer counts by region, and daily status churn rates.

Architectural Expectations

Deterministic State Resolution

Within each micro-batch, windowing must pick the change event with the highest LSN / timestamp before executing MERGE.

Idempotent Delta MERGE

Rerunning a micro-batch must result in the exact same customer state without creating duplicate records.

Graceful Delete Handling

DELETE operations must either execute WHEN MATCHED THEN DELETE or update is_deleted=true / status='DELETED'.

Full Replayability

Truncating Silver and replaying the Bronze event changelog from batch 0 must reconstruct the exact current state.

CDC Architecture Principle

Never perform Delta MERGE directly on raw streaming batches containing duplicate customer_ids. Always reduce and deduplicate to the latest change per customer within the batch first!

Development Requirements

Implement the streaming ingestion, windowed deduplication, MERGE logic, and Snowflake sync.

Developer Responsibilities

The implementation must fulfill the following 17 engineering requirements across streaming ingestion, change ordering, state materialization, and validation.

CDC Stream Ingestion & Checkpointing

Establish reliable streaming ingestion with state checkpoints.

Required Checklist

✓Implement PySpark Structured Streaming or micro-batch reader for CDC event files/streams.

✓Configure persistent checkpoint directories for fault-tolerant state recovery.

✓Enforce schema validation on incoming JSON change records.

Expected Outcome

Resilient streaming ingestion without dropped change messages.

Bronze Event Changelog Persistence

Persist an immutable audit log of all raw change events.

Required Checklist

✓Append raw CDC events to Bronze Delta table partitioned by ingestion_date.

✓Preserve original event_id, operation, metadata.lsn, and event_timestamp.

✓Enable Delta Change Data Feed (CDF) on the table if applicable.

Expected Outcome

Complete historical event ledger enabling full replayability.

Operation Code Validation

Validate and categorize incoming operation types.

Required Checklist

✓Assert operation is strictly one of: 'INSERT', 'UPDATE', 'DELETE'.

✓Standardize operation string casing (uppercase trimmed).

✓Quarantine events with unknown or null operation codes.

Expected Outcome

Clean operation routing preventing pipeline failures.

Micro-Batch Change Deduplication

Deduplicate multiple changes for the same customer within a single batch.

Required Checklist

✓Apply PySpark Window: partitionBy('customer_id').orderBy(col('metadata.lsn').desc(), col('event_timestamp').desc()).

✓Filter row_number == 1 to retain only the latest state per customer in the batch.

Expected Outcome

Clean, single-row-per-customer DataFrame ready for MERGE.

Out-of-Order Event Protection

Prevent older delayed events from overwriting newer state in Silver.

Required Checklist

✓Compare incoming event_timestamp / LSN against existing Silver record's last_updated_timestamp.

✓Ensure MERGE updates only apply when incoming_event_timestamp >= existing_last_updated_timestamp.

Expected Outcome

State consistency immune to out-of-order message delivery.

INSERT Operation Processing

Process new customer creations.

Required Checklist

✓Insert new record into Silver table when customer_id does not currently exist.

✓Populate created_at and last_updated_at timestamps.

✓Set is_deleted = false and is_active = true.

Expected Outcome

Accurate new customer onboarding in Silver master.

UPDATE Operation Processing

Apply attribute modifications to existing customer records.

Required Checklist

✓Update customer attributes (name, email, city, status) when customer_id matches.

✓Update last_updated_at timestamp and latest_lsn in Silver.

✓Handle partial updates where unspecified attributes retain current values.

Expected Outcome

Up-to-date customer profile attributes.

DELETE Operation Processing

Handle customer account deletions and GDPR purge requests.

Required Checklist

✓Implement soft-delete logic: set is_deleted = true, status = 'DELETED', deleted_at = event_timestamp.

✓Support physical deletion mode where configured: execute WHEN MATCHED AND operation = 'DELETE' THEN DELETE.

✓Ensure deleted records are filtered from downstream active analytics views.

Expected Outcome

Compliant, accurate customer deletion processing.

Idempotent Delta MERGE Implementation

Execute atomic upsert operations against the Silver table.

Required Checklist

✓Write PySpark DeltaTable.merge() statement with WHEN MATCHED and WHEN NOT MATCHED clauses.

✓Ensure merge condition matches strictly on target.customer_id == source.customer_id.

✓Wrap batch execution in foreachBatch handler for Structured Streaming.

Expected Outcome

Atomic, idempotent state updates across all micro-batches.

Late-Arriving CDC Event Handling

Handle events arriving significantly after their occurrence window.

Required Checklist

✓Log late-arriving events in telemetry audit table.

✓Apply late event only if customer record has not been updated by a subsequent transaction.

Expected Outcome

Graceful handling of delayed change messages.

Silver Current-State Customer Dataset

Maintain the query-optimized Silver customer master table.

Required Checklist

✓Optimize Silver Delta table with Z-ORDER BY (city, status).

✓Include operational audit columns: last_event_id, last_operation, last_lsn, last_updated_at.

Expected Outcome

High-performance Silver customer master table.

Gold Customer Lifecycle Analytics

Generate analytical metrics and aggregated business views.

Required Checklist

✓Calculate active customer count by city and country.

✓Compute daily new customer registration counts and churn rates.

✓Summarize customer status distribution across active, inactive, and deleted accounts.

Expected Outcome

Business-ready Gold tables for executive reporting.

Snowflake Synchronization Pipeline

Replicate current customer state and analytical marts to Snowflake.

Required Checklist

✓Synchronize Silver customer master to Snowflake DIM_CUSTOMER_CURRENT.

✓Synchronize Gold analytics tables to Snowflake AGG_CUSTOMER_METRICS.

✓Ensure synchronization is idempotent and maintains referential consistency.

Expected Outcome

Synchronized Snowflake warehouse accessible by BI users.

Audit History & Lineage Tracking

Maintain complete traceability for compliance and audits.

Required Checklist

✓Log every micro-batch execution with batch_id, records_processed, inserts, updates, deletes, duration.

✓Emit structured JSON metrics to operational logging table.

Expected Outcome

Full auditability for data governance and compliance.

Disaster Recovery & Replay Pipeline

Provide automated mechanism to rebuild Silver state from Bronze.

Required Checklist

✓Build replay script that truncates Silver and re-applies Bronze events in LSN order.

✓Demonstrate that replayed state reconciles 100% with original state.

Expected Outcome

Proven disaster recovery and state reconstruction capability.

Data Quality Assertions & Quarantine

Prevent dirty CDC records from corrupting master tables.

Required Checklist

✓Assert customer_id is non-null and matches valid format.

✓Assert event_timestamp is valid ISO date not in the future.

✓Route invalid events to CDC quarantine Delta table.

Expected Outcome

Zero master data corruption from malformed messages.

Configuration & Environment Management

Externalize all streaming parameters and storage paths.

Required Checklist

✓Externalize checkpoint paths, batch trigger intervals, schema names, and storage URIs in YAML/JSON configs.

Expected Outcome

Seamless deployment across dev, test, and production environments.

Implementation Guidance

You have full freedom to choose your exact merge syntax, streaming trigger intervals, and delete strategy (soft vs hard). Document your rationale and prove its correctness through testing.

Testing Requirements

Thoroughly test INSERT, UPDATE, DELETE, out-of-order events, duplicate streams, and disaster replay.

Required Test Scenarios

Validate the CDC pipeline against the following 13 test scenarios:

New Customer INSERT Scenario

Scenario

Stream an INSERT event for a new customer_id.

Expected Result

Silver table creates a new active customer record with correct attribute values and last_operation = 'INSERT'.

Customer Profile UPDATE Scenario

Scenario

Stream an UPDATE event modifying email and city for an existing customer.

Expected Result

Silver table updates the customer record in-place with new email/city, updating last_updated_at.

Customer Account DELETE Scenario

Scenario

Stream a DELETE event for an existing customer.

Expected Result

Silver table marks record as is_deleted = true / status = 'DELETED' (or removes row if hard-delete configured).

Multiple Changes in a Single Micro-Batch

Scenario

Provide a micro-batch containing 1 INSERT and 2 sequential UPDATEs for the same customer_id.

Expected Result

Pipeline applies the final UPDATE state; Silver contains exactly 1 row with the latest attributes.

Duplicate CDC Event Suppression

Scenario

Re-send identical CDC event messages with same event_id and LSN.

Expected Result

Pipeline processes events idempotently without corrupting data or creating duplicate rows.

Out-of-Order Event Arrival

Scenario

Deliver an older UPDATE event (timestamp 10:00) after a newer UPDATE event (timestamp 11:00) has already been processed.

Expected Result

Pipeline detects stale timestamp/LSN and ignores the older event, preserving the latest state.

Late-Arriving Event Recovery

Scenario

Inject a valid change event with a timestamp from 24 hours prior.

Expected Result

Pipeline logs late-data telemetry and incorporates change only if no newer version exists.

Streaming Crash & Checkpoint Recovery

Scenario

Simulate cluster termination mid-stream and restart pipeline from checkpoint directory.

Expected Result

Pipeline resumes from exact checkpoint offset without data loss or duplicate state corruption.

Full Historical Replay from Bronze

Scenario

Truncate Silver table and re-run batch replay script across all Bronze events.

Expected Result

Reconstructed Silver table matches 100% with previous master table state.

Invalid Operation Code Rejection

Scenario

Deliver an event with operation = 'INVALID_OP'.

Expected Result

Event is rejected and routed to quarantine table; valid events in the batch proceed normally.

Missing Primary Identifier (customer_id = null)

Scenario

Deliver an event missing customer_id.

Expected Result

Event is captured in quarantine table with failure_reason = 'NULL_CUSTOMER_ID'.

State Reconciliation vs Expected Benchmark

Scenario

Process a benchmark dataset of 10,000 mixed INSERT, UPDATE, and DELETE operations.

Expected Result

Final Silver table row counts, active counts, and attribute values reconcile 100% with benchmark.

Snowflake Target Synchronization Test

Scenario

Query Snowflake DIM_CUSTOMER_CURRENT after streaming execution.

Expected Result

Snowflake table matches Silver Delta master table in row counts, active states, and attribute values.

Testing Principle

CDC systems must be rock-solid against out-of-order delivery, crashes, and network re-deliveries. Every edge case must be demonstrated with automated tests.

Acceptance Criteria

Verify that the CDC pipeline fulfills all production Definition of Done criteria.

Definition of Done

The implementation is complete when all 12 criteria are satisfied:

INSERTs Create Customer Records

New customer creations are populated correctly in Silver with all attributes.

UPDATEs Produce Latest State

Modifications update Silver records in-place with latest attributes and timestamps.

DELETEs Handled Correctly

Deletions update soft-delete flags or remove records according to defined delete policy.

Multi-Change Batches Resolved

Multiple updates for the same customer within a batch resolve to the single latest state.

Duplicate Messages Suppressed

Duplicate CDC events do not corrupt state or inflate master table rows.

Out-of-Order Events Controlled

Stale events arriving out-of-order do not overwrite newer master records.

Bronze Changelog Preserved

Immutable append-only Bronze table stores all raw events for audit and replay.

Disaster Recovery Replay Proven

Replaying Bronze changelog reconstructs 100% accurate Silver master state.

Fault-Tolerant Checkpointing

Pipeline recovers seamlessly from unexpected process failures without data loss.

Snowflake Target Synchronized

Current customer master is synchronized to Snowflake with matching row counts.

Dead-Letter Quarantine Active

Malformed payloads and null keys are segregated with rejection diagnostics.

All 13 Test Scenarios Pass

Complete test suite executes successfully with attached validation proof.

Acceptance Rule

Approval requires execution logs proving that multi-event batches, duplicate replays, and out-of-order events leave the Silver master in a 100% correct state.

Developer Deliverables

Submit all streaming code, MERGE logic, replay scripts, tests, and evidence.

Required Deliverables

Provide the following 11 deliverables in your repository:

Streaming CDC Ingestion Module

PySpark streaming/micro-batch module consuming CDC events and managing checkpoints.

Bronze Changelog Persistence Logic

Append-only writer logging all raw change events with ingestion metadata.

Batch Windowing & Deduplication Logic

PySpark logic picking the highest LSN / latest timestamp per customer within each batch.

Silver Delta MERGE Module

Idempotent MERGE implementation applying INSERT, UPDATE, and DELETE actions.

Dead-Letter Quarantine Handler

Module isolating malformed operations or null keys into quarantine storage.

Gold Analytics Aggregation Pipeline

Pipelines computing active customer counts, churn rates, and geographic distributions.

Snowflake Synchronization Module

Connector script replicating current customer master to Snowflake target tables.

Disaster Recovery & Replay Script

Automated script rebuilding Silver state from scratch by replaying Bronze changelog.

CDC Event Generator / Mock Harness

Test harness simulating streaming INSERT, UPDATE, DELETE, and out-of-order events.

Automated Test Suite

Complete test suite covering all 13 required verification scenarios.

CDC Architecture & Operations Guide

README documentation covering state reconciliation, LSN ordering, and recovery playbooks.

Submission Principle

Ensure your test harness allows another engineer to simulate a stream of 10,000 mixed events and verify the outcome.

Engineering Constraints

Adhere to operational and technical boundaries for mission-critical CDC pipelines.

Required Boundaries

The implementation must strictly comply with the following 10 constraints:

Zero Silent Event Loss

Every received CDC event must be logged to Bronze or routed to quarantine with explicit diagnostics.

Mandatory LSN / Timestamp Ordering

State mutations must strictly obey event sequencing; stale events must never overwrite newer state.

Deduplication Before MERGE

Raw batches must be reduced to one change per customer before executing Delta MERGE statements.

Explicit Delete Handling

DELETE events must not be ignored; they must execute configured soft or hard deletion.

Idempotent State Mutations

Re-running any micro-batch must result in deterministic, uncorrupted customer state.

Observable Stream Telemetry

Every batch execution must emit records processed, latency lag, and operation counts.

Configuration Externalization

Storage URIs, checkpoint paths, trigger intervals, and Snowflake credentials must not be hard-coded.

Scalable Distributed Processing

Processing must scale across Spark worker nodes without accumulating unbounded state in driver memory.

Immutable Bronze Ledger

Bronze changelog tables must remain append-only and never be updated or deleted in place.

Production Code Quality

Code must be modular, fully unit-tested, and formatted according to PEP-8 standards.

Constraint Notice

Failing to handle out-of-order events or executing MERGE on un-deduplicated streaming batches will result in code review rejection.

Suggested Project Structure

Recommended repository layout for production CDC streaming pipelines.

Recommended Project Layout

Structure your repository to separate streaming ingestion, deduplication windowing, Delta MERGE, recovery, and tests:

DEV-005-cdc-customer-pipeline/
│
├── README.md
│
├── config/
│   ├── dev.yaml
│   └── prod.yaml
│
├── src/
│   ├── ingestion/
│   │   ├── stream_reader.py    # Structured Streaming / micro-batch consumer
│   │   └── bronze_logger.py    # Immutable append-only Bronze changelog
│   ├── processing/
│   │   ├── window_dedup.py     # LSN & timestamp windowing deduplication
│   │   ├── delta_merger.py     # Idempotent Delta MERGE for INSERT/UPDATE/DELETE
│   │   └── quarantine.py       # Bad record & invalid operation handler
│   ├── analytics/
│   │   └── customer_kpis.py    # Gold customer metrics & churn aggregation
│   ├── replication/
│   │   └── snowflake_sync.py   # Synchronize master state to Snowflake
│   └── recovery/
│       └── replay_engine.py    # Rebuild Silver state from Bronze changelog
│
├── tests/
│   ├── mock_cdc_stream.py      # Event generator (INSERT, UPDATE, DELETE)
│   ├── test_multi_change_batch.py
│   ├── test_out_of_order.py
│   ├── test_duplicate_suppression.py
│   └── test_replay_recovery.py
│
└── docs/
    ├── cdc_architecture.md     # Architecture diagram and state machine
    └── recovery_playbook.md    # Disaster recovery and replay steps

Module Responsibilities

src/ingestion/

Manages streaming readers, checkpoint state, and append-only Bronze changelog writing.

src/processing/

Contains windowing deduplication (row_number by LSN) and idempotent Delta MERGE logic.

src/analytics/

Computes Gold customer lifecycle metrics, active account counts, and churn rates.

src/replication/

Synchronizes current customer master and analytical marts to Snowflake.

src/recovery/

Provides automated disaster recovery engine to rebuild Silver state from Bronze.

tests/

Complete test suite and mock event generator for multi-event, out-of-order, and replay tests.

docs/

State transition diagrams, LSN ordering documentation, and disaster recovery playbooks.

Design Rationale

Decoupling the windowing deduplication module from the Delta MERGE module allows unit-testing the state resolution logic in isolation without needing live database connections.

Submission Checklist

Final engineering quality checklist before submitting DEV-005.

Final Review Checklist

Verify every checklist item before submitting your CDC pipeline:

✓

Streaming Ingestion & Checkpointing Configured

Structured Streaming / micro-batch pipeline runs with fault-tolerant checkpoint directories.

✓

Bronze Changelog Ledger Preserved

Append-only Bronze table stores all raw change events with LSN and timestamps.

✓

Micro-Batch Windowing Deduplication Enforced

Windowing selects the highest LSN change per customer before executing MERGE.

✓

INSERT, UPDATE, DELETE Operations Verified

All three operation types mutate Silver master state accurately according to business rules.

✓

Out-of-Order & Duplicate Protection Tested

Stale events and duplicate deliveries do not overwrite newer master records.

✓

Idempotent Delta MERGE Validated

Re-running micro-batches yields identical, deterministic customer master state.

✓

Disaster Recovery Replay Demonstrated

Replaying Bronze changelog from scratch reconstructs 100% accurate Silver state.

✓

Gold Lifecycle Analytics Populated

Active customer counts and churn metrics are computed accurately.

✓

Snowflake Synchronization Active

Current customer master is synchronized to Snowflake target tables without row inflation.

✓

Quarantine Dead-Letter Active

Malformed payloads and null customer_ids are diverted with rejection telemetry.

✓

All 13 Test Scenarios Executed

Automated test suite passes with execution logs and state reconciliation evidence.

✓

Configuration Externalized

All storage URIs, trigger intervals, and database credentials are fully externalized.

✓

Documentation Complete

README contains architecture diagrams, state machines, and disaster recovery playbooks.

Ready for Review

Submit DEV-005 only after the streaming ingestion, Bronze changelog, windowing deduplication, Delta MERGE, Snowflake sync, disaster replay, and test suites have been verified.