Build Incremental Sales Data Pipeline
Build a production-oriented incremental sales pipeline that processes continuously arriving transaction data, handles reprocessing safely, and prepares trusted datasets for downstream analytics.
Project Brief
Understand the business problem before designing the technical solution.
Business Problem
A retail business receives sales transaction files from multiple stores throughout the day. New files may arrive at different times, and previously delivered files may occasionally be reprocessed because of upstream retries, operational recovery, or file delivery issues.
The existing process does not provide a reliable incremental processing mechanism. As a result, the data engineering team needs a pipeline that can identify newly available transaction data, process it safely, prevent unnecessary duplication, and make the resulting sales data available for downstream analytics.
Development Objective
Build an incremental sales data pipeline that can process newly arriving transaction data while safely handling duplicate deliveries, repeated execution, and late-arriving records.
The pipeline should produce a trusted transaction-level dataset and a business-ready sales dataset that can be consumed by downstream reporting and analytics workloads.
Expected Outcome
The task is intentionally focused on the engineering problem rather than prescribing a single implementation. Determine an appropriate incremental processing strategy based on the source characteristics, business requirements, and engineering constraints defined throughout this task.
Source System
Understand the incoming sales data, delivery behavior, and source contract.
Source Overview
The source system delivers sales transaction files to cloud storage throughout the business day. Files are generated by individual stores and may arrive at different times rather than following a single predictable delivery window.
A file may contain transactions for multiple stores, and a previously delivered file may occasionally be delivered again because of upstream retries or operational recovery.
Transaction Schema
Each source record represents a sales transaction. The following attributes are available to the pipeline.
Source Delivery Characteristics
The source contract describes what the pipeline receives. The developer is responsible for determining how the incoming data should be discovered, validated, processed, and persisted safely.
Expected Architecture
Define a reliable architecture for incremental transaction processing and downstream analytics.
Architecture Objective
The solution should separate raw ingestion, incremental processing, trusted transaction data, business aggregation, and downstream publishing responsibilities.
The architecture should also provide a reliable mechanism for determining what data needs to be processed during each execution while supporting repeated execution and late-arriving transactions.
Architecture Expectations
The architecture describes the responsibilities and data flow that the solution must satisfy. It does not prescribe a single implementation pattern. The developer is expected to choose appropriate technologies, processing strategies, and control mechanisms within the requirements of this task.
Development Requirements
Build the pipeline capabilities required to process sales data incrementally and reliably.
Required Capabilities
The implementation must satisfy the following functional and engineering requirements. The developer is responsible for determining the appropriate implementation approach.
These requirements define the capabilities the solution must provide. They intentionally do not prescribe a specific notebook structure, incremental-processing mechanism, merge strategy, or orchestration pattern.
Testing Requirements
Demonstrate that the incremental pipeline behaves correctly under normal, repeated, delayed, and invalid processing conditions.
Required Test Scenarios
The developer must validate both the functional output and the incremental behavior of the pipeline. Tests should demonstrate that the solution continues to produce correct results when input conditions change or processing is repeated.
Successful execution of the pipeline for one input dataset is not sufficient. The solution must demonstrate correct behavior when data is repeated, delayed, invalid, reordered, or processed again after a failure.
Acceptance Criteria
The implementation must satisfy these conditions before the development task can be considered complete.
Definition of Done
The pipeline is considered complete only when the following conditions have been demonstrated through implementation, testing, and execution evidence.
A pipeline that runs successfully once is not automatically considered complete. Acceptance depends on demonstrating correct incremental behavior, data correctness, recoverability, downstream consistency, and test evidence.
Developer Deliverables
The completed task should be submitted with the implementation, validation evidence, and supporting engineering artifacts.
Required Deliverables
The implementation is not considered complete until the required development artifacts and supporting evidence are available for review.
The final submission should demonstrate not only that the pipeline was implemented, but also that its incremental, duplicate-handling, late-data, and recovery behavior was validated.
Engineering Constraints
The solution must operate within the following technical and operational boundaries.
Required Engineering Boundaries
The implementation should satisfy the following constraints. These constraints are intended to guide engineering decisions without prescribing one specific implementation pattern.
Constraints define the engineering boundaries of the solution. They should influence implementation decisions while still allowing the developer to justify the specific design pattern chosen.
Suggested Project Structure
Organize the implementation into clear responsibilities so the solution remains maintainable and reviewable.
Recommended Structure
The implementation should be organized so that ingestion, incremental processing, transformation, validation, testing, configuration, and publishing responsibilities remain understandable and maintainable.
DEV-002-incremental-sales-pipeline/
│
├── README.md
│
├── notebooks/
│ ├── 01_ingestion
│ ├── 02_incremental_processing
│ ├── 03_silver_transactions
│ └── 04_gold_sales
│
├── src/
│ ├── ingestion/
│ ├── transformations/
│ ├── quality/
│ ├── incremental/
│ └── publishing/
│
├── tests/
│ ├── unit/
│ ├── integration/
│ └── data_quality/
│
├── config/
│ ├── dev/
│ └── prod/
│
└── docs/
├── architecture.md
└── processing_strategy.mdDirectory Responsibilities
The structure above is a recommendation rather than a mandatory implementation layout. Equivalent structures are acceptable when they maintain clear separation of responsibilities and remain easy for another engineer to understand and maintain.
Submission Checklist
Complete this checklist before submitting the development task for review.
Final Developer Checklist
Before submitting DEV-002, verify each item below. The submission should provide enough evidence for another engineer to understand, execute, test, and review the solution.
Submit DEV-002 only after the implementation, tests, execution evidence, documentation, and acceptance criteria have been reviewed by the developer.