Difficulty

Expert

Reward

800 XP · 🚀 Production Data Engineer

Prerequisites

GitHub Actions · PySpark · CI/CD · Cloud

Primary Stack

GitHub Actions · Databricks · Snowflake

Target

Multi-Environment Deployment

Development Task

DEV-008

Production CI/CD Data Platform

Design and implement a production-grade CI/CD automation platform for data engineering pipelines with GitHub Actions, automated linting, unit and integration test runners, multi-environment isolation (Dev → Staging → Prod), secret vault management, and automated zero-downtime rollback capabilities.

Project Brief

Understand the engineering challenges of automated, secure, multi-environment DataOps deployments.

Business Scenario

The enterprise data engineering team maintains dozens of PySpark pipelines, Delta Lake tables, and Snowflake reporting marts. However, code releases are currently executed manually via developer laptops and ad-hoc Databricks workspace uploads.

Manual deployments have resulted in recurring production outages caused by un-tested schema modifications, leaked credentials in source repositories, configuration drift between testing and production environments, and lack of rollback mechanisms when pipeline jobs fail during midnight processing windows.

Development Objective

Architect and build an end-to-end DataOps CI/CD deployment platform using GitHub Actions, Databricks Asset Bundles / CLI, and Snowflake automation. The workflow must enforce automated linting and formatting, execute comprehensive unit and integration tests against isolated test environments, manage secrets securely without committing credentials, orchestrate gated approvals into staging and production, and support one-click rollback on deployment regression.

Expected Outcome

Automated PR Quality Gates

Block pull requests automatically upon linting violations, test regressions, or syntax errors.

Isolated Test Execution

Run integration tests against ephemeral test schemas before promoting code to staging.

Multi-Environment Staging

Promote artifacts deterministically through Dev → Staging/QA → Production with gated approvals.

Zero Committed Secrets

Inject all credentials via GitHub Secrets, cloud secret vaults, and least-privilege service principals.

Automated Rollback Engine

Restore previous stable pipeline versions and table snapshots automatically upon deployment failure.

Complete Auditability

Track every deployment release with Git SHA, author, approvals, test logs, and target environment state.

Developer Focus

Expert data engineering requires mastering release engineering and DataOps. Your platform must demonstrate that pipelines can be modified, tested, validated, and deployed to production continuously without human intervention or credential exposure.

Source System

Inspect the repository structure, target cloud environments, and deployment artifact contracts.

Environment Topologies Overview

The CI/CD platform deploys across three strictly separated environments with isolated storage, compute clusters, and access policies.

CI/CD Orchestrator

GitHub Actions Workflows

Branching Strategy

Trunk-Based / GitFlow (Feature -> Main)

Dev Environment

Ephemeral test schemas & sandbox clusters

Staging Environment

Pre-production data mirror & integration tests

Prod Environment

Protected live Lakehouse & Snowflake DW

Security Model

OIDC Service Principals & Secret Vaults

Deployment Pipeline Stages

01. Code Hygiene*

LINT / FORMAT

Flake8, Black, isort, and sqlfluff validate Python and SQL formatting.

02. Unit Testing*

PYTEST

Executes mocked PySpark transformation unit tests with 100% logic coverage.

03. Build & Package*

WHEEL / BUNDLE

Compiles Python wheels and packages Databricks Asset Bundles / DDL manifests.

04. Integration Test*

STAGE DEPLOY

Deploys to Staging environment and executes end-to-end data pipeline with validation tests.

05. Production Release*

PROD DEPLOY

Gated manual approval deploys pipeline to Production with post-deployment sanity smoke tests.

Known CI/CD Operational Constraints

Strict Zero Secrets in Code

Any committed API token, password, or private key immediately fails the pre-commit and CI build.

Protected Production Branches

Direct pushes to main/master are blocked; all changes must arrive through validated pull requests.

Environment Configuration Parity

Code must be identical across environments; only externalized configs and secrets may vary.

Failed Deployment Rollback

If post-deployment smoke tests fail, the workflow must revert cluster jobs and schemas to previous release.

Security Standard

Hard-coding database credentials or using administrative root credentials in CI workflows is strictly prohibited. Use scoped service principals with least-privilege RBAC.

Expected Architecture

Visualizing the automated deployment lifecycle from developer commit to production release.

CI/CD Pipeline Flow Architecture

The workflow automates validation on pull requests, tests artifacts in staging, and releases to production through gated workflows.

Developer Pull Request

Git / GitHub PR

Feature branch submitted against main; triggers automated CI validation matrix.

↓

Automated Quality Gates

Black / Flake8 / Pytest / SQLFluff

Run static code analysis, security credential scan (TruffleHog), and fast PySpark unit tests.

↓

Artifact Build & Release Staging

Python Wheel / Databricks CLI

Package source modules into versioned wheels and generate environment deployment manifests.

↓

Staging Deployment & Integration Test

Databricks Staging / Snowflake QA

Deploy pipeline to staging, trigger test execution on sample dataset, and assert metric reconciliation.

↓

Gated Production Deployment

GitHub Environments / Approvals

Authorized engineer reviews test summary and approves production release with zero-downtime cutover.

↓

Post-Deploy Validation & Monitoring

Smoke Tests / Datadog / PagerDuty

Execute production smoke tests and monitor pipeline health; trigger automated rollback on failure.

Architecture Expectations

Deterministic Artifact Promotion

The exact compiled artifact tested in Staging must be deployed to Production—never rebuild separately.

Secrets Isolation

Development, Staging, and Production secrets must exist in distinct secret vaults with zero cross-environment access.

Auditable Git Traceability

Every deployed pipeline job in Databricks and Snowflake table DDL must be tagged with its originating Git SHA.

Automated Health Assertions

Deployments must automatically verify database connectivity and table accessibility before closing the release.

DataOps Principle

A great data engineer treats deployment pipelines with the same rigor as data pipelines. Automate everything, eliminate manual steps, and ensure every release is fully reversible.

Development Requirements

Implement the CI workflows, automated testing frameworks, environment separation, and deployment scripts.

Developer Responsibilities

The implementation must fulfill the following 15 engineering requirements across CI automation, testing, secret management, and multi-environment deployment.

Repository Structure & Git Workflow

Establish a clean repository layout and branch protection rules.

Required Checklist

✓Structure repository with separated src/, tests/, config/, ddl/, and .github/workflows/.

✓Enforce branch protection requiring linear history and green PR checks before merge.

Expected Outcome

Clean, standard repository ready for CI/CD automation.

Automated Linting & Code Hygiene

Enforce strict code formatting and static analysis.

Required Checklist

✓Configure Flake8, Black, isort, and SQLFluff in pre-commit and GitHub Actions.

✓Fail PR builds immediately upon linting or styling violations.

Expected Outcome

Consistent, readable code standards across the engineering team.

Credential & Secret Scanning

Prevent accidental leakage of passwords, keys, and tokens.

Required Checklist

✓Integrate TruffleHog or Gitleaks into PR validation workflows.

✓Scan all commits, environment files, and configuration templates for leaked credentials.

Expected Outcome

Zero secrets committed to Git history.

Unit Testing Automation (Pytest)

Execute fast, isolated unit tests on every pull request.

Required Checklist

✓Write unit tests with pytest for all transformation, parsing, and cleaning functions.

✓Use local PySpark sessions or mocked DataFrames without cloud cluster dependencies.

✓Enforce minimum 85% test coverage requirement.

Expected Outcome

Rapid PR feedback in under 2 minutes.

Integration Testing in Ephemeral Environments

Verify end-to-end pipeline execution against real compute.

Required Checklist

✓Deploy pipeline code to a dedicated ephemeral Staging schema.

✓Execute integration test run on sample input files and verify output Delta tables.

✓Clean up ephemeral test artifacts upon test completion.

Expected Outcome

High confidence that pipeline operates correctly on target infrastructure.

Packaging & Artifact Compilation

Compile reproducible, versioned deployment artifacts.

Required Checklist

✓Build Python wheel packages containing reusable library modules.

✓Generate version tags based on Git release tags (e.g., v1.4.2-sha).

Expected Outcome

Immutable, versioned release artifacts.

Multi-Environment Configuration Separation

Decouple code from environment-specific parameters.

Required Checklist

✓Maintain distinct dev.yaml, staging.yaml, and prod.yaml configuration files.

✓Externalize database names, warehouse sizes, storage paths, and cluster specs.

Expected Outcome

Single codebase deployed seamlessly across all environments.

Secret Management & Least Privilege

Securely inject credentials into deployment runners.

Required Checklist

✓Use GitHub Encrypted Secrets and OpenID Connect (OIDC) cloud federation.

✓Ensure deployment runners authenticate using scoped service principals.

✓Mask all secret values in workflow console logs.

Expected Outcome

Secure, non-interactive credential injection.

Databricks Pipeline & Job Deployment

Automate deployment of Databricks workflows and jobs.

Required Checklist

✓Deploy updated wheel files and job definitions using Databricks Asset Bundles / CLI.

✓Update job schedules, cluster configurations, and dependency libraries automatically.

Expected Outcome

Automated, declarative Databricks job updates.

Snowflake DDL & Schema Migration

Automate database schema migrations and view updates.

Required Checklist

✓Implement idempotent SQL DDL migration runner for Snowflake DIM_*, FACT_*, and VIEW definitions.

✓Apply table alterations and grants cleanly without dropping existing data.

Expected Outcome

Automated, version-controlled database schema management.

Gated Production Promotion & Approvals

Enforce authorized review before modifying production.

Required Checklist

✓Configure GitHub Environment protection rules with required team approvals for 'production'.

✓Display automated test summaries and staging run evidence in the PR release notes.

Expected Outcome

Controlled, compliant production release process.

Post-Deployment Smoke Tests

Validate production health immediately following release.

Required Checklist

✓Execute smoke test suite verifying database connectivity, table permissions, and sample queries.

✓Assert pipeline execution completes with exit code 0.

Expected Outcome

Immediate verification of production stability.

Automated Rollback Mechanism

Restore previous stable state upon deployment failure.

Required Checklist

✓Implement automated rollback workflow that reverts Databricks jobs to previous Git tag.

✓Utilize Delta Lake RESTORE TABLE TO VERSION_AS_OF if schema migration causes regression.

Expected Outcome

Instant recovery capability minimizing production downtime.

Deployment Auditing & Telemetry

Maintain full operational audit trail of all releases.

Required Checklist

✓Log release metadata: deployment_id, git_sha, author, timestamp, status, and duration.

✓Send notifications to Slack/Teams with deployment outcomes and test reports.

Expected Outcome

Complete visibility for data governance and compliance.

Comprehensive CI/CD Documentation

Document DataOps procedures and developer guides.

Required Checklist

✓Write documentation detailing branching conventions, local pre-commit setup, and emergency rollback.

Expected Outcome

Complete developer and operations documentation.

Implementation Guidance

Write clean, modular GitHub Actions YAML workflows in `.github/workflows/` with reusable composite actions for testing, linting, and cloud authentication.

Testing Requirements

Verify PR quality gates, test execution, deployment failures, security scans, and rollback mechanisms.

Required Test Scenarios

Validate the CI/CD platform against the following 12 test scenarios:

Successful Pull Request Validation

Scenario

Submit a valid feature branch PR with clean code and passing unit tests.

Expected Result

GitHub Actions workflow runs linting, secret scanning, and pytest; all checks pass green.

Unit Test Failure Gate

Scenario

Introduce an intentional assertion failure into a transformation unit test.

Expected Result

Pytest fails; GitHub Actions blocks PR merge with clear test failure diagnostics.

Linting & Code Formatting Failure

Scenario

Submit code with trailing whitespace and unused imports violating Flake8 rules.

Expected Result

Flake8 step fails and halts the CI workflow before executing downstream tests.

Committed Secret Detection

Scenario

Introduce a dummy API token string into a configuration file.

Expected Result

Secret scanner detects credential, fails the build, and alerts the developer.

Staging Integration Test Execution

Scenario

Merge approved PR to staging branch.

Expected Result

Workflow deploys code to Staging environment, runs integration test pipeline, and verifies output tables.

Unauthorized Production Deployment Attempt

Scenario

Attempt to trigger production deployment without required environment approval.

Expected Result

GitHub Environment gate holds deployment in pending state until authorized reviewer signs off.

Successful Production Promotion

Scenario

Authorized lead approves production release.

Expected Result

Artifact is deployed to Production; Databricks jobs and Snowflake tables are updated with 0 errors.

Post-Deployment Smoke Test Failure

Scenario

Simulate a broken database connection during post-deploy verification.

Expected Result

Smoke test step fails, flags deployment as failed, and triggers rollback alert.

Automated Rollback Execution

Scenario

Trigger the automated rollback workflow following a failed release.

Expected Result

Workflow reverts Databricks job definition to previous Git tag and restores stable schema.

Configuration Drift Validation

Scenario

Deploy with mismatched configuration keys between staging and production.

Expected Result

Config validation step identifies missing keys and aborts deployment prior to modifying resources.

Snowflake DDL Migration Idempotency

Scenario

Execute the Snowflake DDL migration runner twice in succession.

Expected Result

Second execution completes cleanly without dropping tables or altering existing data.

Deployment Audit Trail Verification

Scenario

Inspect release history in GitHub Actions and deployment logging table.

Expected Result

Every release records Git SHA, committer, timestamp, test outputs, and target environment.

Testing Principle

A deployment pipeline is only trustworthy if you have proven how it responds to failures. Demonstrating that broken builds fail safely and trigger automated rollback is essential.

Acceptance Criteria

Verify that the CI/CD platform fulfills all production Definition of Done criteria.

Definition of Done

The CI/CD platform is accepted when all 10 criteria are met:

Automated PR Quality Gates Active

Pull requests automatically run linting, secret scanning, and unit tests with enforced branch protection.

Zero Secrets in Code Enforced

No API tokens, private keys, or passwords exist in repository source files or commit history.

Unit & Integration Tests Automated

Pytest unit tests and Staging integration test suites execute automatically within workflows.

Multi-Environment Separation Complete

Dev, Staging, and Production environments use isolated configs, storage, and secret vaults.

Gated Production Approval Workflow

Production releases require explicit review and approval from authorized engineering leads.

Databricks & Snowflake Automation

Databricks workflows and Snowflake DDL schemas are deployed declaratively without manual UI steps.

Automated Rollback Proven

Failed deployments can be rolled back to the previous stable release with a single workflow trigger.

All 12 Test Scenarios Pass

Complete test suite passes with documented workflow execution logs and validation proof.

Deployment Auditability Established

All production changes are traceable to specific pull requests, Git SHAs, and approval records.

Comprehensive Runbook Published

Documentation contains developer setup guides, branching conventions, and incident rollback procedures.

Acceptance Rule

Sign-off requires execution logs demonstrating both a successful automated deployment and a clean automated rollback following a simulated failure.

Developer Deliverables

Submit all workflow definitions, testing suites, migration scripts, and DataOps documentation.

Required Deliverables

The submission must include the following 10 artifacts:

GitHub Actions CI Workflow (.github/workflows/ci.yml)

Workflow running linting, secret scanning, formatting, and unit tests on pull requests.

Staging Deployment Workflow (.github/workflows/deploy-staging.yml)

Workflow deploying artifacts to Staging and executing integration test suites.

Production Release Workflow (.github/workflows/deploy-prod.yml)

Gated workflow deploying to Production with smoke testing and approval gates.

Automated Rollback Workflow (.github/workflows/rollback.yml)

Workflow reverting Databricks jobs and database schemas to previous stable tags.

Pytest Unit & Integration Test Suites

Comprehensive test suite covering transformations, null handling, schemas, and pipeline jobs.

Snowflake DDL Migration Runner

Python / SQL migration script applying schema updates idempotently across environments.

Environment Configuration Templates

dev.yaml, staging.yaml, and prod.yaml externalizing cluster parameters and database names.

Pre-Commit Hooks Configuration (.pre-commit-config.yaml)

Local developer pre-commit hooks enforcing Black, Flake8, and credential checks before commit.

CI/CD Execution & Rollback Evidence

Logs and screenshots of successful PR validation, staging integration test, and rollback runs.

DataOps Guide & Operational Runbook

README documentation covering branching strategy, secret management, and emergency rollbacks.

Submission Principle

Ensure your GitHub Actions YAML workflows are syntactically valid and modularized into clear jobs with descriptive step names.

Engineering Constraints

Adhere to strict operational, security, and architectural boundaries for enterprise CI/CD.

Required Boundaries

The solution must strictly comply with the following 10 constraints:

Zero Secrets in Source Control

Never commit passwords, API keys, or private certificates to repository branches.

No Direct Pushes to Production

Direct commits to main/production branches must be blocked by repository branch protection.

No Manual Production Deployments

All production releases must flow through the automated GitHub Actions pipeline with auditable logs.

Immutable Build Artifacts

Promote the exact compiled artifact tested in Staging to Production; never recompile code for prod.

Isolated Secret Vaults

Development, Staging, and Production secrets must exist in distinct scopes with zero shared keys.

Idempotent Database Migrations

Snowflake schema migration scripts must be safely re-runnable without dropping tables or duplicating data.

Mandatory Automated Testing Gate

No PR may be merged unless all unit and linting tests have passed green.

Explicit Deployment Timeouts

All workflow jobs must define timeout limits to prevent hanging runs during network outages.

Deterministic Configuration Resolution

Code must never infer environment based on hostname or IP; environment must be explicitly configured.

Auditable Change Traceability

Every deployed pipeline job in Databricks and Snowflake must be tagged with its originating Git SHA.

Constraint Notice

Bypassing quality gates, disabling secret scanners, or performing manual un-tracked production releases violates enterprise compliance standards.

Suggested Project Structure

Recommended repository layout for production CI/CD DataOps engineering.

Recommended Project Layout

Structure your repository to separate workflows, application code, test suites, database migrations, and configs:

DEV-008-production-cicd-platform/
│
├── README.md
├── .pre-commit-config.yaml      # Local pre-commit lint & secret scan
├── pyproject.toml               # Build dependencies & tool configs
│
├── .github/
│   ├── workflows/
│   │   ├── ci.yml               # PR validation: lint, secret scan, pytest
│   │   ├── deploy-staging.yml   # Auto-deploy to Staging + integration test
│   │   ├── deploy-prod.yml      # Gated deployment to Production
│   │   └── rollback.yml         # Automated rollback to previous Git tag
│   └── actions/
│       └── setup-dataops/       # Composite action for auth & tool setup
│
├── config/
│   ├── dev.yaml
│   ├── staging.yaml
│   └── prod.yaml
│
├── src/
│   ├── pipeline/
│   │   ├── ingest.py            # Core pipeline ingestion logic
│   │   ├── transform.py         # PySpark business transformations
│   │   └── export.py            # Target warehouse export logic
│   └── utils/
│       ├── config_loader.py     # Environment configuration resolver
│       └── telemetry.py         # Structured execution telemetry
│
├── ddl/
│   ├── migrations/              # Versioned SQL migration scripts
│   │   ├── V1__init_schema.sql
│   │   └── V2__add_indexes.sql
│   └── migration_runner.py      # Idempotent Snowflake migration runner
│
├── tests/
│   ├── unit/                    # Fast isolated transformation unit tests
│   │   ├── test_transform.py
│   │   └── test_config.py
│   └── integration/             # End-to-end pipeline integration tests
│       └── test_pipeline_e2e.py
│
└── docs/
    ├── branching_strategy.md    # GitFlow / Trunk-based workflow guide
    └── rollback_playbook.md     # Emergency production recovery playbook

Module Responsibilities

.github/workflows/

GitHub Actions workflow definitions for CI quality gates, staging, prod, and rollback.

src/pipeline/

Production pipeline application code packaged into deployable Python wheel artifacts.

ddl/migrations/

Idempotent SQL migration scripts defining tables, clustering keys, and views in Snowflake.

tests/unit/

Fast mocked Pytest unit tests executing in under 2 minutes during PR validation.

tests/integration/

End-to-end pipeline validation tests executed in Staging environment.

config/

Environment-specific YAML configuration files resolving paths, database names, and clusters.

docs/

DataOps architecture documentation, branching rules, and incident recovery playbooks.

Design Rationale

Separating unit tests from integration tests allows PR checks to run rapidly on lightweight runners, reserving heavy cloud cluster compute for post-merge staging integration tests.

Submission Checklist

Final engineering quality checklist before submitting DEV-008.

Final Review Checklist

Verify every checklist item before submitting your CI/CD platform:

✓

GitHub Actions CI Workflow Operational

PR validation runs Black, Flake8, secret scanning, and pytest with green status.

✓

Branch Protection & Linear History Enforced

Direct pushes to main are blocked; PRs require passing checks before merge.

✓

Secret Scanner Active & Tested

TruffleHog / Gitleaks successfully detects and blocks committed credential strings.

✓

Multi-Environment Configuration Separated

dev.yaml, staging.yaml, and prod.yaml cleanly isolate all environment-specific paths.

✓

Staging Deployment & Integration Tests Passed

Pipeline deploys to Staging and executes end-to-end integration tests on sample data.

✓

Gated Production Approval Verified

Production deployment halts for authorized approval before executing.

✓

Databricks & Snowflake Deployment Automated

Databricks workflows and Snowflake DDL migrations execute declaratively via CLI.

✓

Automated Rollback Workflow Tested

Rollback workflow restores previous stable Git tag and schema state upon failure.

✓

All 12 Test Scenarios Pass

Complete test suite passes with documented workflow execution logs and validation proof.

✓

DataOps Runbook Published

docs/rollback_playbook.md contains complete developer and incident recovery guides.

✓

Code Quality & Security Compliance

Workflows and scripts are modular, documented, and free of hard-coded credentials.

Ready for Review

Submit DEV-008 only after the PR quality gates, staging integration tests, production approval workflows, automated rollback, and DataOps documentation have been thoroughly verified.