Skip to main content
DatabricksPySparkSnowflakeApache Airflow

ISA-006 — Duplicate Transaction Removal

🟢 Beginner

⏱ 15 Minutes

⭐ 10 Points

🔓 Free

💻 SQL | PySpark

🏢 Retail Analytics

Business Context​

A retail company receives transaction data from multiple store systems.

Due to a source-system bug, some transactions are loaded multiple times into the analytics platform.

These duplicate records inflate revenue calculations and create incorrect business reports.

The analytics team must remove duplicate transactions before loading data into reporting tables.

Business Impact​

Duplicate transactions can cause:

  • Inflated revenue
  • Incorrect KPIs
  • Reporting inaccuracies
  • Business decision errors

Accurate transaction counts are critical for trustworthy analytics.

Dataset​

customer_transactions
transaction_idcustomer_idamount
T001101100
T001101100
T002102200
T003103150
T003103150

Task​

Remove duplicate transaction records.

Return only unique transactions.

Expected Output​

expected_output
transaction_idcustomer_idamount
T001101100
T002102200
T003103150

Constraints​

  • Duplicate rows may exist.
  • Only unique transaction records should remain.
  • Return all distinct transactions.

Supported Languages​

✅ SQL

✅ PySpark

Hint

Look for a method that returns unique rows from a dataset.

Solution​

🔒 Premium Solution

Premium members receive:

  • SQL Solution
  • PySpark Solution
  • Step-by-Step Explanation
  • Optimization Discussion

Notebook Workspace

Cell 1Dataset: customer_transactions
⏱ 15:00
Loading...
Run Results
✓ Workspace Ready
Rows Returned: --
Execution Time: --
Engine: Coming Soon
Execution Engine Coming Soon