Remove Exact Duplicate Rows

PySpark practice question · Easy · Nulls & Cleaning

A faulty ingestion job loaded some orders twice. Remove rows that are exact duplicates across ALL columns (two rows with NULL in the same column count as equal). Rows that differ in any column are kept.

Example 1

Input orders
order_idcustomer_idregionproductqtyamountorder_date
1101NLaptop1500.02024-01-01
1101NLaptop1500.02024-01-01
2102SMobile1300.02024-01-02
Expected output
order_idcustomer_idregionproductqtyamountorder_date
1101NLaptop15002024-01-01
2102SMobile13002024-01-02

Example 2

Input orders
order_idcustomer_idregionproductqtyamountorder_date
3103EMonitor1900.02024-01-03
3103EMonitor2900.02024-01-03
Expected output
order_idcustomer_idregionproductqtyamountorder_date
3103EMonitor19002024-01-03
3103EMonitor29002024-01-03

Constraints

Return all columns of orders in their original order. Row order does not matter.

Solve it in the live editor on this page — run on sample data, then submit against hidden tests.

‹ Drop Rows Missing Key Fields · Profile NULLs per Column ›

Remove Exact Duplicate Rows – PySpark Interview Question | TechTown
T
Tech Town
Hi, {{ authName }} · Dashboard Log Out Log In Sign Up Free
PySpark Interview Practice

Practice PySpark Online

{{ totalCount }} questions · {{ shHardPct }}% Hard. Real PySpark DataFrame and Spark SQL interview questions. Write PySpark in your browser on TechTown’s Spark engine, run sample tests, then submit against hidden tests.

{{ totalCount }}
Questions
{{ shTopicCount }}
Topics
{{ solvedCount }}
Solved
{{ accuracy }}
Acceptance
Bookmark Question Topic Difficulty Status
{{ r.n }} - {{ r.title }}{{ r.badge }}{{ r.lock }} {{ r.topic }} {{ r.difficulty }} {{ r.statusIcon }}
No questions match these filters.
Showing {{ shRange }} of {{ shFilteredCount }}

Guests can solve {{ freeLimit }} problems for free. Log in or sign up (free) to unlock everything, keep your streak and appear on the leaderboard. New to a topic? Learn it first in the Spark course.

{{ p.num }}. {{ p.title }} {{ p.difficulty }} {{ p.topic }} {{ p.solvedLabel }}
{{ shPointsLabel }} ⏱ {{ shTimer }}
Embed this questionPaste on a blog, course page or LMS. Free to use — educator guide
{{ shEmbedCode }}

{{ p.statement }}

Example {{ ex.n }}
{{ tb.title }}
{{ c.v }}
Expected result
{{ c.v }}
Constraints

{{ p.constraints }}

  1. {{ h }}
Learn the concept first: {{ p.topic }} in the Spark course
Your submissions (last 20, saved on this device)

No submissions yet. Press Submit to judge your code against all tests.

{{ sb.verdict }}
{{ sb.meta }}
🔒 Solution locked

The model solution and step-by-step explanation unlock when you solve this problem or after 3 unsuccessful submissions ({{ shFailsLabel }}). Struggling? Try the hints on the Question tab first.

Why this approach — step by step
  1. {{ st }}
Optimal solution (PySpark)
{{ p.solution }}
Runs in an isolated browser sandbox · {{ engineLabel }}

Write real PySpark: from pyspark.sql import functions as F, Window, spark.sql(...). It runs in your browser on a lightweight Spark engine, and the same code runs unchanged on Databricks or any Spark cluster. Use df.show() or print() to debug.

{{ verdict }} {{ verdictDetail }}
{{ errorText }}
{{ tr.label }} {{ tr.msg }}
Expected
{{ c.v }}
Your output
{{ c.v }}
stdout:
{{ stdout }}

Enjoying your practice journey?

Create your free TechTown account to unlock every problem, track your progress, bookmark questions and keep preparing for interviews.

Log in / Sign up