Select Columns from a Config Table

PySpark practice question · Hard · Basics & Select

A metadata-driven pipeline stores which columns of orders to publish in a small config table (column_name, position). Return the orders table restricted to exactly the configured columns, in ascending position order. The config always lists at least one valid column of orders. Row order does not matter.

Example 1

Input orders
order_idcustomer_idregionproductqtyamountorder_date
1011NorthLaptop155000.02024-01-05
1022SouthPhone230000.02024-01-18
1031NorthMouse31500.02024-02-02
1043WestTablet122000.02024-02-15
1054SouthLaptop155000.02024-03-01
1062EastHeadphones24000.02024-03-09
Input config
column_nameposition
order_id1
amount2
Expected output
order_idamount
10155000
10230000
1031500
10422000
10555000
1064000

Example 2

Input orders
order_idcustomer_idregionproductqtyamountorder_date
2015EastPhone115000.02024-04-12
2026WestKeyboard46000.02024-04-20
2035NorthMouse2999.52024-05-03
2047NorthTablet125000.02024-05-21
Input config
column_nameposition
amount3
region1
order_id2
Expected output
regionorder_idamount
East20115000
West2026000
North203999.5
North20425000

Constraints

The output columns are exactly the configured column names, ordered by position.

Solve it in the live editor on this page — run on sample data, then submit against hidden tests.

‹ Most Recent Order · Products in the Top 3 Price Tiers ›

Select Columns from a Config Table – PySpark Interview Question | TechTown
T
Tech Town
Hi, {{ authName }} · Dashboard Log Out Log In Sign Up Free
PySpark Interview Practice

Practice PySpark Online

{{ totalCount }} questions · {{ shHardPct }}% Hard. Real PySpark DataFrame and Spark SQL interview questions. Write PySpark in your browser on TechTown’s Spark engine, run sample tests, then submit against hidden tests.

{{ totalCount }}
Questions
{{ shTopicCount }}
Topics
{{ solvedCount }}
Solved
{{ accuracy }}
Acceptance
Bookmark Question Topic Difficulty Status
{{ r.n }} - {{ r.title }}{{ r.badge }}{{ r.lock }} {{ r.topic }} {{ r.difficulty }} {{ r.statusIcon }}
No questions match these filters.
Showing {{ shRange }} of {{ shFilteredCount }}

Guests can solve {{ freeLimit }} problems for free. Log in or sign up (free) to unlock everything, keep your streak and appear on the leaderboard. New to a topic? Learn it first in the Spark course.

{{ p.num }}. {{ p.title }} {{ p.difficulty }} {{ p.topic }} {{ p.solvedLabel }}
{{ shPointsLabel }} ⏱ {{ shTimer }}
Embed this questionPaste on a blog, course page or LMS. Free to use — educator guide
{{ shEmbedCode }}

{{ p.statement }}

Example {{ ex.n }}
{{ tb.title }}
{{ c.v }}
Expected result
{{ c.v }}
Constraints

{{ p.constraints }}

  1. {{ h }}
Learn the concept first: {{ p.topic }} in the Spark course
Your submissions (last 20, saved on this device)

No submissions yet. Press Submit to judge your code against all tests.

{{ sb.verdict }}
{{ sb.meta }}
🔒 Solution locked

The model solution and step-by-step explanation unlock when you solve this problem or after 3 unsuccessful submissions ({{ shFailsLabel }}). Struggling? Try the hints on the Question tab first.

Why this approach — step by step
  1. {{ st }}
Optimal solution (PySpark)
{{ p.solution }}
Runs in an isolated browser sandbox · {{ engineLabel }}

Write real PySpark: from pyspark.sql import functions as F, Window, spark.sql(...). It runs in your browser on a lightweight Spark engine, and the same code runs unchanged on Databricks or any Spark cluster. Use df.show() or print() to debug.

{{ verdict }} {{ verdictDetail }}
{{ errorText }}
{{ tr.label }} {{ tr.msg }}
Expected
{{ c.v }}
Your output
{{ c.v }}
stdout:
{{ stdout }}

Enjoying your practice journey?

Create your free TechTown account to unlock every problem, track your progress, bookmark questions and keep preparing for interviews.

Log in / Sign up