Section 1: Spark Architecture & Execution

Q1: What is the role of the Driver in a Spark application?

A. Runs tasks on worker nodes
B. Converts user code into Spark jobs and schedules tasks
C. Stores cached data in memory
D. Acts as a distributed file system

✅ Correct Answer: B
Explanation: The Driver converts user code into Spark jobs, schedules tasks, and collects results, while Executors run tasks and store cached data.

Q2: What is a Spark Executor responsible for?

A. Managing cluster resources
B. Running tasks and storing cached data
C. Defining the DAG
D. Handling user authentication

✅ Correct Answer: B
Explanation: Executors run tasks assigned by the Driver and store cached data in memory/disk.

Q3: What is Lazy Evaluation in Spark?

A. Spark executes transformations immediately
B. Spark delays execution until an action is called
C. Spark skips failed tasks automatically
D. Spark optimizes disk storage

✅ Correct Answer: B
Explanation: Spark lazily evaluates transformations (e.g., filter(), select()) and only executes them when an action (e.g., count(), collect()) is triggered.

Q4: What is a DAG in Spark?

A. A distributed file system
B. A logical execution plan of transformations
C. A storage format for DataFrames
D. A type of join operation

✅ Correct Answer: B
Explanation: The Directed Acyclic Graph (DAG) represents the sequence of transformations before optimization.

Q5: How does Spark recover from an Executor failure?

A. Restarts the Driver
B. Recomputes lost partitions using lineage
C. Switches to a backup cluster
D. Ignores the failed tasks

✅ Correct Answer: B
Explanation: Spark uses RDD lineage to recompute lost data partitions from the source.

Section 2: Spark DataFrame API

Q6: What is the difference between cache() and persist()?

A. cache() is used for Disk storage, persist() for Memory
B. cache() defaults to MEMORY_ONLY, persist() allows custom storage levels
C. persist() is only for streaming DataFrames
D. cache() is deprecated in Spark 3.0

✅ Correct Answer: B
Explanation:

• cache() = persist(StorageLevel.MEMORY_ONLY)

• persist() allows custom levels like MEMORY_AND_DISK.

Q7: How do you read a CSV file in Spark?

A. spark.read.text("file.csv")
B. spark.read.csv("file.csv", header=True, inferSchema=True)
C. spark.load.csv("file.csv")
D. spark.csv.read("file.csv")

✅ Correct Answer: B
Explanation: The correct method is spark.read.csv() with optional parameters like header and inferSchema.

Q8: Which operation is a transformation in Spark?

A. count()
B. collect()
C. filter()
D. show()

✅ Correct Answer: C
Explanation:

• Transformations (lazy): filter(), select(), groupBy()

• Actions (eager): count(), collect(), show()

Q9: What does coalesce() do?

A. Merges small partitions into fewer partitions without shuffling
B. Increases the number of partitions with a full shuffle
C. Converts a DataFrame to an RDD
D. Drops NULL values from a DataFrame

✅ Correct Answer: A
Explanation: coalesce() reduces partitions without a full shuffle, unlike repartition().

Q10: How do you rename a column in a DataFrame?

A. df.withColumnRenamed("old", "new")
B. df.rename("old", "new")
C. df.select("old").alias("new")
D. df.changeColumnName("old", "new")

✅ Correct Answer: A
Explanation: The correct method is withColumnRenamed().

Section 3: Spark SQL & Optimizations

Q11: How do you register a DataFrame as a SQL table?

A. df.saveAsTable("table")
B. df.createOrReplaceTempView("table")
C. df.registerTable("table")
D. df.toSQL("table")

✅ Correct Answer: B
Explanation: createOrReplaceTempView() registers a temporary SQL view.

Q12: Which join is most efficient when joining large tables?

A. broadcast join
B. sort-merge join
C. cross join
D. full outer join

✅ Correct Answer: B
Explanation: Sort-merge join is Spark’s default for large tables, while broadcast join is best for small tables.

Q13: What is predicate pushdown?

A. Pushing filters to the storage layer to reduce I/O
B. Increasing partition size for faster reads
C. Converting SQL queries to RDDs
D. A type of shuffle operation

✅ Correct Answer: A
Explanation: Predicate pushdown applies filters early (e.g., in Parquet files) to minimize data read.

Section 4: Debugging & Performance Tuning

Q14: How do you check the execution plan of a DataFrame?

A. df.explain()
B. df.showPlan()
C. df.plan()
D. df.debug()

✅ Correct Answer: A
Explanation: explain() shows the logical and physical execution plans.

Q15: What does spark.sql.shuffle.partitions control?

A. Number of input partitions when reading data
B. Number of partitions after a shuffle operation
C. Size of cached DataFrames
D. Number of executors in the cluster

✅ Correct Answer: B
Explanation: This setting controls shuffle partition count (default: 200).

Section 5: Delta Lake (Bonus for Databricks)

Q16: What is Delta Lake?

A. A distributed file system
B. An optimized storage layer with ACID transactions
C. A streaming framework
D. A machine learning library

✅ Correct Answer: B
Explanation: Delta Lake provides ACID transactions, schema enforcement, and time travel.

Section 6: Advanced DataFrame Operations

Q17: You need to add a new column with a conditional value. Which method should you use?

A) df.withColumn("status", when(col("age") >= 18, "adult").otherwise("minor"))
B) df.addColumn("status", ifelse(col("age") >= 18, "adult", "minor"))
C) df.select(col("*"), expr("IF(age >= 18, 'adult', 'minor') AS status"))
D) df.updateColumn("status", col("age") >= 18 ? "adult" : "minor")

✅ Correct Answer: A
Explanation: when().otherwise() is Spark’s conditional expression. Option C (SQL-style) also works but is less idiomatic for DataFrame API.

Q18: How do you remove duplicates from a DataFrame?

A) df.distinct()
B) df.dropDuplicates()
C) df.unique()
D) Both A and B

✅ Correct Answer: D
Explanation: Both distinct() and dropDuplicates() work, but dropDuplicates(subset=["col1", "col2"]) allows targeting specific columns.

Section 7: Performance Tuning

Q19: A join operation is slow. What’s the FIRST optimization to try?

A) Increase spark.sql.shuffle.partitions
B) Use broadcast() if one table is small
C) Cache both DataFrames
D) Switch to RDDs

✅ Correct Answer: B
Explanation: Broadcasting the smaller table avoids shuffling (most impactful fix). Tuning shuffle partitions (A) helps but is secondary.

Q20: What does .repartition(10, col("country")) do?

A) Creates 10 partitions randomly
B) Repartitions into 10 partitions, hashing the “country” column
C) Coalesces to 10 partitions
D) Sorts the DataFrame by “country”

✅ Correct Answer: B
Explanation: This partitions data by hash of “country”, ensuring same-country records land in the same partition.

Section 8: Spark SQL Deep Dive

Q21: How do you calculate the average salary by department in SQL?

A) SELECT AVG(salary) FROM employees
B) SELECT department, AVG(salary) FROM employees GROUP BY department
C) SELECT department, MEAN(salary) FROM employees
D) SELECT department, salary.mean() FROM employees

✅ Correct Answer: B
Explanation: SQL uses AVG() for averages, and GROUP BY for aggregations.

Q22: What’s wrong with this query?

SELECT user_id, COUNT(*) 
FROM transactions 
WHERE COUNT(*) > 5 
GROUP BY user_id

A) Missing HAVING clause
B) WHERE can’t filter aggregates
C) Both A and B
D) Nothing – it’s correct

✅ Correct Answer: C
Explanation: Aggregate filters require HAVING, not WHERE. Correct version:

SELECT user_id, COUNT(*) 
FROM transactions 
GROUP BY user_id 
HAVING COUNT(*) > 5

Section 9: Debugging & Errors

Q23: You get “Out of Memory” errors. What’s the LEAST likely solution?

A) Increase spark.executor.memory
B) Use coalesce() to reduce partitions
C) Cache fewer datasets
D) Set spark.sql.shuffle.partitions=10

✅ Correct Answer: D
Explanation: Too few shuffle partitions (D) causes oversized partitions (worsens OOMs). Solutions A-C directly address memory.

Q24: A task fails with “FileNotFoundException”. What’s the most likely cause?

A) Driver ran out of memory
B) Input file was moved/deleted during job
C) Executor CPU overloaded
D) Shuffle partitions too small

✅ Correct Answer: B
Explanation: Spark reads file paths at runtime. If files disappear mid-job, this error occurs.

Section 10: Delta Lake (Databricks-Specific)

Q25: How do you time-travel to view Delta table data as of yesterday?

A) spark.read.delta("path", versionAsOf=1)
B) spark.read.format("delta").option("timestampAsOf", "2023-01-01").load("path")
C) spark.sql("SELECT * FROM delta.path VERSION AS OF 1")
D) Both B and C

✅ Correct Answer: D
Explanation: Delta supports time travel via timestamp or version number (both syntaxes work).

Section 11: Structured Streaming

Q26: You’re processing a live stream of IoT data. How do you handle late-arriving data?

A) Increase spark.streaming.blockInterval
B) Use withWatermark() and window functions
C) Set spark.sql.streaming.forceDeleteTempCheckpointLocation=true
D) Disable spark.streaming.backpressure.enabled

✅ Correct Answer: B
Explanation: Watermarking (withWatermark()) defines how late data can be while still being included in windowed aggregations. Example:

df.withWatermark("eventTime", "10 minutes").groupBy(window("eventTime", "5 minutes"))

Q27: What happens if you don’t specify a watermark in streaming aggregation?

A) Spark automatically sets a 1-minute watermark
B) State information grows indefinitely
C) The query fails immediately
D) Late data is always processed

✅ Correct Answer: B
Explanation: Without watermarking, Spark must maintain all state forever (memory risk). Watermark tells Spark when to discard old state.

Section 12: Advanced Optimizations

Q28: Your DataFrame join has skew (one key has 10M rows, others have 100). How do you fix it?

A) df.repartition(200)
B) spark.sql.adaptive.skewJoin.enabled=true (Spark 3.0+)
C) df.hint("skew", "join_key", [10000000])
D) Both B and C

✅ Correct Answer: D
Explanation: Spark 3.0+ can handle skew automatically (B), or you can manually salt keys (C). Example manual fix:

# Add random salt to skewed key
skewed_df = df.withColumn("salted_key", concat(col("join_key"), lit("_"), floor(rand()*10)))

Q29: When should you use .persist(StorageLevel.DISK_ONLY)?

A) When RAM is limited but SSD storage is fast
B) For iterative machine learning workflows
C) When recomputation is cheaper than disk I/O
D) Never – MEMORY_ONLY is always better

✅ Correct Answer: A
Explanation: DISK_ONLY is useful when cluster memory is exhausted but you have fast SSDs. Common in large shuffle stages.

Section 13: Spark SQL Internals

Q30: What does this query plan indicate?

== Physical Plan ==
*(2) Project [id#0L, (id#0L * 2) AS double_id#2L]
+- *(1) Filter (id#0L > 5)
   +- *(1) Scan csv [id#0L]

A) Projection happens before filtering
B) Filtering happens before projection
C) The query uses a broadcast join
D) The plan is invalid

✅ Correct Answer: B
Explanation: Spark optimizes by filtering (id > 5) first, then projecting (id*2). The *(1)/*(2) notation shows execution steps.

Section 14: Error Handling

Q31: Your job fails with “ExecutorLostFailure: Container killed by YARN for exceeding memory limits”. What’s the best fix?

A) Decrease spark.executor.memoryOverhead
B) Increase spark.executor.memory
C) Add more filter() operations early
D) All of the above

✅ Correct Answer: D
Explanation: This error means executors exceeded memory. Solutions:

• Increase memory allocation (B)

• Reduce overhead (A)

• Process less data via early filtering (C)

Section 15: Delta Lake Deep Dive

Q32: How do you compact small files in Delta Lake?

A) OPTIMIZE delta./path/to/table ZORDER BY timestamp
B) VACUUM delta./path/to/table RETAIN 168 HOURS
C) ALTER TABLE delta./path/to/table COMPACT SMALLFILES
D) spark.sql("REPAIR TABLE delta./path/to/table")

✅ Correct Answer: A
Explanation: OPTIMIZE combines small files and can co-locate data via ZORDER BY. VACUUM (B) only removes old files.

Section 16: PySpark Specifics

Q33: Why is this UDF slow compared to native Spark functions?

from pyspark.sql.functions import udf
slow_udf = udf(lambda x: x.upper())
df.withColumn("upper", slow_udf("name"))

A) UDFs can’t be vectorized
B) Data must be serialized to Python and back
C) No predicate pushdown
D) All of the above

✅ Correct Answer: D
Explanation: Python UDFs (vs. native Spark SQL functions) suffer from:

• Serialization overhead (B)

• No whole-stage code generation (A)

• Broken optimization (C)

Section 17: Cluster Configuration

Q34: Your 100-core cluster uses only 10 cores during execution. What’s likely misconfigured?

A) spark.executor.cores=10 with 10 executors
B) spark.default.parallelism=10
C) spark.dynamicAllocation.enabled=true
D) spark.sql.shuffle.partitions=200

✅ Correct Answer: B
Explanation: spark.default.parallelism controls initial partition count for RDDs. Too low = underutilization. For 100 cores, set to at least 200-300.

Section 18: Advanced Troubleshooting

Q35: Your Spark UI shows tasks stuck in “Scheduling” state. What’s the most likely cause?

A) Too many executors idle
B) Driver network latency
C) Insufficient cluster resources
D) DataFrame caching is enabled

✅ Correct Answer: C
Diagnosis: When tasks remain scheduled but not executing, the cluster lacks resources (CPU/memory). Check:

• spark.executor.instances vs. available cluster nodes

• YARN/Mesos/K8s resource queues

• spark.dynamicAllocation.enabled=true to auto-scale

Section 19: Query Execution Mysteries

Q36: Why does this query scan all files despite the filter?

df = spark.read.parquet("/data/year=*/month=*")
df.filter(col("year") == 2023).count()

A) Partition pruning isn’t supported with wildcards
B) The filter is applied after file scan
C) Parquet file metadata is corrupted
D) Need to use .where() instead of .filter()

✅ Correct Answer: A
Solution: Wildcards disable partition pruning. Instead use:

spark.read.parquet("/data/year=2023/month=*")

Section 20: Performance Paradoxes

Q37: Adding .cache() makes your job SLOWER. Why?

A) Cache overhead exceeds recomputation cost
B) Executors are over-provisioned
C) Disk persistence is enabled
D) Shuffle partitions are too small

✅ Correct Answer: A
Rule of Thumb: Only cache when:

• Data is reused multiple times

• Recomputing is expensive (complex transformations)

• Cluster has available memory

Section 21: API Gotchas

Q38: Why does this fail at runtime?

df.select("price").filter(col("price") > 100).show()

A) Column object vs. string mismatch
B) Missing parentheses
C) show() isn’t an action
D) Need to register temp view first

✅ Correct Answer: A
Fixed Version:

df.select(col("price")).filter(col("price") > 100).show()
# OR
df.select("price").filter("price > 100").show()

Section 22: Resource Tuning

Q39: Your executors keep crashing. Which config is most critical to adjust?

A) spark.executor.memoryOverhead
B) spark.sql.shuffle.partitions
C) spark.default.parallelism
D) spark.log.level

✅ Correct Answer: A
Memory Tuning Guide:

1. Set spark.executor.memory (heap)

2. Add 10-20% memoryOverhead (off-heap)

3. Monitor peak usage in Spark UI

Section 23: The Infamous OOM

Q40: Driver OOMs during collect(). What’s the safest alternative?

A) Increase driver memory
B) Use .take(1000) or .show()
C) Switch to RDD API
D) Disable DAG visualization

✅ Correct Answer: B
Golden Rule: Never collect() large datasets to driver. Prefer:

• .take(N) for samples

• Write to storage (df.write.parquet()) for full exports

Section 24: Schema Nightmares

Q41: Merging 1000 Parquet files fails with schema mismatch. How to resolve?

A) spark.sql.parquet.mergeSchema=true
B) Manually edit each file’s schema
C) Convert to CSV first
D) Use df.schema() to override

✅ Correct Answer: A
Alternative: spark.read.option(“mergeSchema”, “true”).parquet(“/path”)

Section 25: The Ultimate Optimization

Q42: Your 4-hour job must run in 1 hour. What’s the first thing to try?

A) Increase executor count and memory
B) Rewrite in Scala
C) Switch to Pandas UDFs
D) Pre-warm the cluster

✅ Correct Answer: A
Optimization Hierarchy:

1. Throw resources at it (easiest)

2. Fix data skew

3. Optimize shuffle partitions

4. Improve algorithms (last resort)

Bonus: 3 Pro Tips for the Exam

1. Memorize these key configs:

   spark.sql.shuffle.partitions  # Default: 200
   spark.executor.memory         # Default: 1G
   spark.dynamicAllocation.enabled # Default: false

2. Know the execution order:
   WHERE → GROUP BY → HAVING → SELECT → ORDER BY

3. When in doubt:

   • For performance: Check Spark UI

   • For errors: Search the 300+ MB of Spark logs

Final Challenge

Q43: What’s the output of this code?

spark.range(5).repartition(3).rdd.glom().collect()

A) [[0,1,2,3,4]]
B) [[0], [1], [2,3,4]]
C) Array of 3 sub-arrays with distributed values
D) Throws an error

✅ Correct Answer: C
Explanation:

• glom() shows partition contents

• Repartitioning splits the range unpredictably

• Example output: [[0,3], [1,4], [2]]