Free DATABRICKS-CERTIFIED-DATA-ENGINEER-ASSOCIATE Web TestEngine Demo

Q40 Single choice

A data engineer is troubleshooting two different pipeline failures: Pipeline A fails with a java.lang.OutOfMemoryError is immediately thrown after the command display(df.collect()) is called on a 100GB dataset. Pipeline B fails during a wide transformation (a join of two large tables) with an ExecutorLostFailure error message, indicating executor memory exhaustion during shuffle.

Which action should the data engineer take to fix these two issues?

A
For Pipeline A, switch to a Storage Optimized cluster; for Pipeline B, use a Compute Optimized cluster.
B
For Pipeline A, increase the spark.driver.memory; for Pipeline B, increase the spark.sql.shuffle.partitions or executor memory.
C
For Pipeline A, increase the number of executors; for Pipeline B, increase the driver memory.
D
For both pipelines, enable Adaptive Query Execution (AQE) as it automatically manages memory allocation for both the driver and executors.

DATABRICKS-CERTIFIED-DATA-ENGINEER-ASSOCIATE Web TestEngine demo