Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Questions & Answers

• Question 91:

  Which of the following describes a shuffle?

  A. A shuffle is a process that is executed during a broadcast hash join.
  B. A shuffle is a process that compares data across executors.
  C. A shuffle is a process that compares data across partitions.
  D. A shuffle is a Spark operation that results from DataFrame.coalesce().
  E. A shuffle is a process that allocates partitions to executors.
  C. A shuffle is a process that compares data across partitions.

  ---

  A shuffle is a Spark operation that results from DataFrame.coalesce(). No. DataFrame.coalesce() does not result in a shuffle. A shuffle is a process that allocates partitions to executors.

  This is incorrect.

  A shuffle is a process that is executed during a broadcast hash join. No, broadcast hash joins avoid shuffles and yield performance benefits if at least one of the two tables is small in size (<= 10 MB by default). Broadcast hash joins can avoid

  shuffles because instead of exchanging partitions between executors, they broadcast a small table to all executors that then perform the rest of the join operation locally. A shuffle is a process that compares data across executors. No, in a

  shuffle, data is compared across partitions, and not executors. More info: Spark Repartition and Coalesce - Explained (https://bit.ly/32KF7zS)

• Question 92:

  Which of the following describes the role of tasks in the Spark execution hierarchy?

  A. Tasks are the smallest element in the execution hierarchy.
  B. Within one task, the slots are the unit of work done for each partition of the data.
  C. Tasks are the second-smallest element in the execution hierarchy.
  D. Stages with narrow dependencies can be grouped into one task.
  E. Tasks with wide dependencies can be grouped into one stage.
  A. Tasks are the smallest element in the execution hierarchy.

  ---

  Stages with narrow dependencies can be grouped into one task. Wrong, tasks with narrow dependencies can be grouped into one stage. Tasks with wide dependencies can be grouped into one stage. Wrong, since a wide transformation causes a shuffle which always marks the boundary of a stage. So, you cannot bundle multiple tasks that have wide dependencies into a stage. Tasks are the second-smallest element in the execution hierarchy. No, they are the smallest element in the execution hierarchy. Within one task, the slots are the unit of work done for each partition of the data. No, tasks are the unit of work done per partition. Slots help Spark parallelize work. An executor can have multiple slots which enable it to process multiple tasks in parallel.

• Question 93:

  The code block shown below should return a DataFrame with two columns, itemId and col. In this DataFrame, for each element in column attributes of DataFrame itemDf there should be a separate

  row in which the column itemId contains the associated itemId from DataFrame itemsDf. The new DataFrame should only contain rows for rows in DataFrame itemsDf in which the column attributes

  contains the element cozy.

  A sample of DataFrame itemsDf is below.

  Code block:

  itemsDf.__1__(__2__).__3__(__4__, __5__(__6__))

  A. 1. filter 2. array_contains("cozy") 3. select 4. "itemId" 5. explode 6. "attributes"
  B. 1. where 2. "array_contains(attributes, 'cozy')" 3. select 4. itemId 5. explode 6. attributes
  C. 1. filter 2. "array_contains(attributes, 'cozy')" 3. select 4. "itemId" 5. map 6. "attributes"
  D. 1. filter 2. "array_contains(attributes, cozy)" 3. select 4. "itemId" 5. explode 6. "attributes"
  E. 1. filter 2. "array_contains(attributes, 'cozy')" 3. select 4. "itemId" 5. explode 6. "attributes"
  E. 1. filter 2. "array_contains(attributes, 'cozy')" 3. select 4. "itemId" 5. explode 6. "attributes"

  ---

  The correct code block is:

  itemsDf.filter("array_contains(attributes, 'cozy')").select("itemId", explode("attributes")) The key here is understanding how to use array_contains(). You can either use it as an expression in a string, or you can import it from

  pyspark.sql.functions. In that case, the following would also

  work:

  itemsDf.filter(array_contains("attributes", "cozy")).select("itemId", explode("attributes")) Static notebook | Dynamic notebook: See test 1, 29 (Databricks import instructions) (https://flrs.github.io/spark_practice_tests_code/#1/29.html , https://

  bit.ly/sparkpracticeexams_import_instructions)

• Question 94:

  The code block shown below should return a column that indicates through boolean variables whether rows in DataFrame transactionsDf have values greater or equal to 20 and smaller or equal to 30 in column storeId and have the value 2 in column productId. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  transactionsDf.__1__((__2__.__3__) __4__ (__5__))

  A. 1. select 2. col("storeId") 3. between(20, 30) 4. and 5. col("productId")==2
  B. 1. where 2. col("storeId") 3. geq(20).leq(30) 4. and 5. col("productId")==2
  C. 1. select 2. "storeId" 3. between(20, 30) 4. andand 5. col("productId")==2
  D. 1. select 2. col("storeId") 3. between(20, 30) 4. andand 5. col("productId")=2
  E. 1. select 2. col("storeId") 3. between(20, 30) 4. and 5. col("productId")==2
  D. 1. select 2. col("storeId") 3. between(20, 30) 4. andand 5. col("productId")=2

  ---

  Correct code block:

  transactionsDf.select((col("storeId").between(20, 30)) and (col("productId")==2)) Although this may make you think that it asks for a filter or where statement, it does not. It asks explicity to return a column with booleans ?this should point you to

  the select

  statement.

  Another trick here is the rarely used between() method. It exists and resolves to ((storeId >= 20) AND (storeId <= 30)) in SQL. geq() and leq() do not exist. Another riddle here is how to chain the two conditions. The only valid answer here is and.

  Operators like andand or and are not valid. Other boolean operators that would be valid in Spark are | and.

  Static notebook | Dynamic notebook: See test 1, 42 (Databricks import instructions)

• Question 95:

  Which of the following statements about data skew is incorrect?

  A. Spark will not automatically optimize skew joins by default.
  B. Broadcast joins are a viable way to increase join performance for skewed data over sort- merge joins.
  C. In skewed DataFrames, the largest and the smallest partition consume very different amounts of memory.
  D. To mitigate skew, Spark automatically disregards null values in keys when joining.
  E. Salting can resolve data skew.
  D. To mitigate skew, Spark automatically disregards null values in keys when joining.

  ---

  To mitigate skew, Spark automatically disregards null values in keys when joining. This statement is incorrect, and thus the correct answer to the question. Joining keys that contain null values is of particular concern with regard to data skew. In real-world applications, a table may contain a great number of records that do not have a value assigned to the column used as a join key. During the join, the data is at risk of being heavily skewed. This is because all records with a null-value join key are then evaluated as a single large partition, standing in stark contrast to the potentially diverse key values (and therefore small partitions) of the non-null-key records. Spark specifically does not handle this automatically. However, there are several strategies to mitigate this problem like discarding null values temporarily, only to merge them back later (see last link below). In skewed DataFrames, the largest and the smallest partition consume very different amounts of memory. This statement is correct. In fact, having very different partition sizes is the very definition of skew. Skew can degrade Spark performance because the largest partition occupies a single executor for a long time. This blocks a Spark job and is an inefficient use of resources, since other executors that processed smaller partitions need to idle until the large partition is processed. Salting can resolve data skew. This statement is correct. The purpose of salting is to provide Spark with an opportunity to repartition data into partitions of similar size, based on a salted partitioning key. A salted partitioning key typically is a column that consists of uniformly distributed random numbers. The number of unique entries in the partitioning key column should match the number of your desired number of partitions. After repartitioning by the salted key, all partitions should have roughly the same size. Spark does not automatically optimize skew joins by default. This statement is correct. Automatic skew join optimization is a feature of Adaptive Query Execution (AQE). By default, AQE is disabled in Spark. To enable it, Spark's spark.sql.adaptive.enabled configuration option needs to be set to true instead of leaving it at the default false. To automatically optimize skew joins, Spark's spark.sql.adaptive.skewJoin.enabled options also needs to be set to true, which it is by default. When skew join optimization is enabled, Spark recognizes skew joins and optimizes them by splitting the bigger partitions into smaller partitions which leads to performance increases. Broadcast joins are a viable way to increase join performance for skewed data over sort- merge joins. This statement is correct. Broadcast joins can indeed help increase join performance for skewed data, under some conditions. One of the DataFrames to be joined needs to be small enough to fit into each executor's memory, along a partition from the other DataFrame. If this is the case, a broadcast join increases join performance over a sort-merge join. The reason is that a sort-merge join with skewed data involves excessive shuffling. During shuffling, data is sent around the cluster, ultimately slowing down the Spark application. For skewed data, the amount of data, and thus the slowdown, is particularly big. Broadcast joins, however, help reduce shuffling data. The smaller table is directly stored on all executors, eliminating a great amount of network traffic, ultimately increasing join performance relative to the sort-merge join. It is worth noting that for optimizing skew join behavior it may make sense to manually adjust Spark's spark.sql.autoBroadcastJoinThreshold configuration property if the smaller DataFrame is bigger than the 10 MB set by default. More info:

  -Performance Tuning - Spark 3.0.0 Documentation

  -Data Skew and Garbage Collection to Improve Spark Performance

  -Section 1.2 - Joins on Skewed Data ?GitBook

• Question 96:

  The code block displayed below contains an error. The code block should combine data from DataFrames itemsDf and transactionsDf, showing all rows of DataFrame itemsDf that have a matching value in column itemId with a value in

  column transactionsId of DataFrame transactionsDf.

  Find the error.

  Code block:

  itemsDf.join(itemsDf.itemId==transactionsDf.transactionId)

  A. The join statement is incomplete.
  B. The union method should be used instead of join.
  C. The join method is inappropriate.
  D. The merge method should be used instead of join.
  E. The join expression is malformed.
  A. The join statement is incomplete.

• Question 97:

  Which of the following is the idea behind dynamic partition pruning in Spark?

  A. Dynamic partition pruning is intended to skip over the data you do not need in the results of a query.
  B. Dynamic partition pruning concatenates columns of similar data types to optimize join performance.
  C. Dynamic partition pruning performs wide transformations on disk instead of in memory.
  D. Dynamic partition pruning reoptimizes physical plans based on data types and broadcast variables.
  E. Dynamic partition pruning reoptimizes query plans based on runtime statistics collected during query execution.
  A. Dynamic partition pruning is intended to skip over the data you do not need in the results of a query.

• Question 98:

  Which of the following describes Spark's standalone deployment mode?

  A. Standalone mode uses a single JVM to run Spark driver and executor processes.
  B. Standalone mode means that the cluster does not contain the driver.
  C. Standalone mode is how Spark runs on YARN and Mesos clusters.
  D. Standalone mode uses only a single executor per worker per application.
  E. Standalone mode is a viable solution for clusters that run multiple frameworks, not only Spark.
  D. Standalone mode uses only a single executor per worker per application.

  ---

  Standalone mode uses only a single executor per worker per application. This is correct and a limitation of Spark's standalone mode. Standalone mode is a viable solution for clusters that run multiple frameworks. Incorrect. A limitation of

  standalone mode is that Apache Spark must be the only framework running on the cluster. If you would want to run multiple frameworks on the same cluster in parallel, for example Apache Spark and Apache Flink, you would consider the

  YARN deployment mode.

  Standalone mode uses a single JVM to run Spark driver and executor processes.

  No, this is what local mode does.

  Standalone mode is how Spark runs on YARN and Mesos clusters. No. YARN and Mesos modes are two deployment modes that are different from standalone mode. These modes allow Spark to run alongside other frameworks on a cluster.

  When Spark is run in standalone mode, only the Spark framework can run on the cluster. Standalone mode means that the cluster does not contain the driver. Incorrect, the cluster does not contain the driver in client mode, but in standalone

  mode the driver runs on a node in the cluster.

  More info: Learning Spark, 2nd Edition, Chapter 1

• Question 99:

  Which of the following describes the role of the cluster manager?

  A. The cluster manager schedules tasks on the cluster in client mode.
  B. The cluster manager schedules tasks on the cluster in local mode.
  C. The cluster manager allocates resources to Spark applications and maintains the executor processes in client mode.
  D. The cluster manager allocates resources to Spark applications and maintains the executor processes in remote mode.
  E. The cluster manager allocates resources to the DataFrame manager.
  C. The cluster manager allocates resources to Spark applications and maintains the executor processes in client mode.

  ---

  The cluster manager allocates resources to Spark applications and maintains the executor processes in client mode.

  Correct. In cluster mode, the cluster manager is located on a node other than the client machine. From there it starts and ends executor processes on the cluster nodes as required by the Spark

  application running on the Spark driver.

  The cluster manager allocates resources to Spark applications and maintains the executor processes in remote mode.

  Wrong, there is no "remote" execution mode in Spark. Available execution modes are local, client, and cluster.

  The cluster manager allocates resources to the DataFrame manager Wrong, there is no "DataFrame manager" in Spark.

  The cluster manager schedules tasks on the cluster in client mode. No, in client mode, the Spark driver schedules tasks on the cluster ?not the cluster manager.

  The cluster manager schedules tasks on the cluster in local mode. Wrong: In local mode, there is no "cluster". The Spark application is running on a single machine, not on a cluster of machines.

• Question 100:

  The code block shown below should return the number of columns in the CSV file stored at location filePath. From the CSV file, only lines should be read that do not start with a # character. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  Code block:

  __1__(__2__.__3__.csv(filePath, __4__).__5__)

  A. 1. size 2. spark 3. read() 4. escape='#' 5. columns
  B. 1. DataFrame 2. spark 3. read() 4. escape='#' 5. shape[0]
  C. 1. len 2. pyspark 3. DataFrameReader 4. comment='#' 5. columns
  D. 1. size 2. pyspark 3. DataFrameReader 4. comment='#' 5. columns
  E. 1. len 2. spark 3. read 4. comment='#' 5. columns
  E. 1. len 2. spark 3. read 4. comment='#' 5. columns