Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Questions & Answers

• Question 71:

  Which of the following statements about garbage collection in Spark is incorrect?

  A. Garbage collection information can be accessed in the Spark UI's stage detail view.
  B. Optimizing garbage collection performance in Spark may limit caching ability.
  C. Manually persisting RDDs in Spark prevents them from being garbage collected.
  D. In Spark, using the G1 garbage collector is an alternative to using the default Parallel garbage collector.
  E. Serialized caching is a strategy to increase the performance of garbage collection.
  C. Manually persisting RDDs in Spark prevents them from being garbage collected.

• Question 72:

  Which of the following code blocks creates a new DataFrame with 3 columns, productId, highest, and lowest, that shows the biggest and smallest values of column value per value in column

  productId from DataFrame transactionsDf?

  Sample of DataFrame transactionsDf:

  1.+-------------+---------+-----+-------+---------+----+

  2.|transactionId|predError|value|storeId|productId| f|

  3.+-------------+---------+-----+-------+---------+----+

  4.| 1| 3| 4| 25| 1|null|

  5.| 2| 6| 7| 2| 2|null|

  6.| 3| 3| null| 25| 3|null|

  7.| 4| null| null| 3| 2|null|

  8.| 5| null| null| null| 2|null|

  9.| 6| 3| 2| 25| 2|null|

  10.+-------------+---------+-----+-------+---------+----+

  A. transactionsDf.max('value').min('value')
  B. transactionsDf.agg(max('value').alias('highest'), min('value').alias('lowest'))
  C. transactionsDf.groupby(col(productId)).agg(max(col(value)).alias("highest"), min(col(value)).alias("lowest"))
  D. transactionsDf.groupby('productId').agg(max('value').alias('highest'), min('value').alias('lowest'))
  E. transactionsDf.groupby("productId").agg({"highest": max("value"), "lowest": min("value")})
  D. transactionsDf.groupby('productId').agg(max('value').alias('highest'), min('value').alias('lowest'))

• Question 73:

  Which of the following is the deepest level in Spark's execution hierarchy?

  A. Job
  B. Task
  C. Executor
  D. Slot
  E. Stage
  B. Task

• Question 74:

  The code block shown below should return a DataFrame with columns transactionsId, predError, value, and f from DataFrame transactionsDf. Choose the answer that correctly fills the blanks in the code block to accomplish this. transactionsDf.__1__(__2__)

  A. 1. filter 2. "transactionId", "predError", "value", "f"
  B. 1. select 2. "transactionId, predError, value, f"
  C. 1. select 2. ["transactionId", "predError", "value", "f"]
  D. 1. where 2. col("transactionId"), col("predError"), col("value"), col("f") E. 1. select 2. col(["transactionId", "predError", "value", "f"])
  C. 1. select 2. ["transactionId", "predError", "value", "f"]

• Question 75:

  Which of the following statements about storage levels is incorrect?

  A. The cache operator on DataFrames is evaluated like a transformation.
  B. In client mode, DataFrames cached with the MEMORY_ONLY_2 level will not be stored in the edge node's memory.
  C. Caching can be undone using the DataFrame.unpersist() operator.
  D. MEMORY_AND_DISK replicates cached DataFrames both on memory and disk.
  E. DISK_ONLY will not use the worker node's memory.
  D. MEMORY_AND_DISK replicates cached DataFrames both on memory and disk.

  ---

  MEMORY_AND_DISK replicates cached DataFrames both on memory and disk. Correct, this statement is wrong. Spark prioritizes storage in memory, and will only store data on disk that does not fit into memory.

  DISK_ONLY will not use the worker node's memory.

  Wrong, this statement is correct. DISK_ONLY keeps data only on the worker node's disk, but not in memory.

  In client mode, DataFrames cached with the MEMORY_ONLY_2 level will not be stored in the edge node's memory.

  Wrong, this statement is correct. In fact, Spark does not have a provision to cache DataFrames in the driver (which sits on the edge node in client mode). Spark caches DataFrames in the executors' memory.

  Caching can be undone using the DataFrame.unpersist() operator. Wrong, this statement is correct. Caching, as achieved via the DataFrame.cache() or DataFrame.persist() operators can be undone using the DataFrame.unpersist() operator.

  This operator will

  remove all of its parts from the executors' memory and disk. The cache operator on DataFrames is evaluated like a transformation. Wrong, this statement is correct. DataFrame.cache() is evaluated like a transformation:

  Through lazy evaluation. This means that after calling DataFrame.cache() the command will not have any

  effect until you call a subsequent action, like DataFrame.cache().count(). More info: pyspark.sql.DataFrame.unpersist -- PySpark 3.1.2 documentation

• Question 76:

  The code block displayed below contains an error. The code block should use Python method find_most_freq_letter to find the letter present most in column itemName of DataFrame itemsDf and return it in a new column most_frequent_letter. Find the error.

  Code block:

  1.

  find_most_freq_letter_udf = udf(find_most_freq_letter)

  2.

  itemsDf.withColumn("most_frequent_letter", find_most_freq_letter("itemName"))

  A. Spark is not using the UDF method correctly.
  B. The UDF method is not registered correctly, since the return type is missing.
  C. The "itemName" expression should be wrapped in col().
  D. UDFs do not exist in PySpark.
  E. Spark is not adding a column.
  A. Spark is not using the UDF method correctly.

  ---

  Correct code block: find_most_freq_letter_udf = udf(find_most_frequent_letter) itemsDf.withColumn("most_frequent_letter", find_most_freq_letter_udf("itemName")) Spark should use the previously registered find_most_freq_letter_udf method here ?but it is not doing that in the original codeblock. There, it just uses the non-UDF version of the Python method. Note that typically, we would have to specify a return type for udf(). Except in this case, since the default return type for udf() is a string which is what we are expecting here. If we wanted to return an integer variable instead, we would have to register the Python function as UDF using find_most_freq_letter_udf = udf(find_most_freq_letter, IntegerType()). More info: pyspark.sql.functions.udf -- PySpark 3.1.1 documentation

• Question 77:

  Which of the following code blocks returns a single-column DataFrame showing the number of words in column supplier of DataFrame itemsDf?

  Sample of DataFrame itemsDf:

  1.+------+-----------------------------+-------------------+

  2.|itemId|attributes |supplier |

  3.+------+-----------------------------+-------------------+

  4.|1 |[blue, winter, cozy] |Sports Company Inc.|

  5.|2 |[red, summer, fresh, cooling]|YetiX |

  6.|3 |[green, summer, travel] |Sports Company Inc.| 7.+------+-----------------------------+-------------------+

  A. itemsDf.split("supplier", " ").count()
  B. itemsDf.split("supplier", " ").size()
  C. itemsDf.select(word_count("supplier"))
  D. spark.select(size(split(col(supplier), " ")))
  E. itemsDf.select(size(split("supplier", " ")))
  E. itemsDf.select(size(split("supplier", " ")))

  ---

  Output of correct code block:

  +----------------------------+

  |size(split(supplier, , -1))|

  +----------------------------+

  | 3|

  | 1|

  | 3|

  +----------------------------+

  This shows a typical use case for the split command: Splitting a string into words. An additional difficulty is that you are asked to count the words. Although it is tempting to use the

  count method here, the size method (as in: size of an array) is actually the correct one to use. Familiarize yourself with the split and the size methods using the linked documentation below.

  More info:

  Split method: pyspark.sql.functions.split -- PySpark 3.1.2 documentation Size method: pyspark.sql.functions.size -- PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2, 29 (Databricks import instructions)

• Question 78:

  Which of the following code blocks returns only rows from DataFrame transactionsDf in which values in column productId are unique?

  A. transactionsDf.distinct("productId")
  B. transactionsDf.dropDuplicates(subset=["productId"])
  C. transactionsDf.drop_duplicates(subset="productId")
  D. transactionsDf.unique("productId")
  E. transactionsDf.dropDuplicates(subset="productId")
  B. transactionsDf.dropDuplicates(subset=["productId"])

  ---

  Although the suggests using a method called unique() here, that method does not actually exist in PySpark. In PySpark, it is called distinct(). But then, this method is not the right one to use here, since with distinct() we could filter out unique values in a specific column. However, we want to return the entire rows here. So the trick is to use dropDuplicates with the subset keyword parameter. In the documentation for dropDuplicates, the examples show that subset should be used with a list. And this is exactly the key to solving this question: The productId column needs to be fed into the subset argument in a list, even though it is just a single column. More info: pyspark.sql.DataFrame.dropDuplicates -- PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1, 45 (Databricks import instructions)

• Question 79:

  Which of the following describes a narrow transformation?

  A. narrow transformation is an operation in which data is exchanged across partitions.
  B. A narrow transformation is a process in which data from multiple RDDs is used.
  C. A narrow transformation is a process in which 32-bit float variables are cast to smaller float variables, like 16-bit or 8-bit float variables.
  D. A narrow transformation is an operation in which data is exchanged across the cluster.
  E. A narrow transformation is an operation in which no data is exchanged across the cluster.
  E. A narrow transformation is an operation in which no data is exchanged across the cluster.

  ---

  A narrow transformation is an operation in which no data is exchanged across the cluster. Correct! In narrow transformations, no data is exchanged across the cluster, since these transformations do not require any data from outside of the

  partition they are applied on.

  Typical narrow

  transformations include filter, drop, and coalesce. A narrow transformation is an operation in which data is exchanged across partitions. No, that would be one definition of a wide transformation, but not of a narrow transformation. Wide

  transformations typically cause a shuffle, in which data is exchanged across partitions, executors,

  and the cluster.

  A narrow transformation is an operation in which data is exchanged across the cluster.

  No, see explanation just above this one.

  A narrow transformation is a process in which 32-bit float variables are cast to smaller float variables, like 16-bit or 8-bit float variables.

  No, type conversion has nothing to do with narrow transformations in Spark. A narrow transformation is a process in which data from multiple RDDs is used. No. A resilient distributed dataset (RDD) can be described as a collection of

  partitions. In a narrow transformation, no data is exchanged between partitions. Thus, no data is exchanged between

  RDDs.

  One could say though that a narrow transformation and, in fact, any transformation results in a new RDD being created. This is because a transformation results in a change to an existing RDD

  (RDDs are the foundation of other Spark data structures, like DataFrames). But, since RDDs are immutable, a new RDD needs to be created to reflect the change caused by the transformation.

  More info: Spark Transformation and Action: A Deep Dive | by Misbah Uddin | CodeX | Medium

• Question 80:

  The code block shown below should return a new 2-column DataFrame that shows one attribute from column attributes per row next to the associated itemName, for all suppliers in column supplier whose name includes Sports. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  Sample of DataFrame itemsDf:

  1.+------+----------------------------------+-----------------------------+-------------------+

  2.|itemId|itemName |attributes |supplier |

  3.+------+----------------------------------+-----------------------------+-------------------+

  4.|1 |Thick Coat for Walking in the Snow|[blue, winter, cozy] |Sports Company Inc.| 5.|2 |Elegant Outdoors Summer Dress |[red, summer, fresh, cooling]|YetiX |

  6.|3 |Outdoors Backpack |[green, summer, travel] |Sports Company Inc.|

  7.+------+----------------------------------+-----------------------------+-------------------+

  Code block:

  itemsDf.__1__(__2__).select(__3__, __4__)

  A. 1. filter 2. col("supplier").isin("Sports") 3. "itemName" 4. explode(col("attributes"))
  B. 1. where 2. col("supplier").contains("Sports") 3. "itemName" 4. "attributes"
  C. 1. where 2. col(supplier).contains("Sports") 3. explode(attributes) 4. itemName
  D. 1. where 2. "Sports".isin(col("Supplier")) 3. "itemName" 4. array_explode("attributes")
  E. 1. filter 2. col("supplier").contains("Sports") 3. "itemName" 4. explode("attributes")
  E. 1. filter 2. col("supplier").contains("Sports") 3. "itemName" 4. explode("attributes")

  ---

  Output of correct code block:

  +----------------------------------+------+

  |itemName |col |

  +----------------------------------+------+

  |Thick Coat for Walking in the Snow|blue |

  |Thick Coat for Walking in the Snow|winter|

  |Thick Coat for Walking in the Snow|cozy |

  |Outdoors Backpack |green |

  |Outdoors Backpack |summer|

  |Outdoors Backpack |travel|

  +----------------------------------+------+

  The key to solving this is knowing about Spark's explode operator. Using this operator, you can extract values from arrays into single rows. The following guidance steps through

  the

  answers systematically from the first to the last gap. Note that there are many ways to solving the gap questions and filtering out wrong answers, you do not always have to start filtering out from the

  first gap, but can also exclude some answers based on obvious problems you see with them.

  The answers to the first gap present you with two options: filter and where. These two are actually synonyms in PySpark, so using either of those is fine. The answer options to this gap therefore do

  not help us in selecting the right answer.

  The second gap is more interesting. One answer option includes "Sports".isin(col("Supplier")). This construct does not work, since Python's string does not have an isin method. Another option

  contains col(supplier). Here, Python will try to interpret supplier as a variable. We have not set this variable, so this is not a viable answer. Then, you are left with answers options that include col

  ("supplier").contains("Sports") and col("supplier").isin("Sports"). The states that we are looking for suppliers whose name includes Sports, so we have to go for the contains operator

  here.

  We would use the isin operator if we wanted to filter out for supplier names that match any entries in a list of supplier names.

  Finally, we are left with two answers that fill the third gap both with "itemName" and the fourth gap either with explode("attributes") or "attributes". While both are correct Spark syntax, only explode

  ("attributes") will help us achieve our goal. Specifically, the asks for one attribute from column attributes per row - this is what the explode() operator does. One answer option also includes array_explode() which is not a valid operator in

  PySpark.

  More info: pyspark.sql.functions.explode -- PySpark 3.1.2 documentation

  Static notebook | Dynamic notebook: See test 3, 39 (Databricks import instructions)