Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Questions & Answers

• Question 21:

  Which of the following code blocks returns a single row from DataFrame transactionsDf?

  Full DataFrame transactionsDf:

  1.+-------------+---------+-----+-------+---------+----+

  2.|transactionId|predError|value|storeId|productId| f|

  3.+-------------+---------+-----+-------+---------+----+

  4.| 1| 3| 4| 25| 1|null|

  5.| 2| 6| 7| 2| 2|null|

  6.| 3| 3| null| 25| 3|null|

  7.| 4| null| null| 3| 2|null|

  8.| 5| null| null| null| 2|null|

  9.| 6| 3| 2| 25| 2|null|

  10.+-------------+---------+-----+-------+---------+----+

  A. transactionsDf.where(col("storeId").between(3,25))
  B. transactionsDf.filter((col("storeId")!=25) | (col("productId")==2))
  C. transactionsDf.filter(col("storeId")==25).select("predError","storeId").distinct()
  D. transactionsDf.select("productId", "storeId").where("storeId == 2 OR storeId != 25")
  E. transactionsDf.where(col("value").isNull()).select("productId", "storeId").distinct()
  C. transactionsDf.filter(col("storeId")==25).select("predError","storeId").distinct()

• Question 22:

  Which of the elements in the labeled panels represent the operation performed for broadcast variables?

  Larger image

  A. 2, 5
  B. 3
  C. 2, 3
  D. 1, 2
  E. 1, 3, 4
  C. 2, 3

  ---

  2,3

  Correct! Both panels 2 and 3 represent the operation performed for broadcast variables. While a broadcast operation may look like panel 3, with the driver being the bottleneck, it most probably

  looks like panel 2.

  This is because the torrent protocol sits behind Spark's broadcast implementation. In the torrent protocol, each executor will try to fetch missing broadcast variables from the driver or other nodes,

  preventing the driver from being the bottleneck.

  1,2

  Wrong. While panel 2 may represent broadcasting, panel 1 shows bi-directional communication which does not occur in broadcast operations. No. While broadcasting may materialize like shown in panel 3, its use of the torrent protocol also

  enables communciation as shown in panel 2 (see first explanation).

  1,3,4

  No. While panel 2 shows broadcasting, panel 1 shows bi-directional communication ?not a characteristic of broadcasting. Panel 4 shows uni-directional communication, but in the wrong direction.

  Panel 4 resembles more an accumulator variable than a broadcast variable.

  2,5

  Incorrect. While panel 2 shows broadcasting, panel 5 includes bi-directional communication ?not a characteristic of broadcasting.

  More info: Broadcast Join with Spark ?henning.kropponline.de

• Question 23:

  Which of the following code blocks silently writes DataFrame itemsDf in avro format to location fileLocation if a file does not yet exist at that location?

  A. itemsDf.write.avro(fileLocation)
  B. itemsDf.write.format("avro").mode("ignore").save(fileLocation)
  C. itemsDf.write.format("avro").mode("errorifexists").save(fileLocation)
  D. itemsDf.save.format("avro").mode("ignore").write(fileLocation)
  E. spark.DataFrameWriter(itemsDf).format("avro").write(fileLocation)
  A. itemsDf.write.avro(fileLocation)

• Question 24:

  Which of the following statements about executors is correct?

  A. Executors are launched by the driver.
  B. Executors stop upon application completion by default.
  C. Each node hosts a single executor.
  D. Executors store data in memory only.
  E. An executor can serve multiple applications.
  B. Executors stop upon application completion by default.

• Question 25:

  Which of the following code blocks applies the Python function to_limit on column predError in table transactionsDf, returning a DataFrame with columns transactionId and result?

  A. 1.spark.udf.register("LIMIT_FCN", to_limit) 2.spark.sql("SELECT transactionId, LIMIT_FCN(predError) AS result FROM transactionsDf")
  B. 1.spark.udf.register("LIMIT_FCN", to_limit) 2.spark.sql("SELECT transactionId, LIMIT_FCN(predError) FROM transactionsDf AS result")
  C. 1.spark.udf.register("LIMIT_FCN", to_limit) 2.spark.sql("SELECT transactionId, to_limit(predError) AS result FROM transactionsDf") spark.sql("SELECT transactionId, udf(to_limit(predError)) AS result FROM transactionsDf")
  D. 1.spark.udf.register(to_limit, "LIMIT_FCN") 2.spark.sql("SELECT transactionId, LIMIT_FCN(predError) AS result FROM transactionsDf")
  A. 1.spark.udf.register("LIMIT_FCN", to_limit) 2.spark.sql("SELECT transactionId, LIMIT_FCN(predError) AS result FROM transactionsDf")

  ---

  spark.udf.register("LIMIT_FCN", to_limit)

  spark.sql("SELECT transactionId, LIMIT_FCN(predError) AS result FROM transactionsDf") Correct! First, you have to register to_limit as UDF to use it in a sql statement. Then, you can use it under the LIMIT_FCN name, correctly naming the

  resulting column result.

  spark.udf.register(to_limit, "LIMIT_FCN")

  spark.sql("SELECT transactionId, LIMIT_FCN(predError) AS result FROM transactionsDf") No. In this answer, the arguments to spark.udf.register are flipped.

  spark.udf.register("LIMIT_FCN", to_limit)

  spark.sql("SELECT transactionId, to_limit(predError) AS result FROM transactionsDf") Wrong, this answer does not use the registered LIMIT_FCN in the sql statement, but tries to access the to_limit method directly. This will fail, since Spark

  cannot access it. spark.sql("SELECT transactionId, udf(to_limit(predError)) AS result FROM transactionsDf") Incorrect, there is no udf method in Spark's SQL.

  spark.udf.register("LIMIT_FCN", to_limit)

  spark.sql("SELECT transactionId, LIMIT_FCN(predError) FROM transactionsDf AS result") False. In this answer, the column that results from applying the UDF is not correctly renamed to result.

  Static notebook | Dynamic notebook: See test 3, 52 (Databricks import instructions)

• Question 26:

  Which of the following code blocks returns about 150 randomly selected rows from the 1000-row DataFrame transactionsDf, assuming that any row can appear more than once in the returned DataFrame?

  A. transactionsDf.resample(0.15, False, 3142)
  B. transactionsDf.sample(0.15, False, 3142)
  C. transactionsDf.sample(0.15)
  D. transactionsDf.sample(0.85, 8429)
  E. transactionsDf.sample(True, 0.15, 8261)
  E. transactionsDf.sample(True, 0.15, 8261)

  ---

  Answering this correctly depends on whether you understand the arguments to the DataFrame.sample() method (link to the documentation below). The arguments are as follows: DataFrame.sample(withReplacement=None, fraction=None, seed=None). The first argument withReplacement specified whether a row can be drawn from the DataFrame multiple times. By default, this option is disabled in Spark. But we have to enable it here, since the question asks for a row being able to appear more than once. So, we need to pass True for this argument. About replacement: "Replacement" is easiest explained with the example of removing random items from a box. When you remove those "with replacement" it means that after you have taken an item out of the box, you put it back inside. So, essentially, if you would randomly take 10 items out of a box with 100 items, there is a chance you take the same item twice or more times. "Without replacement" means that you would not put the item back into the box after removing it. So, every time you remove an item from the box, there is one less item in the box and you can never take the same item twice. The second argument to the withReplacement method is fraction. This referes to the fraction of items that should be returned. In the we are asked for 150 out of 1000 items ?a fraction of 0.15. The last argument is a random seed. A random seed makes a randomized processed repeatable. This means that if you would re-run the same sample() operation with the same random seed, you would get the same rows returned from the sample() command. There is no behavior around the random seed specified in the question. The varying random seeds are only there to confuse you!

  More info: pyspark.sql.DataFrame.sample -- PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1, 49 (Databricks import instructions)

• Question 27:

  The code block displayed below contains at least one error. The code block should return a DataFrame with only one column, result. That column should include all values in column value from DataFrame transactionsDf raised to the power of 5, and a null value for rows in which there is no value in column value. Find the error(s). Code block: 1.from pyspark.sql.functions import udf 2.from pyspark.sql import types as T

  3.

  4.transactionsDf.createOrReplaceTempView('transactions')

  5.

  6.def pow_5(x):

  7.

  return x**5

  8.

  9.spark.udf.register(pow_5, 'power_5_udf', T.LongType())

  10.spark.sql('SELECT power_5_udf(value) FROM transactions')

  A. The pow_5 method is unable to handle empty values in column value and the name of the column in the returned DataFrame is not result.
  B. The returned DataFrame includes multiple columns instead of just one column.
  C. The pow_5 method is unable to handle empty values in column value, the name of the column in the returned DataFrame is not result, and the SparkSession cannot access the transactionsDf DataFrame.
  D. The pow_5 method is unable to handle empty values in column value, the name of the column in the returned DataFrame is not result, and Spark driver does not call the UDF function appropriately.
  E. The pow_5 method is unable to handle empty values in column value, the UDF function is not registered properly with the Spark driver, and the name of the column in the returned DataFrame is not result.
  D. The pow_5 method is unable to handle empty values in column value, the name of the column in the returned DataFrame is not result, and Spark driver does not call the UDF function appropriately.

  ---

  Correct code block: from pyspark.sql.functions import udf from pyspark.sql import types as T transactionsDf.createOrReplaceTempView('transactions') def pow_5(x): if x: return x**5 return x spark.udf.register('power_5_udf', pow_5, T.LongType()) spark.sql('SELECT power_5_udf(value) AS result FROM transactions') Here it is important to understand how the pow_5 method handles empty values. In the wrong code block above, the pow_5 method is unable to handle empty values and will throw an error, since Python's ** operator cannot deal with any null value Spark passes into method pow_5. The order of arguments for registering the UDF function with Spark via spark.udf.register matters. In the code snippet in the question, the arguments for the SQL method name and the actual Python function are switched. You can read more about the arguments of spark.udf.register and see some examples of its usage in the documentation (link below). Finally, you should recognize that in the original code block, an expression to rename column created through the UDF function is missing. The renaming is done by SQL's AS result argument. Omitting that argument, you end up with the column name power_5_udf(value) and not result. More info: pyspark.sql.functions.udf -- PySpark 3.1.1 documentation

• Question 28:

  Which of the elements that are labeled with a circle and a number contain an error or are misrepresented?

  A. 1, 10
  B. 1, 8
  C. 10
  D. 7, 9, 10
  E. 1, 4, 6, 9
  B. 1, 8

• Question 29:

  Which of the following code blocks returns a DataFrame showing the mean value of column "value" of DataFrame transactionsDf, grouped by its column storeId?

  A. transactionsDf.groupBy(col(storeId).avg())
  B. transactionsDf.groupBy("storeId").avg(col("value"))
  C. transactionsDf.groupBy("storeId").agg(avg("value"))
  D. transactionsDf.groupBy("storeId").agg(average("value"))
  E. transactionsDf.groupBy("value").average()
  C. transactionsDf.groupBy("storeId").agg(avg("value"))

• Question 30:

  The code block displayed below contains an error. When the code block below has executed, it should have divided DataFrame transactionsDf into 14 parts, based on columns storeId and

  transactionDate (in this order). Find the error.

  Code block:

  transactionsDf.coalesce(14, ("storeId", "transactionDate"))

  A. The parentheses around the column names need to be removed and .select() needs to be appended to the code block.
  B. Operator coalesce needs to be replaced by repartition, the parentheses around the column names need to be removed, and .count() needs to be appended to the code block.
  C. Operator coalesce needs to be replaced by repartition, the parentheses around the column names need to be removed, and .select() needs to be appended to the code block.
  D. Operator coalesce needs to be replaced by repartition and the parentheses around the column names need to be replaced by square brackets.
  E. Operator coalesce needs to be replaced by repartition.
  B. Operator coalesce needs to be replaced by repartition, the parentheses around the column names need to be removed, and .count() needs to be appended to the code block.

  ---

  Correct code block:

  transactionsDf.repartition(14, "storeId", "transactionDate").count() Since we do not know how many partitions DataFrame transactionsDf has, we cannot safely use coalesce, since it would not make any change if the current number of

  partitions is smaller than 14.

  So, we need to use repartition.

  In the Spark documentation, the call structure for repartition is shown like this:

  DataFrame.repartition(numPartitions, *cols). The * operator means that any argument after numPartitions will be

  interpreted as column. Therefore, the brackets need to be removed. Finally, the specifies that after the execution the DataFrame should be divided. So, indirectly this is asking us to append an action to the code block. Since .select()

  is a transformation. the only possible choice here is .count(). More info: pyspark.sql.DataFrame.repartition -- PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1, 40 (Databricks import instructions)