Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Questions & Answers

• Question 141:

  Which of the following code blocks creates a new DataFrame with two columns season and wind_speed_ms where column season is of data type string and column wind_speed_ms is of data type double?

  A. spark.DataFrame({"season": ["winter","summer"], "wind_speed_ms": [4.5, 7.5]})
  B. spark.createDataFrame([("summer", 4.5), ("winter", 7.5)], ["season", "wind_speed_ms"])
  C. 1. from pyspark.sql import types as T 2. spark.createDataFrame((("summer", 4.5), ("winter", 7.5)),
  D. StructType([T.StructField("season", T.CharType()), T.StructField("season",
  E. DoubleType())]))
  F. spark.newDataFrame([("summer", 4.5), ("winter", 7.5)], ["season", "wind_speed_ms"])
  G. spark.createDataFrame({"season": ["winter","summer"], "wind_speed_ms": [4.5, 7.5]})
  B. spark.createDataFrame([("summer", 4.5), ("winter", 7.5)], ["season", "wind_speed_ms"])

• Question 142:

  Which of the following code blocks creates a new 6-column DataFrame by appending the rows of the 6-column DataFrame yesterdayTransactionsDf to the rows of the 6-column DataFrame todayTransactionsDf, ignoring that both DataFrames have different column names?

  A. union(todayTransactionsDf, yesterdayTransactionsDf)
  B. todayTransactionsDf.unionByName(yesterdayTransactionsDf, allowMissingColumns=True)
  C. todayTransactionsDf.unionByName(yesterdayTransactionsDf)
  D. todayTransactionsDf.concat(yesterdayTransactionsDf)
  E. todayTransactionsDf.union(yesterdayTransactionsDf)
  E. todayTransactionsDf.union(yesterdayTransactionsDf)

  ---

  todayTransactionsDf.union(yesterdayTransactionsDf) Correct. The union command appends rows of yesterdayTransactionsDf to the rows of todayTransactionsDf, ignoring that both DataFrames have different column names. The resulting DataFrame will have the column names of DataFrame todayTransactionsDf. todayTransactionsDf.unionByName(yesterdayTransactionsDf) No. unionByName specifically tries to match columns in the two DataFrames by name and only appends values in columns with identical names across the two DataFrames. In the form presented above, the command is a great fit for joining DataFrames that have exactly the same columns, but in a different order. In this case though, the command will fail because the two DataFrames have different columns. todayTransactionsDf.unionByName(yesterdayTransactionsDf, allowMissingColumns=True) No. The unionByName command is described in the previous explanation. However, with the allowMissingColumns argument set to True, it is no longer an issue that the two DataFrames have different column names. Any columns that do not have a match in the other DataFrame will be filled with null where there is no value. In the case at hand, the resulting DataFrame will have 7 or more columns though, so it this command is not the right answer. union(todayTransactionsDf, yesterdayTransactionsDf) No, there is no union method in pyspark.sql.functions. todayTransactionsDf.concat(yesterdayTransactionsDf) Wrong, the DataFrame class does not have a concat method. More info: pyspark.sql.DataFrame.union -- PySpark 3.1.2 documentation, pyspark.sql.DataFrame.unionByName -- PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 3, 18 (Databricks import instructions)

• Question 143:

  The code block displayed below contains an error. The code block is intended to return all columns of DataFrame transactionsDf except for columns predError, productId, and value.

  Find the error.

  Excerpt of DataFrame transactionsDf:

  transactionsDf.select(~col("predError"), ~col("productId"), ~col("value"))

  A. The select operator should be replaced by the drop operator and the arguments to the drop operator should be column names predError, productId and value wrapped in the col operator so they should be expressed like drop(col(predError), col(productId), col(value)).
  B. The select operator should be replaced with the deselect operator.
  C. The column names in the select operator should not be strings and wrapped in the col operator, so they should be expressed like select(~col(predError), ~col(productId), ~col(value)).
  D. The select operator should be replaced by the drop operator.
  E. The select operator should be replaced by the drop operator and the arguments to the drop operator should be column names predError, productId and value as strings.
  E. The select operator should be replaced by the drop operator and the arguments to the drop operator should be column names predError, productId and value as strings.

  ---

  Correct code block:

  transactionsDf.drop("predError", "productId", "value") Static notebook | Dynamic notebook: See test 1, 37 (Databricks import instructions)

• Question 144:

  The code block displayed below contains multiple errors. The code block should return a DataFrame that contains only columns transactionId, predError, value and storeId of DataFrame

  transactionsDf. Find the errors.

  Code block:

  transactionsDf.select([col(productId), col(f)])

  Sample of transactionsDf:

  1.+-------------+---------+-----+-------+---------+----+

  2.|transactionId|predError|value|storeId|productId| f| 3.+-------------+---------+-----+-------+---------+----+

  4.| 1| 3| 4| 25| 1|null|

  5.| 2| 6| 7| 2| 2|null|

  6.| 3| 3| null| 25| 3|null|

  7.+-------------+---------+-----+-------+---------+----+

  A. The column names should be listed directly as arguments to the operator and not as a list.
  B. The select operator should be replaced by a drop operator, the column names should be listed directly as arguments to the operator and not as a list, and all column names should be expressed as strings without being wrapped in a col() operator.
  C. The select operator should be replaced by a drop operator.
  D. The column names should be listed directly as arguments to the operator and not as a list and following the pattern of how column names are expressed in the code block, columns productId and f should be replaced by transactionId, predError, value and storeId.
  E. The select operator should be replaced by a drop operator, the column names should be listed directly as arguments to the operator and not as a list, and all col() operators should be removed.
  B. The select operator should be replaced by a drop operator, the column names should be listed directly as arguments to the operator and not as a list, and all column names should be expressed as strings without being wrapped in a col() operator.

• Question 145:

  Which of the following code blocks returns a copy of DataFrame transactionsDf where the column storeId has been converted to string type?

  A. transactionsDf.withColumn("storeId", convert("storeId", "string"))
  B. transactionsDf.withColumn("storeId", col("storeId", "string"))
  C. transactionsDf.withColumn("storeId", col("storeId").convert("string"))
  D. transactionsDf.withColumn("storeId", col("storeId").cast("string"))
  E. transactionsDf.withColumn("storeId", convert("storeId").as("string"))
  D. transactionsDf.withColumn("storeId", col("storeId").cast("string"))

• Question 146:

  The code block displayed below contains an error. The code block should return DataFrame transactionsDf, but with the column storeId renamed to storeNumber. Find the error.

  Code block:

  transactionsDf.withColumn("storeNumber", "storeId")

  A. Instead of withColumn, the withColumnRenamed method should be used.
  B. Arguments "storeNumber" and "storeId" each need to be wrapped in a col() operator.
  C. Argument "storeId" should be the first and argument "storeNumber" should be the second argument to the withColumn method.
  D. The withColumn operator should be replaced with the copyDataFrame operator.
  E. Instead of withColumn, the withColumnRenamed method should be used and argument "storeId" should be the first and argument "storeNumber" should be the second argument to that method.
  E. Instead of withColumn, the withColumnRenamed method should be used and argument "storeId" should be the first and argument "storeNumber" should be the second argument to that method.

  ---

  Correct code block:

  transactionsDf.withColumnRenamed("storeId", "storeNumber") More info: pyspark.sql.DataFrame.withColumnRenamed -- PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1, 38 (Databricks import instructions)

• Question 147:

  The code block displayed below contains multiple errors. The code block should remove column transactionDate from DataFrame transactionsDf and add a column transactionTimestamp in which

  dates that are expressed as strings in column transactionDate of DataFrame transactionsDf are converted into unix timestamps. Find the errors.

  Sample of DataFrame transactionsDf:

  1.+-------------+---------+-----+-------+---------+----+----------------+

  2.|transactionId|predError|value|storeId|productId| f| transactionDate|

  3.+-------------+---------+-----+-------+---------+----+----------------+

  4.| 1| 3| 4| 25| 1|null|2020-04-26 15:35|

  5.| 2| 6| 7| 2| 2|null|2020-04-13 22:01|

  6.| 3| 3| null| 25| 3|null|2020-04-02 10:53|

  7.+-------------+---------+-----+-------+---------+----+----------------+

  Code block:

  1.transactionsDf = transactionsDf.drop("transactionDate")

  2.transactionsDf["transactionTimestamp"] = unix_timestamp("transactionDate", "yyyy-MM- dd")

  A. Column transactionDate should be dropped after transactionTimestamp has been written. The string indicating the date format should be adjusted. The withColumn operator should be used instead of the existing column assignment. Operator to_unixtime() should be used instead of unix_timestamp().
  B. Column transactionDate should be dropped after transactionTimestamp has been written. The withColumn operator should be used instead of the existing column assignment. Column transactionDate should be wrapped in a col() operator.
  C. Column transactionDate should be wrapped in a col() operator.
  D. The string indicating the date format should be adjusted. The withColumnReplaced operator should be used instead of the drop and assign pattern in the code block to replace column transactionDate with the new column transactionTimestamp.
  E. Column transactionDate should be dropped after transactionTimestamp has been written. The string indicating the date format should be adjusted. The withColumn operator should be used instead of the existing column assignment.
  E. Column transactionDate should be dropped after transactionTimestamp has been written. The string indicating the date format should be adjusted. The withColumn operator should be used instead of the existing column assignment.

  ---

  This requires a lot of thinking to get right. For solving it, you may take advantage of the digital notepad that is provided to you during the test. You have probably seen that the code block

  includes multiple errors. In the test, you are usually confronted with a code block that only contains a single error. However, since you are practicing here, this challenging multi-error will

  make it easier for you to deal with single-error questions in the real exam. You can clearly see that column transactionDate should be dropped only after transactionTimestamp has been written. This is because to generate column

  transactionTimestamp, Spark needs to

  read the values from column transactionDate.

  Values in column transactionDate in the original transactionsDf DataFrame look like 2020- 04-26 15:35. So, to convert those correctly, you would have to pass yyyy-MM-dd HH:mm.

  In other words:

  The string indicating the date format should be adjusted. While you might be tempted to change unix_timestamp() to to_unixtime() (in line with the from_unixtime() operator), this function does not exist in Spark. unix_timestamp() is the correct

  operator to use here.

  Also, there is no DataFrame.withColumnReplaced() operator. A similar operator that exists is DataFrame.withColumnRenamed().

  Whether you use col() or not is irrelevant with unix_timestamp() - the command is fine with both.

  Finally, you cannot assign a column like transactionsDf["columnName"] = ... in Spark. This is Pandas syntax (Pandas is a popular Python package for data analysis), but it is not supported in Spark.

  So, you need to use Spark's DataFrame.withColumn() syntax instead. More info: pyspark.sql.functions.unix_timestamp -- PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 3, 28 (Databricks import instructions)

• Question 148:

  Which of the following statements about reducing out-of-memory errors is incorrect?

  A. Concatenating multiple string columns into a single column may guard against out-of- memory errors.
  B. Reducing partition size can help against out-of-memory errors.
  C. Limiting the amount of data being automatically broadcast in joins can help against out- of-memory errors.
  D. Setting a limit on the maximum size of serialized data returned to the driver may help prevent out-of-memory errors.
  E. Decreasing the number of cores available to each executor can help against out-of- memory errors.
  A. Concatenating multiple string columns into a single column may guard against out-of- memory errors.

  ---

  Concatenating multiple string columns into a single column may guard against out-of- memory errors. Exactly, this is an incorrect answer! Concatenating any string columns does not reduce the size of the data, it just structures it a different way. This does little to how Spark processes the data and definitely does not reduce out-of-memory errors. Reducing partition size can help against out-of-memory errors. No, this is not incorrect. Reducing partition size is a viable way to aid against out-of- memory errors, since executors need to load partitions into memory before processing them. If the executor does not have enough memory available to do that, it will throw an out-of-memory error. Decreasing partition size can therefore be very helpful for preventing that. Decreasing the number of cores available to each executor can help against out-ofmemory errors. No, this is not incorrect. To process a partition, this partition needs to be loaded into the memory of an executor. If you imagine that every core in every executor processes a partition, potentially in parallel with other executors, you can imagine that memory on the machine hosting the executors fills up quite quickly. So, memory usage of executors is a concern, especially when multiple partitions are processed at the same time. To strike a balance between performance and memory usage, decreasing the number of cores may help against out-of-memory errors. Setting a limit on the maximum size of serialized data returned to the driver may help prevent out-of-memory errors. No, this is not incorrect. When using commands like collect() that trigger the transmission of potentially large amounts of data from the cluster to the driver, the driver may experience out-of-memory errors. One strategy to avoid this is to be careful about using commands like collect() that send back large amounts of data to the driver. Another strategy is setting the parameter spark.driver.maxResultSize. If data to be transmitted to the driver exceeds the threshold specified by the parameter, Spark will abort the job and therefore prevent an out-of- memory error. Limiting the amount of data being automatically broadcast in joins can help against out-of- memory errors. Wrong, this is not incorrect. As part of Spark's internal optimization, Spark may choose to speed up operations by broadcasting (usually relatively small) tables to executors. This broadcast is happening from the driver, so all the broadcast tables are loaded into the driver first. If these tables are relatively big, or multiple mid-size tables are being broadcast, this may lead to an out-ofmemory error. The maximum table size for which Spark will consider broadcasting is set by the spark.sql.autoBroadcastJoinThreshold parameter. More info: Configuration - Spark 3.1.2 Documentation and Spark OOM Error -- Closeup. Does the following look familiar when... | by Amit Singh Rathore | The Startup | Medium

• Question 149:

  Which of the following code blocks shuffles DataFrame transactionsDf, which has 8 partitions, so that it has 10 partitions?

  A. transactionsDf.repartition(transactionsDf.getNumPartitions()+2)
  B. transactionsDf.repartition(transactionsDf.rdd.getNumPartitions()+2)
  C. transactionsDf.coalesce(10)
  D. transactionsDf.coalesce(transactionsDf.getNumPartitions()+2)
  E. transactionsDf.repartition(transactionsDf._partitions+2)
  B. transactionsDf.repartition(transactionsDf.rdd.getNumPartitions()+2)

• Question 150:

  Which of the following DataFrame operators is never classified as a wide transformation?

  A. DataFrame.sort()
  B. DataFrame.aggregate()
  C. DataFrame.repartition()
  D. DataFrame.select()
  E. DataFrame.join()
  D. DataFrame.select()

  ---

  As a general rule: After having gone through the practice tests you probably have a good feeling for what classifies as a wide and what classifies as a narrow transformation. If you are unsure, feel free to play around in Spark and display the

  explanation of the Spark execution plan via DataFrame.[operation, for example sort()].explain(). If repartitioning is involved, it would count as a wide transformation.

  DataFrame.select()

  Correct! A wide transformation includes a shuffle, meaning that an input partition maps to one or more output partitions. This is expensive and causes traffic across the cluster. With the select() operation however, you pass commands to

  Spark that tell Spark to perform an operation on a specific slice of any partition. For this, Spark does not need to exchange data across partitions, each partition can be worked on independently. Thus, you do not cause a wide transformation.

  DataFrame.repartition()

  Incorrect. When you repartition a DataFrame, you redefine partition boundaries. Data will flow across your cluster and end up in different partitions after the repartitioning is completed. This is known as a shuffle and, in turn, is classified as a

  wide transformation.

  DataFrame.aggregate()

  No. When you aggregate, you may compare and summarize data across partitions. In the process, data are exchanged across the cluster, and newly formed output partitions depend on one or more input partitions. This is a typical

  characteristic of a shuffle, meaning that the aggregate operation may classify as a wide transformation.

  DataFrame.join()

  Wrong. Joining multiple DataFrames usually means that large amounts of data are exchanged across the cluster, as new partitions are formed. This is a shuffle and therefore DataFrame.join() counts as a wide transformation.

  DataFrame.sort()

  False. When sorting, Spark needs to compare many rows across all partitions to each other. This is an expensive operation, since data is exchanged across the cluster and new partitions are formed as data is reordered. This process

  classifies as a shuffle and, as a result, DataFrame.sort() counts as wide transformation.

  More info: Understanding Apache Spark Shuffle | Philipp Brunenberg