Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Questions & Answers

• Question 41:

  The code block shown below should write DataFrame transactionsDf as a parquet file to path storeDir, using brotli compression and replacing any previously existing file. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  transactionsDf.__1__.format("parquet").__2__(__3__).option(__4__, "brotli").__5__(storeDir)

  A. 1. save 2. mode 3. "ignore" 4. "compression" 5. path
  B. 1. store 2. with 3. "replacement" 4. "compression" 5. path
  C. 1. write 2. mode 3. "overwrite" 4. "compression" 5. save (Correct)
  D. 1. save 2. mode 3. "replace" 4. "compression" 5. path
  E. 1. write 2. mode 3. "overwrite" 4. compression 5. parquet
  C. 1. write 2. mode 3. "overwrite" 4. "compression" 5. save (Correct)

• Question 42:

  In which order should the code blocks shown below be run in order to create a DataFrame that shows the mean of column predError of DataFrame transactionsDf per column storeId and productId, where productId should be either 2 or 3 and the returned DataFrame should be sorted in ascending order by column storeId, leaving out any nulls in that column?

  DataFrame transactionsDf:

  1.+-------------+---------+-----+-------+---------+----+

  2.|transactionId|predError|value|storeId|productId| f|

  3.+-------------+---------+-----+-------+---------+----+

  4.| 1| 3| 4| 25| 1|null|

  5.| 2| 6| 7| 2| 2|null|

  6.| 3| 3| null| 25| 3|null|

  7.| 4| null| null| 3| 2|null|

  8.| 5| null| null| null| 2|null|

  9.| 6| 3| 2| 25| 2|null|

  10.+-------------+---------+-----+-------+---------+----+

  1.

  .mean("predError")

  2.

  .groupBy("storeId")

  3.

  .orderBy("storeId")

  4.

  transactionsDf.filter(transactionsDf.storeId.isNotNull())

  5.

  .pivot("productId", [2, 3])

  A. 4, 5, 2, 3, 1
  B. 4, 2, 1
  C. 4, 1, 5, 2, 3
  D. 4, 2, 5, 1, 3
  E. 4, 3, 2, 5, 1
  D. 4, 2, 5, 1, 3

  ---

  Correct code block:

  transactionsDf.filter(transactionsDf.storeId.isNotNull()).groupBy("storeId").pivot("productId", [2, 3]).mean("predError").orderBy("storeId")

  Output of correct code block:

  +-------+----+----+

  |storeId| 2| 3|

  +-------+----+----+

  | 2| 6.0|null|

  | 3|null|null|

  | 25| 3.0| 3.0|

  +-------+----+----+

  This is quite convoluted and requires you to think hard about the correct order of operations. The pivot method also makes an appearance - a method that you may not know all that much about (yet).

  At the first position in all answers is code block 4, so the is essentially just about the ordering of the remaining 4 code blocks. The states that the returned DataFrame should be sorted by column storeId. So, it should make sense to have

  code block 3 which includes the orderBy operator at the very end of the code

  block. This leaves you with only two answer options. Now, it is useful to know more about the context of pivot in PySpark. A common pattern is groupBy, pivot, and then another aggregating function, like mean. In the documentation linked

  below you can see that pivot is a method of pyspark.sql.GroupedData - meaning that before pivoting, you have to use groupBy. The only answer option matching this requirement is the one in which code block 2 (which includes groupBy) is

  stated before code block 5 (which includes pivot).

  More info: pyspark.sql.GroupedData.pivot -- PySpark 3.1.2 documentation

  Static notebook | Dynamic notebook: See test 3, 43 (Databricks import instructions)

• Question 43:

  Which of the following describes Spark actions?

  A. Writing data to disk is the primary purpose of actions.
  B. Actions are Spark's way of exchanging data between executors.
  C. The driver receives data upon request by actions.
  D. Stage boundaries are commonly established by actions.
  E. Actions are Spark's way of modifying RDDs.
  C. The driver receives data upon request by actions.

  ---

  The driver receives data upon request by actions.

  Correct! Actions trigger the distributed execution of tasks on executors which, upon task completion, transfer result data back to the driver. Actions are Spark's way of exchanging data between executors. No. In Spark, data is exchanged

  between executors via shuffles. Writing data to disk is the primary purpose of actions. No. The primary purpose of actions is to access data that is stored in Spark's RDDs and return the data, often in aggregated form, back to the driver.

  Actions are Spark's way of modifying RDDs.

  Incorrect. Firstly, RDDs are immutable ?they cannot be modified. Secondly, Spark generates new RDDs via transformations and not actions. Stage boundaries are commonly established by actions. Wrong. A stage boundary is commonly

  established by a shuffle, for example caused by a wide transformation.

• Question 44:

  Which of the following code blocks applies the boolean-returning Python function evaluateTestSuccess to column storeId of DataFrame transactionsDf as a user-defined function?

  A. 1.from pyspark.sql import types as T 2.evaluateTestSuccessUDF = udf(evaluateTestSuccess, T.BooleanType()) 3.transactionsDf.withColumn("result", evaluateTestSuccessUDF(col("storeId")))
  B. 1.evaluateTestSuccessUDF = udf(evaluateTestSuccess) 2.transactionsDf.withColumn("result", evaluateTestSuccessUDF(storeId))
  C. 1.from pyspark.sql import types as T 2.evaluateTestSuccessUDF = udf(evaluateTestSuccess, T.IntegerType()) 3.transactionsDf.withColumn("result", evaluateTestSuccess(col("storeId")))
  D. 1.evaluateTestSuccessUDF = udf(evaluateTestSuccess) 2.transactionsDf.withColumn("result", evaluateTestSuccessUDF(col("storeId")))
  E. 1.from pyspark.sql import types as T 2.evaluateTestSuccessUDF = udf(evaluateTestSuccess, T.BooleanType()) 3.transactionsDf.withColumn("result", evaluateTestSuccess(col("storeId")))
  A. 1.from pyspark.sql import types as T 2.evaluateTestSuccessUDF = udf(evaluateTestSuccess, T.BooleanType()) 3.transactionsDf.withColumn("result", evaluateTestSuccessUDF(col("storeId")))

• Question 45:

  Which of the following statements about stages is correct?

  A. Different stages in a job may be executed in parallel.
  B. Stages consist of one or more jobs.
  C. Stages ephemerally store transactions, before they are committed through actions.
  D. Tasks in a stage may be executed by multiple machines at the same time.
  E. Stages may contain multiple actions, narrow, and wide transformations.
  D. Tasks in a stage may be executed by multiple machines at the same time.

• Question 46:

  Which of the following code blocks returns a DataFrame where columns predError and productId are removed from DataFrame transactionsDf?

  Sample of DataFrame transactionsDf:

  1.+-------------+---------+-----+-------+---------+----+

  2.|transactionId|predError|value|storeId|productId|f |

  3.+-------------+---------+-----+-------+---------+----+

  4.|1 |3 |4 |25 |1 |null|

  5.|2 |6 |7 |2 |2 |null|

  6.|3 |3 |null |25 |3 |null|

  7.+-------------+---------+-----+-------+---------+----+

  A. transactionsDf.withColumnRemoved("predError", "productId")
  B. transactionsDf.drop(["predError", "productId", "associateId"])
  C. transactionsDf.drop("predError", "productId", "associateId")
  D. transactionsDf.dropColumns("predError", "productId", "associateId")
  E. transactionsDf.drop(col("predError", "productId"))
  D. transactionsDf.dropColumns("predError", "productId", "associateId")

  ---

  The key here is to understand that columns that are passed to DataFrame.drop() are ignored if they do not exist in the DataFrame. So, passing column name associateId to transactionsDf.drop()

  does not have any effect.

  Passing a list to transactionsDf.drop() is not valid. The documentation (link below) shows the call structure as DataFrame.drop(*cols). The * means that all arguments that are passed to DataFrame.drop() are read as columns. However, since

  a list of columns, for example ["predError", "productId", "associateId"] is not a column, Spark will run into an error. More info: pyspark.sql.DataFrame.drop -- PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1, 50

  (Databricks import instructions)

• Question 47:

  In which order should the code blocks shown below be run in order to return the number of records that are not empty in column value in the DataFrame resulting from an inner join of DataFrame transactionsDf and itemsDf on columns productId and itemId, respectively?

  1.

  .filter(~isnull(col('value')))

  2.

  .count()

  3.

  transactionsDf.join(itemsDf, col("transactionsDf.productId")==col("itemsDf.itemId"))

  4.

  transactionsDf.join(itemsDf, transactionsDf.productId==itemsDf.itemId, how='inner')

  5.

  .filter(col('value').isnotnull())

  6.

  .sum(col('value'))

  A. 4, 1, 2
  B. 3, 1, 6
  C. 3, 1, 2
  D. 3, 5, 2
  E. 4, 6
  A. 4, 1, 2

  ---

  Correct code block:

  transactionsDf.join(itemsDf, transactionsDf.productId==itemsDf.itemId, how='inner').filter(~isnull(col('value'))).count() Expressions col("transactionsDf.productId") and col("itemsDf.itemId") are invalid. col() does not accept the name of a

  DataFrame, only column names.

  Static notebook | Dynamic notebook: See test 2, 56 (Databricks import instructions)

• Question 48:

  The code block displayed below contains an error. The code block should return a copy of DataFrame transactionsDf where the name of column transactionId has been changed to transactionNumber. Find the error.

  Code block:

  transactionsDf.withColumn("transactionNumber", "transactionId")

  A. The arguments to the withColumn method need to be reordered.
  B. The arguments to the withColumn method need to be reordered and the copy() operator should be appended to the code block to ensure a copy is returned.
  C. The copy() operator should be appended to the code block to ensure a copy is returned.
  D. Each column name needs to be wrapped in the col() method and method withColumn should be replaced by method withColumnRenamed.
  E. The method withColumn should be replaced by method withColumnRenamed and the arguments to the method need to be reordered.
  E. The method withColumn should be replaced by method withColumnRenamed and the arguments to the method need to be reordered.

• Question 49:

  Which of the following statements about Spark's DataFrames is incorrect?

  A. Spark's DataFrames are immutable.
  B. Spark's DataFrames are equal to Python's DataFrames.
  C. Data in DataFrames is organized into named columns.
  D. RDDs are at the core of DataFrames.
  E. The data in DataFrames may be split into multiple chunks.
  B. Spark's DataFrames are equal to Python's DataFrames.

  ---

  Spark's DataFrames are equal to Python's or R's DataFrames. No, they are not equal. They are only similar. A major difference between Spark and Python is that Spark's DataFrames are distributed, whereby Python's are not.

• Question 50:

  Which of the following code blocks returns all unique values across all values in columns value and productId in DataFrame transactionsDf in a one-column DataFrame?

  A. tranactionsDf.select('value').join(transactionsDf.select('productId'), col('value')==col('productId'), 'outer')
  B. transactionsDf.select(col('value'), col('productId')).agg({'*': 'count'})
  C. transactionsDf.select('value', 'productId').distinct()
  D. transactionsDf.select('value').union(transactionsDf.select('productId')).distinct()
  E. transactionsDf.agg({'value': 'collect_set', 'productId': 'collect_set'})
  D. transactionsDf.select('value').union(transactionsDf.select('productId')).distinct()