Databricks Databricks Certification DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Questions & Answers

• Question 1:

  The code block shown below should return a one-column DataFrame where the column storeId is converted to string type. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  transactionsDf.__1__(__2__.__3__(__4__))

  A. 1. select

  2.

  col("storeId")

  3.

  cast

  4.

  StringType

  B. 1. select

  2.

  col("storeId")

  3.

  as

  4.

  StringType

  C. 1. cast

  2.

  "storeId"

  3.

  as

  4.

  StringType()

  D. 1. select

  2.

  col("storeId")

  3.

  cast

  4.

  StringType()

  E. 1. select

  2.

  storeId

  3.

  cast

  4.

  StringType()

  Correct Answer: D

  Correct code block: transactionsDf.select(col("storeId").cast(StringType())) Solving this involves understanding that, when using types from the pyspark.sql.types such as StringType, these types need to be instantiated when using them in Spark, or, in simple words, they need to be followed by parentheses like so: StringType(). You could also use .cast ("string") instead, but that option is not given here. More info: pyspark.sql.Column.cast -- PySpark 3.1.2 documentation

  Static notebook | Dynamic notebook: See test 2, 37 (Databricks import instructions)

• Question 2:

  Which of the following code blocks concatenates rows of DataFrames transactionsDf and transactionsNewDf, omitting any duplicates?

  A. transactionsDf.concat(transactionsNewDf).unique()

  B. transactionsDf.union(transactionsNewDf).distinct()

  C. spark.union(transactionsDf, transactionsNewDf).distinct()

  D. transactionsDf.join(transactionsNewDf, how="union").distinct()

  E. transactionsDf.union(transactionsNewDf).unique()

  Correct Answer: B

  DataFrame.unique() and DataFrame.concat() do not exist and union() is not a method of the SparkSession. In addition, there is no union option for the join method in the DataFrame.join() statement.

  More info: pyspark.sql.DataFrame.union -- PySpark 3.1.2 documentation

  Static notebook | Dynamic notebook: See test 2, 43 (Databricks import instructions)

• Question 3:

  The code block displayed below contains an error. The code block is intended to return all columns of DataFrame transactionsDf except for columns predError, productId, and value.

  Find the error.

  Excerpt of DataFrame transactionsDf:

  transactionsDf.select(~col("predError"), ~col("productId"), ~col("value"))

  A. The select operator should be replaced by the drop operator and the arguments to the drop operator should be column names predError, productId and value wrapped in the col operator so they should be expressed like drop(col(predError), col(productId), col(value)).

  B. The select operator should be replaced with the deselect operator.

  C. The column names in the select operator should not be strings and wrapped in the col operator, so they should be expressed like select(~col(predError), ~col(productId), ~col(value)).

  D. The select operator should be replaced by the drop operator.

  E. The select operator should be replaced by the drop operator and the arguments to the drop operator should be column names predError, productId and value as strings.

  Correct Answer: E

  Correct code block:

  transactionsDf.drop("predError", "productId", "value") Static notebook | Dynamic notebook: See test 1, 37

  (Databricks import instructions)

• Question 4:

  The code block displayed below contains an error. The code block should return DataFrame transactionsDf, but with the column storeId renamed to storeNumber. Find the error.

  Code block:

  transactionsDf.withColumn("storeNumber", "storeId")

  A. Instead of withColumn, the withColumnRenamed method should be used.

  B. Arguments "storeNumber" and "storeId" each need to be wrapped in a col() operator.

  C. Argument "storeId" should be the first and argument "storeNumber" should be the second argument to the withColumn method.

  D. The withColumn operator should be replaced with the copyDataFrame operator.

  E. Instead of withColumn, the withColumnRenamed method should be used and argument "storeId" should be the first and argument "storeNumber" should be the second argument to that method.

  Correct Answer: E

  Correct code block:

  transactionsDf.withColumnRenamed("storeId", "storeNumber") More info:

  pyspark.sql.DataFrame.withColumnRenamed -- PySpark 3.1.1 documentation Static notebook | Dynamic

  notebook: See test 1, 38 (Databricks import instructions)

• Question 5:

  The code block displayed below contains an error. The code block should use Python method find_most_freq_letter to find the letter present most in column itemName of DataFrame itemsDf and return it in a new column most_frequent_letter. Find the error.

  Code block:

  1.

  find_most_freq_letter_udf = udf(find_most_freq_letter)

  2.

  itemsDf.withColumn("most_frequent_letter", find_most_freq_letter("itemName"))

  A. Spark is not using the UDF method correctly.

  B. The UDF method is not registered correctly, since the return type is missing.

  C. The "itemName" expression should be wrapped in col().

  D. UDFs do not exist in PySpark.

  E. Spark is not adding a column.

  Correct Answer: A

  Correct code block: find_most_freq_letter_udf = udf(find_most_frequent_letter) itemsDf.withColumn("most_frequent_letter", find_most_freq_letter_udf("itemName")) Spark should use the previously registered find_most_freq_letter_udf method here ?but it is not doing that in the original codeblock. There, it just uses the non-UDF version of the Python method. Note that typically, we would have to specify a return type for udf(). Except in this case, since the default return type for udf() is a string which is what we are expecting here. If we wanted to return an integer variable instead, we would have to register the Python function as UDF using find_most_freq_letter_udf = udf(find_most_freq_letter, IntegerType()). More info: pyspark.sql.functions.udf -- PySpark 3.1.1 documentation

• Question 6:

  Which of the following describes characteristics of the Spark UI?

  A. Via the Spark UI, workloads can be manually distributed across executors.

  B. Via the Spark UI, stage execution speed can be modified.

  C. The Scheduler tab shows how jobs that are run in parallel by multiple users are distributed across the cluster.

  D. There is a place in the Spark UI that shows the property spark.executor.memory.

  E. Some of the tabs in the Spark UI are named Jobs, Stages, Storage, DAGs, Executors, and SQL.

  Correct Answer: D

• Question 7:

  Which of the following code blocks performs an inner join between DataFrame itemsDf and DataFrame transactionsDf, using columns itemId and transactionId as join keys, respectively?

  A. itemsDf.join(transactionsDf, "inner", itemsDf.itemId == transactionsDf.transactionId)

  B. itemsDf.join(transactionsDf, itemId == transactionId)

  C. itemsDf.join(transactionsDf, itemsDf.itemId == transactionsDf.transactionId, "inner")

  D. itemsDf.join(transactionsDf, "itemsDf.itemId == transactionsDf.transactionId", "inner")

  E. itemsDf.join(transactionsDf, col(itemsDf.itemId) == col(transactionsDf.transactionId))

  Correct Answer: C

  More info: pyspark.sql.DataFrame.join -- PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2, 27 (Databricks import instructions)

• Question 8:

  The code block shown below should return a DataFrame with all columns of DataFrame transactionsDf, but only maximum 2 rows in which column productId has at least the value 2. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  transactionsDf.__1__(__2__).__3__

  A. 1. where

  2.

  "productId" > 2

  3.

  max(2)

  B. 1. where

  2.

  transactionsDf[productId] >= 2

  3.

  limit(2)

  C. 1. filter

  2.

  productId > 2

  3.

  max(2)

  D. 1. filter

  2.

  col("productId") >= 2

  3.

  limit(2)

  E. 1. where

  2.

  productId >= 2

  3.

  limit(2)

  Correct Answer: D

  Correct code block:

  transactionsDf.filter(col("productId") >= 2).limit(2) The filter and where operators in gap 1 are just aliases of

  one another, so you cannot use them to pick the right answer.

  The column definition in gap 2 is more helpful. The DataFrame.filter() method takes an argument of type Column or str. From all possible answers, only the one including col("productId") >= 2 fits this profile, since it returns a Column type. The answer option using "productId" > 2 is invalid, since Spark does not understand that "productId" refers to column productId. The answer option using transactionsDf[productId] >= 2 is wrong because you cannot refer to a column using square bracket notation in Spark (if you are coming from Python using Pandas, this is something to watch out for). In all other options, productId is being referred to as a Python variable, so they are relatively easy to eliminate. Also note that the asks for the value in column productId being at least 2. This translates to a "greater or equal" sign (>= 2), but not a "greater" sign (> 2). Another thing worth noting is that there is no DataFrame.max() method. If you picked any option including this, you may be confusing it with the pyspark.sql.functions.max method. The correct method to limit the amount of rows is the DataFrame.limit() method.

  More info:

  -pyspark.sql.DataFrame.filter -- PySpark 3.1.2 documentation

  -pyspark.sql.DataFrame.limit -- PySpark 3.1.2 documentation

  Static notebook | Dynamic notebook: See test 3, 54 (Databricks import instructions)

• Question 9:

  In which order should the code blocks shown below be run in order to return the number of records that are not empty in column value in the DataFrame resulting from an inner join of DataFrame transactionsDf and itemsDf on columns productId and itemId, respectively?

  1.

  .filter(~isnull(col('value')))

  2.

  .count()

  3.

  transactionsDf.join(itemsDf, col("transactionsDf.productId")==col("itemsDf.itemId"))

  4.

  transactionsDf.join(itemsDf, transactionsDf.productId==itemsDf.itemId, how='inner')

  5.

  .filter(col('value').isnotnull())

  6.

  .sum(col('value'))

  A. 4, 1, 2

  B. 3, 1, 6

  C. 3, 1, 2

  D. 3, 5, 2

  E. 4, 6

  Correct Answer: A

  Correct code block:

  transactionsDf.join(itemsDf, transactionsDf.productId==itemsDf.itemId, how='inner').filter(~isnull(col

  ('value'))).count() Expressions col("transactionsDf.productId") and col("itemsDf.itemId") are invalid. col()

  does not accept the name of a DataFrame, only column names.

  Static notebook | Dynamic notebook: See test 2, 56 (Databricks import instructions)

• Question 10:

  The code block shown below should return a column that indicates through boolean variables whether rows in DataFrame transactionsDf have values greater or equal to 20 and smaller or equal to 30 in column storeId and have the value 2 in column productId. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  transactionsDf.__1__((__2__.__3__) __4__ (__5__))

  A. 1. select

  2.

  col("storeId")

  3.

  between(20, 30)

  4.

  and

  5.

  col("productId")==2

  B. 1. where

  2.

  col("storeId")

  3.

  geq(20).leq(30)

  4.

  and

  5.

  col("productId")==2

  C. 1. select

  2.

  "storeId"

  3.

  between(20, 30)

  4.

  andand

  5.

  col("productId")==2

  D. 1. select

  2.

  col("storeId")

  3.

  between(20, 30)

  4.

  andand

  5.

  col("productId")=2

  E. 1. select

  2.

  col("storeId")

  3.

  between(20, 30)

  4.

  and

  5.

  col("productId")==2

  Correct Answer: D

  Correct code block: transactionsDf.select((col("storeId").between(20, 30)) and (col("productId")==2)) Although this may make you think that it asks for a filter or where statement, it does not. It asks explicity to return a column with booleans ?this should point you to the select statement. Another trick here is the rarely used between() method. It exists and resolves to ((storeId >= 20) AND (storeId <= 30)) in SQL. geq() and leq() do not exist. Another riddle here is how to chain the two conditions. The only valid answer here is and. Operators like andand or and are not valid. Other boolean operators that would be valid in Spark are | and. Static notebook | Dynamic notebook: See test 1, 42 (Databricks import instructions)