Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Questions & Answers

• Question 161:

  Which of the following describes characteristics of the Dataset API?

  A. The Dataset API does not support unstructured data.
  B. In Python, the Dataset API mainly resembles Pandas' DataFrame API.
  C. In Python, the Dataset API's schema is constructed via type hints.
  D. The Dataset API is available in Scala, but it is not available in Python.
  E. The Dataset API does not provide compile-time type safety.
  D. The Dataset API is available in Scala, but it is not available in Python.

• Question 162:

  In which order should the code blocks shown below be run in order to create a table of all values in column attributes next to the respective values in column supplier in DataFrame itemsDf?

  1.

  itemsDf.createOrReplaceView("itemsDf")

  2.

  spark.sql("FROM itemsDf SELECT 'supplier', explode('Attributes')")

  3.

  spark.sql("FROM itemsDf SELECT supplier, explode(attributes)")

  4.

  itemsDf.createOrReplaceTempView("itemsDf")

  A. 4, 3
  B. 1, 3
  C. 2
  D. 4, 2
  E. 1, 2
  A. 4, 3

  ---

  Static notebook | Dynamic notebook: See test 1, 56 (Databricks import instructions)

• Question 163:

  Which of the following code blocks reduces a DataFrame from 12 to 6 partitions and performs a full shuffle?

  A. DataFrame.repartition(12)
  B. DataFrame.coalesce(6).shuffle()
  C. DataFrame.coalesce(6)
  D. DataFrame.coalesce(6, shuffle=True)
  E. DataFrame.repartition(6)
  E. DataFrame.repartition(6)

• Question 164:

  The code block shown below should return a copy of DataFrame transactionsDf without columns value and productId and with an additional column associateId that has the value 5. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  transactionsDf.__1__(__2__, __3__).__4__(__5__, 'value')

  A. 1. withColumn 2. 'associateId' 3. 5 4. remove 5. 'productId'
  B. 1. withNewColumn 2. associateId 3. lit(5) 4. drop 5. productId
  C. 1. withColumn 2. 'associateId' 3. lit(5) 4. drop 5. 'productId'
  D. 1. withColumnRenamed 2. 'associateId' 3. 5 4. drop 5. 'productId'
  E. 1. withColumn 2. col(associateId) 3. lit(5) 4. drop 5. col(productId)
  C. 1. withColumn 2. 'associateId' 3. lit(5) 4. drop 5. 'productId'

  ---

  Correct code block:

  transactionsDf.withColumn('associateId', lit(5)).drop('productId', 'value') For solving this it is important that you know the lit() function (link to documentation below). This function enables you to add a column of a constant value to a

  DataFrame.

  More info: pyspark.sql.functions.lit -- PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1, 57 (Databricks import instructions)

• Question 165:

  The code block shown below should store DataFrame transactionsDf on two different executors, utilizing the executors' memory as much as possible, but not writing anything to disk. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  1.from pyspark import StorageLevel 2.transactionsDf.__1__(StorageLevel.__2__).__3__

  A. 1. cache 2. MEMORY_ONLY_2 3. count()
  B. 1. persist 2. DISK_ONLY_2 3. count()
  C. 1. persist 2. MEMORY_ONLY_2 3. select()
  D. 1. cache 2. DISK_ONLY_2 3. count()
  E. 1. persist 2. MEMORY_ONLY_2 3. count()
  E. 1. persist 2. MEMORY_ONLY_2 3. count()

• Question 166:

  Which of the following code blocks performs a join in which the small DataFrame transactionsDf is sent to all executors where it is joined with DataFrame itemsDf on columns storeId and itemId, respectively?

  A. itemsDf.join(transactionsDf, itemsDf.itemId == transactionsDf.storeId, "right_outer")
  B. itemsDf.join(transactionsDf, itemsDf.itemId == transactionsDf.storeId, "broadcast")
  C. itemsDf.merge(transactionsDf, "itemsDf.itemId == transactionsDf.storeId", "broadcast")
  D. itemsDf.join(broadcast(transactionsDf), itemsDf.itemId == transactionsDf.storeId)
  E. itemsDf.join(transactionsDf, broadcast(itemsDf.itemId == transactionsDf.storeId))
  D. itemsDf.join(broadcast(transactionsDf), itemsDf.itemId == transactionsDf.storeId)

• Question 167:

  Which of the following code blocks returns a new DataFrame with the same columns as DataFrame transactionsDf, except for columns predError and value which should be removed?

  A. transactionsDf.drop(["predError", "value"])
  B. transactionsDf.drop("predError", "value")
  C. transactionsDf.drop(col("predError"), col("value"))
  D. transactionsDf.drop(predError, value)
  E. transactionsDf.drop("predError and value")
  B. transactionsDf.drop("predError", "value")

• Question 168:

  The code block displayed below contains an error. The code block should configure Spark so that DataFrames up to a size of 20 MB will be broadcast to all worker nodes when performing a join.

  Find the error.

  Code block:

  A. spark.conf.set("spark.sql.autoBroadcastJoinThreshold", 20)
  B. Spark will only broadcast DataFrames that are much smaller than the default value.
  C. The correct option to write configurations is through spark.config and not spark.conf.
  D. Spark will only apply the limit to threshold joins and not to other joins.
  E. The passed limit has the wrong variable type.
  F. The command is evaluated lazily and needs to be followed by an action.
  B. Spark will only broadcast DataFrames that are much smaller than the default value.

  ---

  This is is hard. Let's assess the different answers one-by-one. Spark will only broadcast DataFrames that are much smaller than the default value. This is correct. The default value is 10 MB (10485760 bytes). Since the configuration for

  spark.sql.autoBroadcastJoinThreshold expects a number in bytes (and not megabytes), the code block sets

  the limits to merely 20 bytes, instead of the requested 20 * 1024 * 1024 (= 20971520) bytes.

  The command is evaluated lazily and needs to be followed by an action.

  No, this command is evaluated right away!

  Spark will only apply the limit to threshold joins and not to other joins. There are no "threshold joins", so this option does not make any sense. The correct option to write configurations is through spark.config and not spark.conf.

  No, it is indeed spark.conf!

  The passed limit has the wrong variable type.

  The configuration expects the number of bytes, a number, as an input. So, the 20 provided in the code block is fine.

• Question 169:

  Which of the following code blocks returns a new DataFrame in which column attributes of DataFrame itemsDf is renamed to feature0 and column supplier to feature1?

  A. itemsDf.withColumnRenamed(attributes, feature0).withColumnRenamed(supplier, feature1)
  B. 1.itemsDf.withColumnRenamed("attributes", "feature0") 2.itemsDf.withColumnRenamed("supplier", "feature1")
  C. itemsDf.withColumnRenamed(col("attributes"), col("feature0"), col("supplier"), col("feature1"))
  D. itemsDf.withColumnRenamed("attributes", "feature0").withColumnRenamed("supplier", "feature1")
  E. itemsDf.withColumn("attributes", "feature0").withColumn("supplier", "feature1")
  D. itemsDf.withColumnRenamed("attributes", "feature0").withColumnRenamed("supplier", "feature1")

• Question 170:

  Which of the following describes characteristics of the Spark driver?

  A. The Spark driver requests the transformation of operations into DAG computations from the worker nodes.
  B. If set in the Spark configuration, Spark scales the Spark driver horizontally to improve parallel processing performance.
  C. The Spark driver processes partitions in an optimized, distributed fashion.
  D. In a non-interactive Spark application, the Spark driver automatically creates the SparkSession object.
  E. The Spark driver's responsibility includes scheduling queries for execution on worker nodes.
  D. In a non-interactive Spark application, the Spark driver automatically creates the SparkSession object.

  ---

  The Spark driver requests the transformation of operations into DAG computations from the worker nodes.

  No, the Spark driver transforms operations into DAG computations itself. If set in the Spark configuration, Spark scales the Spark driver horizontally to improve parallel processing performance.

  No. There is always a single driver per application, but one or more executors. The Spark driver processes partitions in an optimized, distributed fashion.

  No, this is what executors do.

  In a non-interactive Spark application, the Spark driver automatically creates the SparkSession object.

  Wrong. In a non-interactive Spark application, you need to create the SparkSession object. In an interactive Spark shell, the Spark driver instantiates the object for you.