Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Questions & Answers

• Question 11:

  Which of the following describes Spark's Adaptive Query Execution?

  A. Adaptive Query Execution features include dynamically coalescing shuffle partitions, dynamically injecting scan filters, and dynamically optimizing skew joins.
  B. Adaptive Query Execution is enabled in Spark by default.
  C. Adaptive Query Execution reoptimizes queries at execution points.
  D. Adaptive Query Execution features are dynamically switching join strategies and dynamically optimizing skew joins.
  E. Adaptive Query Execution applies to all kinds of queries.
  D. Adaptive Query Execution features are dynamically switching join strategies and dynamically optimizing skew joins.

  ---

  Adaptive Query Execution features include dynamically coalescing shuffle partitions, dynamically injecting scan filters, and dynamically optimizing skew joins. This is almost correct. All of these features, except for dynamically injecting scan filters, are part of Adaptive Query Execution. Dynamically injecting scan filters for join operations to limit the amount of data to be considered in a query is part of Dynamic Partition Pruning and not of Adaptive Query Execution. Adaptive Query Execution reoptimizes queries at execution points. No, Adaptive Query Execution reoptimizes queries at materialization points. Adaptive Query Execution is enabled in Spark by default. No, Adaptive Query Execution is disabled in Spark needs to be enabled through the spark.sql.adaptive.enabled property. Adaptive Query Execution applies to all kinds of queries. No, Adaptive Query Execution applies only to queries that are not streaming queries and that contain at least one exchange (typically expressed through a join, aggregate, or window operator) or one subquery. More info: How to Speed up SQL Queries with Adaptive Query Execution, Learning Spark, 2nd Edition, Chapter 12 (https://bit.ly/3tOh8M1)

• Question 12:

  Which of the following code blocks writes DataFrame itemsDf to disk at storage location filePath, making sure to substitute any existing data at that location?

  A. itemsDf.write.mode("overwrite").parquet(filePath)
  B. itemsDf.write.option("parquet").mode("overwrite").path(filePath)
  C. itemsDf.write(filePath, mode="overwrite")
  D. itemsDf.write.mode("overwrite").path(filePath)
  E. itemsDf.write().parquet(filePath, mode="overwrite")
  A. itemsDf.write.mode("overwrite").parquet(filePath)

• Question 13:

  Which of the following statements about Spark's configuration properties is incorrect?

  A. The maximum number of tasks that an executor can process at the same time is controlled by the spark.task.cpus property.
  B. The maximum number of tasks that an executor can process at the same time is controlled by the spark.executor.cores property.
  C. The default value for spark.sql.autoBroadcastJoinThreshold is 10MB.
  D. The default number of partitions to use when shuffling data for joins or aggregations is 300.
  E. The default number of partitions returned from certain transformations can be controlled by the spark.default.parallelism property.
  D. The default number of partitions to use when shuffling data for joins or aggregations is 300.

• Question 14:

  Which of the following code blocks returns a DataFrame that has all columns of DataFrame transactionsDf and an additional column predErrorSquared which is the squared value of column predError in DataFrame transactionsDf?

  A. transactionsDf.withColumn("predError", pow(col("predErrorSquared"), 2))
  B. transactionsDf.withColumnRenamed("predErrorSquared", pow(predError, 2))
  C. transactionsDf.withColumn("predErrorSquared", pow(col("predError"), lit(2)))
  D. transactionsDf.withColumn("predErrorSquared", pow(predError, lit(2)))
  E. transactionsDf.withColumn("predErrorSquared", "predError"**2)
  C. transactionsDf.withColumn("predErrorSquared", pow(col("predError"), lit(2)))

• Question 15:

  Which of the following describes a way for resizing a DataFrame from 16 to 8 partitions in the most efficient way?

  A. Use operation DataFrame.repartition(8) to shuffle the DataFrame and reduce the number of partitions.
  B. Use operation DataFrame.coalesce(8) to fully shuffle the DataFrame and reduce the number of partitions.
  C. Use a narrow transformation to reduce the number of partitions.
  D. Use a wide transformation to reduce the number of partitions.
  E. Use operation DataFrame.coalesce(0.5) to halve the number of partitions in the DataFrame.
  C. Use a narrow transformation to reduce the number of partitions.

• Question 16:

  Which of the following code blocks returns DataFrame transactionsDf sorted in descending order by column predError, showing missing values last?

  A. transactionsDf.sort(asc_nulls_last("predError"))
  B. transactionsDf.orderBy("predError").desc_nulls_last()
  C. transactionsDf.sort("predError", ascending=False)
  D. transactionsDf.desc_nulls_last("predError")
  E. transactionsDf.orderBy("predError").asc_nulls_last()
  C. transactionsDf.sort("predError", ascending=False)

  ---

  transactionsDf.sort("predError", ascending=False)

  Correct! When using DataFrame.sort() and setting ascending=False, the DataFrame will be sorted by the specified column in descending order, putting all missing values last. An alternative, although not listed as an answer here, would be

  transactionsDf.sort(desc_nulls_last("predError")).

  transactionsDf.sort(asc_nulls_last("predError"))

  Incorrect. While this is valid syntax, the DataFrame will be sorted on column predError in ascending order and not in descending order, putting missing values last.

  transactionsDf.desc_nulls_last("predError")

  Wrong, this is invalid syntax. There is no method DataFrame.desc_nulls_last() in the Spark API. There is a Spark function desc_nulls_last() however (link see below). transactionsDf.orderBy("predError").desc_nulls_last() No. While

  transactionsDf.orderBy("predError") is correct syntax (although it sorts the DataFrame by column predError in ascending order) and returns a DataFrame, there is no method

  DataFrame.desc_nulls_last() in the Spark API. There is a Spark function desc_nulls_last() however (link see below).

  transactionsDf.orderBy("predError").asc_nulls_last() Incorrect. There is no method DataFrame.asc_nulls_last() in the Spark API (see above). More info: pyspark.sql.functions.desc_nulls_last -- PySpark 3.1.2 documentation and

  pyspark.sql.DataFrame.sort -- PySpark 3.1.2 documentation (https://bit.ly/3g1JtbI , https://bit.ly/2R90NCS)

  Static notebook | Dynamic notebook: See test 1, 32 (Databricks import instructions) (https://flrs.github.io/spark_practice_tests_code/#1/32.html , https://bit.ly/sparkpracticeexams_import_instructions)

• Question 17:

  Which of the following code blocks reads the parquet file stored at filePath into DataFrame itemsDf, using a valid schema for the sample of itemsDf shown below?

  Sample of itemsDf:

  1.+------+-----------------------------+-------------------+

  2.|itemId|attributes |supplier |

  3.+------+-----------------------------+-------------------+

  4.|1 |[blue, winter, cozy] |Sports Company Inc.|

  5.|2 |[red, summer, fresh, cooling]|YetiX |

  6.|3 |[green, summer, travel] |Sports Company Inc.|

  7.+------+-----------------------------+-------------------+

  A. 1.itemsDfSchema = StructType([ 2. StructField("itemId", IntegerType()), 3. StructField("attributes", StringType()), 4. StructField("supplier", StringType())]) 5. 6.itemsDf = spark.read.schema(itemsDfSchema).parquet(filePath)
  B. 1.itemsDfSchema = StructType([ 2. StructField("itemId", IntegerType), 3. StructField("attributes", ArrayType(StringType)), 4. StructField("supplier", StringType)]) 5. 6.itemsDf = spark.read.schema(itemsDfSchema).parquet(filePath)
  C. 1.itemsDf = spark.read.schema('itemId integer, attributes , supplier string').parquet(filePath)
  D. 1.itemsDfSchema = StructType([ 2. StructField("itemId", IntegerType()), 3. StructField("attributes", ArrayType(StringType())), 4. StructField("supplier", StringType())]) 5. 6.itemsDf = spark.read.schema(itemsDfSchema).parquet(filePath)
  E. 1.itemsDfSchema = StructType([ 2. StructField("itemId", IntegerType()), 3. StructField("attributes", ArrayType([StringType()])), 4. StructField("supplier", StringType())]) 5. 6.itemsDf = spark.read(schema=itemsDfSchema).parquet(filePath)
  D. 1.itemsDfSchema = StructType([ 2. StructField("itemId", IntegerType()), 3. StructField("attributes", ArrayType(StringType())), 4. StructField("supplier", StringType())]) 5. 6.itemsDf = spark.read.schema(itemsDfSchema).parquet(filePath)

  ---

  The challenge in this comes from there being an array variable in the schema. In addition, you should know how to pass a schema to the DataFrameReader that is invoked by

  spark.read.

  The correct way to define an array of strings in a schema is through ArrayType(StringType()). A schema can be passed to the DataFrameReader by simply appending schema(structType) to the

  read() operator. Alternatively, you can also define a schema as a string. For example, for the schema of itemsDf, the following string would make sense: itemId integer, attributes array,

  supplier string.

  A thing to keep in mind is that in schema definitions, you always need to instantiate the types, like so: StringType(). Just using StringType does not work in pySpark and will fail. Another concern with schemas is whether columns should be

  nullable, so allowed to have null values. In the case at hand, this is not a concern however, since the

  just asks for a

  "valid"

  schema. Both non-nullable and nullable column schemas would be valid here, since no null value appears in the DataFrame sample.

  More info: Learning Spark, 2nd Edition, Chapter 3

  Static notebook | Dynamic notebook: See test 3, 19 (Databricks import instructions)

• Question 18:

  The code block displayed below contains an error. The code block should save DataFrame transactionsDf at path path as a parquet file, appending to any existing parquet file. Find the error.

  Code block:

  A. transactionsDf.format("parquet").option("mode", "append").save(path)
  B. The code block is missing a reference to the DataFrameWriter.
  C. save() is evaluated lazily and needs to be followed by an action.
  D. The mode option should be omitted so that the command uses the default mode.
  E. The code block is missing a bucketBy command that takes care of partitions.
  F. Given that the DataFrame should be saved as parquet file, path is being passed to the wrong method.
  B. The code block is missing a reference to the DataFrameWriter.

  ---

  Correct code block:

  transactionsDf.write.format("parquet").option("mode", "append").save(path)

• Question 19:

  The code block shown below should return a copy of DataFrame transactionsDf with an added column cos. This column should have the values in column value converted to degrees and having the cosine of those converted values taken,

  rounded to two decimals. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  Code block:

  transactionsDf.__1__(__2__, round(__3__(__4__(__5__)),2))

  A. 1. withColumn 2. col("cos") 3. cos 4. degrees 5. transactionsDf.value
  B. 1. withColumnRenamed 2. "cos" 3. cos 4. degrees 5. "transactionsDf.value"
  C. 1. withColumn 2. "cos" 3. cos 4. degrees 5. transactionsDf.value
  D. 1. withColumn 2. col("cos") 3. cos 4. degrees 5. col("value")
  E. 1. withColumn 2. "cos" 3. degrees 4. cos 5. col("value")
  C. 1. withColumn 2. "cos" 3. cos 4. degrees 5. transactionsDf.value

• Question 20:

  Which of the following code blocks uses a schema fileSchema to read a parquet file at location filePath into a DataFrame?

  A. spark.read.schema(fileSchema).format("parquet").load(filePath)
  B. spark.read.schema("fileSchema").format("parquet").load(filePath)
  C. spark.read().schema(fileSchema).parquet(filePath)
  D. spark.read().schema(fileSchema).format(parquet).load(filePath)
  E. spark.read.schema(fileSchema).open(filePath)
  A. spark.read.schema(fileSchema).format("parquet").load(filePath)

  ---

  Pay attention here to which variables are quoted. fileSchema is a variable and thus should not be in quotes. parquet is not a variable and therefore should be in quotes. SparkSession.read (here referenced as spark.read) returns a

  DataFrameReader which all subsequent calls reference - the DataFrameReader is not callable, so you should not use parentheses

  here.

  Finally, there is no open method in PySpark. The method name is load. Static notebook | Dynamic notebook: See test 1, 44 (Databricks import instructions)