Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Questions & Answers

• Question 51:

  Which of the following describes how Spark achieves fault tolerance?

  A. Spark helps fast recovery of data in case of a worker fault by providing the MEMORY_AND_DISK storage level option.
  B. If an executor on a worker node fails while calculating an RDD, that RDD can be recomputed by another executor using the lineage.
  C. Spark builds a fault-tolerant layer on top of the legacy RDD data system, which by itself is not fault tolerant.
  D. Due to the mutability of DataFrames after transformations, Spark reproduces them using observed lineage in case of worker node failure.
  E. Spark is only fault-tolerant if this feature is specifically enabled via the spark.fault_recovery.enabled property.
  B. If an executor on a worker node fails while calculating an RDD, that RDD can be recomputed by another executor using the lineage.

• Question 52:

  The code block displayed below contains an error. The code block is intended to write DataFrame transactionsDf to disk as a parquet file in location /FileStore/transactions_split, using column storeId as key for partitioning. Find the error.

  Code block:

  transactionsDf.write.format("parquet").partitionOn("storeId").save("/FileStore/transactions_s plit")A.

  A. The format("parquet") expression is inappropriate to use here, "parquet" should be passed as first argument to the save() operator and "/FileStore/transactions_split" as the second argument.
  B. Partitioning data by storeId is possible with the partitionBy expression, so partitionOn should be replaced by partitionBy.
  C. Partitioning data by storeId is possible with the bucketBy expression, so partitionOn should be replaced by bucketBy.
  D. partitionOn("storeId") should be called before the write operation.
  E. The format("parquet") expression should be removed and instead, the information should be added to the write expression like so: write("parquet").
  B. Partitioning data by storeId is possible with the partitionBy expression, so partitionOn should be replaced by partitionBy.

• Question 53:

  In which order should the code blocks shown below be run in order to read a JSON file from location jsonPath into a DataFrame and return only the rows that do not have value 3 in column productId?

  1.

  importedDf.createOrReplaceTempView("importedDf")

  2.

  spark.sql("SELECT * FROM importedDf WHERE productId != 3")

  3.

  spark.sql("FILTER * FROM importedDf WHERE productId != 3")

  4.

  importedDf = spark.read.option("format", "json").path(jsonPath)

  5.

  importedDf = spark.read.json(jsonPath)

  A. 4, 1, 2
  B. 5, 1, 3
  C. 5, 2
  D. 4, 1, 3
  E. 5, 1, 2
  E. 5, 1, 2

• Question 54:

  Which of the following code blocks returns a 2-column DataFrame that shows the distinct values in column productId and the number of rows with that productId in DataFrame transactionsDf?

  A. transactionsDf.count("productId").distinct()
  B. transactionsDf.groupBy("productId").agg(col("value").count())
  C. transactionsDf.count("productId")
  D. transactionsDf.groupBy("productId").count()
  E. transactionsDf.groupBy("productId").select(count("value"))
  D. transactionsDf.groupBy("productId").count()

  ---

  transactionsDf.groupBy("productId").count()

  Correct. This code block first groups DataFrame transactionsDf by column productId and then counts the rows in each group.

  transactionsDf.groupBy("productId").select(count("value")) Incorrect. You cannot call select on a GroupedData object (the output of a groupBy) statement.

  transactionsDf.count("productId")

  No. DataFrame.count() does not take any arguments.

  transactionsDf.count("productId").distinct()

  Wrong. Since DataFrame.count() does not take any arguments, this option cannot be right. transactionsDf.groupBy("productId").agg(col("value").count()) False. A Column object, as returned by col("value"), does not have a count() method.

  You can see all available methods for Column object linked in the Spark documentation below. More info: pyspark.sql.DataFrame.count -- PySpark 3.1.2 documentation, pyspark.sql.Column -- PySpark 3.1.2 documentation

  Static notebook | Dynamic notebook: See test 3, 41 (Databricks import instructions)

• Question 55:

  The code block displayed below contains an error. The code block should return all rows of DataFrame transactionsDf, but including only columns storeId and predError. Find the error.

  Code block:

  spark.collect(transactionsDf.select("storeId", "predError"))

  A. Instead of select, DataFrame transactionsDf needs to be filtered using the filter operator.
  B. Columns storeId and predError need to be represented as a Python list, so they need to be wrapped in brackets ([]).
  C. The take method should be used instead of the collect method.
  D. Instead of collect, collectAsRows needs to be called.
  E. The collect method is not a method of the SparkSession object.
  E. The collect method is not a method of the SparkSession object.

  ---

  Correct code block:

  transactionsDf.select("storeId", "predError").collect() collect() is a method of the DataFrame object.

  More info: pyspark.sql.DataFrame.collect -- PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2, 24 (Databricks import instructions)

• Question 56:

  Which of the following code blocks reads in the parquet file stored at location filePath, given that all columns in the parquet file contain only whole numbers and are stored in the most appropriate

  format for this kind of data?

  A. 1.spark.read.schema( 2. StructType( 3. StructField("transactionId", IntegerType(), True), 4. StructField("predError", IntegerType(), True) 5. )).load(filePath)
  B. 1.spark.read.schema([ 2. StructField("transactionId", NumberType(), True), 3. StructField("predError", IntegerType(), True) 4. ]).load(filePath)
  C. 1.spark.read.schema( 2. StructType([ 3. StructField("transactionId", StringType(), True), 4. StructField("predError", IntegerType(), True)] 5. )).parquet(filePath)
  D. 1.spark.read.schema( 2. StructType([ 3. StructField("transactionId", IntegerType(), True), 4. StructField("predError", IntegerType(), True)] 5. )).format("parquet").load(filePath)
  E. 1.spark.read.schema([ 2. StructField("transactionId", IntegerType(), True), 3. StructField("predError", IntegerType(), True) 4. ]).load(filePath, format="parquet")
  D. 1.spark.read.schema( 2. StructType([ 3. StructField("transactionId", IntegerType(), True), 4. StructField("predError", IntegerType(), True)] 5. )).format("parquet").load(filePath)

  ---

  The schema passed into schema should be of type StructType or a string, so all entries in which a list is passed are incorrect.

  In addition, since all numbers are whole numbers, the IntegerType() data type is the correct option here. NumberType() is not a valid data type and StringType() would fail, since the parquet file is

  stored in the "most appropriate format for this kind of data", meaning that it is most likely an IntegerType, and Spark does not convert data types if a schema is provided. Also note that StructType accepts only a single argument (a list of

  StructFields). So, passing multiple arguments is invalid.

  Finally, Spark needs to know which format the file is in. However, all of the options listed are valid here, since Spark assumes parquet as a default when no file format is specifically passed.

  More info: pyspark.sql.DataFrameReader.schema -- PySpark 3.1.2 documentation and StructType -- PySpark 3.1.2 documentation

• Question 57:

  Which of the following code blocks reorders the values inside the arrays in column attributes of DataFrame itemsDf from last to first one in the alphabet?

  1.+------+-----------------------------+-------------------+

  2.|itemId|attributes |supplier |

  3.+------+-----------------------------+-------------------+

  4.|1 |[blue, winter, cozy] |Sports Company Inc.|

  5.|2 |[red, summer, fresh, cooling]|YetiX |

  6.|3 |[green, summer, travel] |Sports Company Inc.|

  7.+------+-----------------------------+-------------------+

  A. itemsDf.withColumn('attributes', sort_array(col('attributes').desc()))
  B. itemsDf.withColumn('attributes', sort_array(desc('attributes')))
  C. itemsDf.withColumn('attributes', sort(col('attributes'), asc=False))
  D. itemsDf.withColumn("attributes", sort_array("attributes", asc=False))
  E. itemsDf.select(sort_array("attributes"))
  D. itemsDf.withColumn("attributes", sort_array("attributes", asc=False))

• Question 58:

  Which of the following code blocks displays the 10 rows with the smallest values of column value in DataFrame transactionsDf in a nicely formatted way?

  A. transactionsDf.sort(asc(value)).show(10)
  B. transactionsDf.sort(col("value")).show(10)
  C. transactionsDf.sort(col("value").desc()).head()
  D. transactionsDf.sort(col("value").asc()).print(10)
  E. transactionsDf.orderBy("value").asc().show(10)
  B. transactionsDf.sort(col("value")).show(10)

• Question 59:

  The code block shown below should return all rows of DataFrame itemsDf that have at least 3 items in column itemNameElements. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  Example of DataFrame itemsDf:

  1.+------+----------------------------------+-------------------+------------------------------------------+

  2.|itemId|itemName |supplier |itemNameElements |

  3.+------+----------------------------------+-------------------+------------------------------------------+

  4.|1 |Thick Coat for Walking in the Snow|Sports Company Inc.|[Thick, Coat, for, Walking, in, the, Snow]|

  5.|2 |Elegant Outdoors Summer Dress |YetiX |[Elegant, Outdoors, Summer, Dress] |

  6.|3 |Outdoors Backpack |Sports Company Inc.|[Outdoors, Backpack] |

  7.+------+----------------------------------+-------------------+------------------------------------------+

  Code block:

  itemsDf.__1__(__2__(__3__)__4__)

  A. 1. select 2. count 3. col("itemNameElements") 4. >3
  B. 1. filter 2. count 3. itemNameElements 4. >=3
  C. 1. select 2. count 3. "itemNameElements" 4. >3
  D. 1. filter 2. size 3. "itemNameElements" 4. >=3
  E. 1. select 2. size 3. "itemNameElements" 4. >3
  D. 1. filter 2. size 3. "itemNameElements" 4. >=3

  ---

  Correct code block:

  itemsDf.filter(size("itemNameElements")>3)

  Output of code block:

  +------+----------------------------------+-------------------+------------------------------------------+ |itemId|itemName |supplier |itemNameElements |

  +------+----------------------------------+-------------------+------------------------------------------+ |1 |Thick Coat for Walking in the Snow|Sports Company Inc.|[Thick, Coat, for, Walking, in, the, Snow]|

  |2 |Elegant Outdoors Summer Dress |YetiX |[Elegant, Outdoors, Summer, Dress] | +------+----------------------------------+-------------------+------------------------------------------+ The big difficulty with this is in knowing the difference between count and

  size (refer to documentation below). size is the correct function to choose here since it returns the number

  of elements in an array on a per-row basis.

  The other consideration for solving this is the difference between select and filter. Since we want to return the rows in the original DataFrame, filter is the right choice. If we would use select, we would simply get a single-column DataFrame

  showing which rows match the criteria, like so:

  +----------------------------+

  |(size(itemNameElements) > 3)|

  +----------------------------+

  |true |

  |true |

  |false |

  +----------------------------+

  More info:

  Count documentation: pyspark.sql.functions.count -- PySpark 3.1.1 documentation Size documentation: pyspark.sql.functions.size -- PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1, 47 (Databricks import

  instructions)

• Question 60:

  The code block shown below should return a DataFrame with only columns from DataFrame transactionsDf for which there is a corresponding transactionId in DataFrame itemsDf. DataFrame itemsDf is very small and much smaller than

  DataFrame transactionsDf. The query should be executed in an optimized way. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  __1__.__2__(__3__, __4__, __5__)

  A. 1. transactionsDf 2. join 3. broadcast(itemsDf) 4. transactionsDf.transactionId==itemsDf.transactionId 5. "outer"
  B. 1. transactionsDf 2. join 3. itemsDf 4. transactionsDf.transactionId==itemsDf.transactionId 5. "anti"
  C. 1. transactionsDf 2. join 3. broadcast(itemsDf) 4. "transactionId" 5. "left_semi"
  D. 1. itemsDf 2. broadcast 3. transactionsDf 4. "transactionId" 5. "left_semi"
  E. 1. itemsDf 2. join 3. broadcast(transactionsDf) 4. "transactionId" 5. "left_semi"
  C. 1. transactionsDf 2. join 3. broadcast(itemsDf) 4. "transactionId" 5. "left_semi"