Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Questions & Answers

• Question 31:

  Which of the following is not a feature of Adaptive Query Execution?

  A. Replace a sort merge join with a broadcast join, where appropriate.
  B. Coalesce partitions to accelerate data processing.
  C. Split skewed partitions into smaller partitions to avoid differences in partition processing time.
  D. Reroute a query in case of an executor failure.
  E. Collect runtime statistics during query execution.
  D. Reroute a query in case of an executor failure.

• Question 32:

  The code block displayed below contains an error. The code block should read the csv file located at path data/transactions.csv into DataFrame transactionsDf, using the first row as column header

  and casting the columns in the most appropriate type. Find the error.

  First 3 rows of transactions.csv:

  1.transactionId;storeId;productId;name

  2.1;23;12;green grass

  3.2;35;31;yellow sun

  4.3;23;12;green grass

  Code block:

  transactionsDf = spark.read.load("data/transactions.csv", sep=";", format="csv", header=True)

  A. The DataFrameReader is not accessed correctly.
  B. The transaction is evaluated lazily, so no file will be read.
  C. Spark is unable to understand the file type.
  D. The code block is unable to capture all columns.
  E. The resulting DataFrame will not have the appropriate schema.
  E. The resulting DataFrame will not have the appropriate schema.

  ---

  Correct code block:

  transactionsDf = spark.read.load("data/transactions.csv", sep=";", format="csv", header=True, inferSchema=True)

  By default, Spark does not infer the schema of the CSV (since this usually takes some time). So, you need to add the inferSchema=True option to the code block.

  More info: pyspark.sql.DataFrameReader.csv -- PySpark 3.1.2 documentation

• Question 33:

  Which of the following code blocks efficiently converts DataFrame transactionsDf from 12 into 24 partitions?

  A. transactionsDf.repartition(24, boost=True)
  B. transactionsDf.repartition()
  C. transactionsDf.repartition("itemId", 24)
  D. transactionsDf.coalesce(24)
  E. transactionsDf.repartition(24)
  E. transactionsDf.repartition(24)

• Question 34:

  Which of the following code blocks returns approximately 1000 rows, some of them potentially being duplicates, from the 2000-row DataFrame transactionsDf that only has unique rows?

  A. transactionsDf.sample(True, 0.5)
  B. transactionsDf.take(1000).distinct()
  C. transactionsDf.sample(False, 0.5)
  D. transactionsDf.take(1000)
  E. transactionsDf.sample(True, 0.5, force=True)
  A. transactionsDf.sample(True, 0.5)

  ---

  To solve this question, you need to know that DataFrame.sample() is not guaranteed to return the exact fraction of the number of rows specified as an argument. Furthermore, since duplicates may be returned, you should understand that the operator's withReplacement argument should be set to True. A force= argument for the operator does not exist. While the take argument returns an exact number of rows, it will just take the first specified number of rows (1000 in this question) from the DataFrame. Since the DataFrame does not include duplicate rows, there is no potential of any of those returned rows being duplicates when using take(), so the correct answer cannot involve take().

  More info: pyspark.sql.DataFrame.sample -- PySpark 3.1.2 documentation

  Static notebook | Dynamic notebook: See test 2, 41 (Databricks import instructions)

• Question 35:

  Which of the following is a problem with using accumulators?

  A. Only unnamed accumulators can be inspected in the Spark UI.
  B. Only numeric values can be used in accumulators.
  C. Accumulator values can only be read by the driver, but not by executors.
  D. Accumulators do not obey lazy evaluation.
  E. Accumulators are difficult to use for debugging because they will only be updated once, independent if a task has to be re-run due to hardware failure.
  C. Accumulator values can only be read by the driver, but not by executors.

• Question 36:

  The code block displayed below contains an error. The code block should count the number of rows that have a predError of either 3 or 6. Find the error.

  Code block:

  transactionsDf.filter(col('predError').in([3, 6])).count()

  A. The number of rows cannot be determined with the count() operator.
  B. Instead of filter, the select method should be used.
  C. The method used on column predError is incorrect.
  D. Instead of a list, the values need to be passed as single arguments to the in operator.
  E. Numbers 3 and 6 need to be passed as string variables.
  C. The method used on column predError is incorrect.

• Question 37:

  The code block displayed below contains an error. The code block below is intended to add a column itemNameElements to DataFrame itemsDf that includes an array of all words in column itemName. Find the error.

  Sample of DataFrame itemsDf:

  1.+------+----------------------------------+-------------------+

  2.|itemId|itemName |supplier |

  3.+------+----------------------------------+-------------------+ 4.|1 |Thick Coat for Walking in the Snow|Sports Company Inc.|

  5.|2 |Elegant Outdoors Summer Dress |YetiX |

  6.|3 |Outdoors Backpack |Sports Company Inc.|

  7.+------+----------------------------------+-------------------+

  Code block:

  itemsDf.withColumnRenamed("itemNameElements", split("itemName"))

  A. All column names need to be wrapped in the col() operator.
  B. Operator withColumnRenamed needs to be replaced with operator withColumn and a second argument "," needs to be passed to the split method.
  C. Operator withColumnRenamed needs to be replaced with operator withColumn and the split method needs to be replaced by the splitString method.
  D. Operator withColumnRenamed needs to be replaced with operator withColumn and a second argument " " needs to be passed to the split method. E. The expressions "itemNameElements" and split("itemName") need to be swapped.
  D. Operator withColumnRenamed needs to be replaced with operator withColumn and a second argument " " needs to be passed to the split method. E. The expressions "itemNameElements" and split("itemName") need to be swapped.

• Question 38:

  Which of the following code blocks returns a single-row DataFrame that only has a column corr which shows the Pearson correlation coefficient between columns predError and value in DataFrame transactionsDf?

  A. transactionsDf.select(corr(["predError", "value"]).alias("corr")).first()
  B. transactionsDf.select(corr(col("predError"), col("value")).alias("corr")).first()
  C. transactionsDf.select(corr(predError, value).alias("corr"))
  D. transactionsDf.select(corr(col("predError"), col("value")).alias("corr"))
  E. transactionsDf.select(corr("predError", "value"))
  D. transactionsDf.select(corr(col("predError"), col("value")).alias("corr"))

  ---

  In difficulty, this is above what you can expect from the exam. What this wants to teach you, however, is to pay attention to the useful details included in the

  documentation.

  pyspark.sql.corr is not a very common method, but it deals with Spark's data structure in an interesting way. The command takes two columns over multiple rows and returns a single row - similar to

  an aggregation function. When examining the documentation (linked below), you will find this code example:

  a = range(20)

  b = [2 * x for x in range(20)]

  df = spark.createDataFrame(zip(a, b), ["a", "b"])

  df.agg(corr("a", "b").alias('c')).collect()

  [Row(c=1.0)]

  See how corr just returns a single row? Once you understand this, you should be suspicious about answers that include first(), since there is no need to just select a single row. A reason to eliminate

  those answers is that DataFrame.first() returns an object of type Row, but not DataFrame, as requested in the question.

  transactionsDf.select(corr(col("predError"), col("value")).alias("corr")) Correct! After calculating the Pearson correlation coefficient, the resulting column is correctly renamed to corr.

  transactionsDf.select(corr(predError, value).alias("corr")) No. In this answer, Python will interpret column names predError and value as variable names.

  transactionsDf.select(corr(col("predError"), col("value")).alias("corr")).first() Incorrect. first() returns a row, not a DataFrame (see above and linked documentation below).

  transactionsDf.select(corr("predError", "value"))

  Wrong. Whie this statement returns a DataFrame in the desired shape, the column will have the name corr(predError, value) and not corr.

  transactionsDf.select(corr(["predError", "value"]).alias("corr")).first() False. In addition to first() returning a row, this code block also uses the wrong call structure for command corr which takes two arguments (the two columns to correlate).

  More info:

  -pyspark.sql.functions.corr -- PySpark 3.1.2 documentation

  -pyspark.sql.DataFrame.first -- PySpark 3.1.2 documentation

  Static notebook | Dynamic notebook: See test 3, 53 (Databricks import instructions)

• Question 39:

  Which of the following code blocks returns all unique values of column storeId in DataFrame transactionsDf?

  A. transactionsDf["storeId"].distinct()
  B. transactionsDf.select("storeId").distinct()
  C. transactionsDf.filter("storeId").distinct()
  D. transactionsDf.select(col("storeId").distinct())
  E. transactionsDf.distinct("storeId")
  B. transactionsDf.select("storeId").distinct()

  ---

  distinct() is a method of a DataFrame. Knowing this, or recognizing this from the documentation, is the key to solving this question. More info: pyspark.sql.DataFrame.distinct -- PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2, 19 (Databricks import instructions)

• Question 40:

  The code block displayed below contains an error. The code block should trigger Spark to cache DataFrame transactionsDf in executor memory where available, writing to disk where insufficient

  executor memory is available, in a fault-tolerant way. Find the error.

  Code block:

  transactionsDf.persist(StorageLevel.MEMORY_AND_DISK)

  A. Caching is not supported in Spark, data are always recomputed.
  B. Data caching capabilities can be accessed through the spark object, but not through the DataFrame API.
  C. The storage level is inappropriate for fault-tolerant storage.
  D. The code block uses the wrong operator for caching.
  E. The DataFrameWriter needs to be invoked.
  C. The storage level is inappropriate for fault-tolerant storage.

  ---

  The storage level is inappropriate for fault-tolerant storage. Correct. Typically, when thinking about fault tolerance and storage levels, you would want to store redundant copies of the dataset. This can be achieved by using a storage level

  such as

  StorageLevel.MEMORY_AND_DISK_2.

  The code block uses the wrong command for caching. Wrong. In this case, DataFrame.persist() needs to be used, since this operator supports passing a storage level. DataFrame.cache() does not support passing a storage level.

  Caching is not supported in Spark, data are always recomputed. Incorrect. Caching is an important component of Spark, since it can help to accelerate Spark programs to great extent. Caching is often a good idea for datasets that need to be

  accessed

  repeatedly.

  Data caching capabilities can be accessed through the spark object, but not through the DataFrame API.

  No. Caching is either accessed through DataFrame.cache() or DataFrame.persist().

  The DataFrameWriter needs to be invoked.

  Wrong. The DataFrameWriter can be accessed via DataFrame.write and is used to write data to external data stores, mostly on disk. Here, we find keywords such as "cache" and "executor

  memory" that point us away from using external data stores. We aim to save data to memory to accelerate the reading process, since reading from disk is comparatively slower.

  The

  DataFrameWriter does not write to memory, so we cannot use it here.

  More info: Best practices for caching in Spark SQL | by David Vrba | Towards Data Science