Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Questions & Answers

• Question 101:

  Which of the following code blocks concatenates rows of DataFrames transactionsDf and transactionsNewDf, omitting any duplicates?

  A. transactionsDf.concat(transactionsNewDf).unique()
  B. transactionsDf.union(transactionsNewDf).distinct()
  C. spark.union(transactionsDf, transactionsNewDf).distinct()
  D. transactionsDf.join(transactionsNewDf, how="union").distinct()
  E. transactionsDf.union(transactionsNewDf).unique()
  B. transactionsDf.union(transactionsNewDf).distinct()

  ---

  DataFrame.unique() and DataFrame.concat() do not exist and union() is not a method of the SparkSession. In addition, there is no union option for the join method in the DataFrame.join() statement.

  More info: pyspark.sql.DataFrame.union -- PySpark 3.1.2 documentation

  Static notebook | Dynamic notebook: See test 2, 43 (Databricks import instructions)

• Question 102:

  The code block shown below should return an exact copy of DataFrame transactionsDf that does not include rows in which values in column storeId have the value 25. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  A. transactionsDf.remove(transactionsDf.storeId==25)
  B. transactionsDf.where(transactionsDf.storeId!=25)
  C. transactionsDf.filter(transactionsDf.storeId==25)
  D. transactionsDf.drop(transactionsDf.storeId==25)
  E. transactionsDf.select(transactionsDf.storeId!=25)
  B. transactionsDf.where(transactionsDf.storeId!=25)

  ---

  transactionsDf.where(transactionsDf.storeId!=25)

  Correct. DataFrame.where() is an alias for the DataFrame.filter() method. Using this method, it is straightforward to filter out rows that do not have value 25 in column storeId.

  transactionsDf.select(transactionsDf.storeId!=25)

  Wrong. The select operator allows you to build DataFrames column-wise, but when using it as shown, it does not filter out rows.

  transactionsDf.filter(transactionsDf.storeId==25)

  Incorrect. Although the filter expression works for filtering rows, the == in the filtering condition is inappropriate. It should be != instead.

  transactionsDf.drop(transactionsDf.storeId==25)

  No. DataFrame.drop() is used to remove specific columns, but not rows, from the DataFrame.

  transactionsDf.remove(transactionsDf.storeId==25)

  False. There is no DataFrame.remove() operator in PySpark. More info: pyspark.sql.DataFrame.where -- PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 3, 48 (Databricks import instructions)

• Question 103:

  The code block shown below should read all files with the file ending .png in directory path into Spark. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  spark.__1__.__2__(__3__).option(__4__, "*.png").__5__(path)

  A. 1. read() 2. format 3. "binaryFile" 4. "recursiveFileLookup" 5. load
  B. 1. read 2. format 3. "binaryFile" 4. "pathGlobFilter" 5. load
  C. 1. read 2. format 3. binaryFile 4. pathGlobFilter 5. load
  D. 1. open 2. format 3. "image" 4. "fileType" 5. open
  E. 1. open 2. as 3. "binaryFile" 4. "pathGlobFilter" 5. load
  B. 1. read 2. format 3. "binaryFile" 4. "pathGlobFilter" 5. load

  ---

  Correct code block: spark.read.format("binaryFile").option("recursiveFileLookup", "*.png").load(path) Spark can deal with binary files, like images. Using the binaryFile format specification in the SparkSession's read API is the way to read in those files. Remember that, to access the read API, you need to start the command with spark.read. The pathGlobFilter option is a great way to filter files by name (and ending). Finally, the path can be specified using the load operator ?the open operator shown in one of the answers does not exist.

• Question 104:

  Which of the following describes properties of a shuffle?

  A. Operations involving shuffles are never evaluated lazily.
  B. Shuffles involve only single partitions.
  C. Shuffles belong to a class known as "full transformations".
  D. A shuffle is one of many actions in Spark.
  E. In a shuffle, Spark writes data to disk.
  E. In a shuffle, Spark writes data to disk.

  ---

  In a shuffle, Spark writes data to disk.

  Correct! Spark's architecture dictates that intermediate results during a shuffle are written to disk.

  A shuffle is one of many actions in Spark.

  Incorrect. A shuffle is a transformation, but not an action.

  Shuffles involve only single partitions.

  No, shuffles involve multiple partitions. During a shuffle, Spark generates output partitions from multiple input partitions.

  Operations involving shuffles are never evaluated lazily. Wrong. A shuffle is a costly operation and Spark will evaluate it as lazily as other transformations. This is, until a subsequent action triggers its evaluation. Shuffles belong to a class

  known as "full transformations". Not quite. Shuffles belong to a class known as "wide transformations". "Full transformation" is not a relevant term in Spark.

  More info: Spark ?The Definitive Guide, Chapter 2 and Spark: disk I/O on stage boundaries explanation - Stack Overflow

• Question 105:

  Which of the following code blocks can be used to save DataFrame transactionsDf to memory only, recalculating partitions that do not fit in memory when they are needed?

  A. from pyspark import StorageLevel transactionsDf.cache(StorageLevel.MEMORY_ONLY)
  B. transactionsDf.cache()
  C. transactionsDf.storage_level('MEMORY_ONLY')
  D. transactionsDf.persist()
  E. transactionsDf.clear_persist()
  F. from pyspark import StorageLevel transactionsDf.persist(StorageLevel.MEMORY_ONLY)
  F. from pyspark import StorageLevel transactionsDf.persist(StorageLevel.MEMORY_ONLY)

• Question 106:

  The code block shown below should write DataFrame transactionsDf as a parquet file to path storeDir, using brotli compression and replacing any previously existing file. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  transactionsDf.__1__.format("parquet").__2__(__3__).option(__4__, "brotli").__5__(storeDir)

  A. 1. save 2. mode 3. "ignore" 4. "compression" 5. path
  B. 1. store 2. with 3. "replacement" 4. "compression" 5. path
  C. 1. write 2. mode 3. "overwrite" 4. "compression" 5. save (Correct)
  D. 1. save 2. mode 3. "replace" 4. "compression" 5. path
  E. 1. write 2. mode 3. "overwrite" 4. compression 5. parquet
  D. 1. save 2. mode 3. "replace" 4. "compression" 5. path

• Question 107:

  Which of the following options describes the responsibility of the executors in Spark?

  A. The executors accept jobs from the driver, analyze those jobs, and return results to the driver.
  B. The executors accept tasks from the driver, execute those tasks, and return results to the cluster manager.
  C. The executors accept tasks from the driver, execute those tasks, and return results to the driver.
  D. The executors accept tasks from the cluster manager, execute those tasks, and return results to the driver.
  E. The executors accept jobs from the driver, plan those jobs, and return results to the cluster manager.
  C. The executors accept tasks from the driver, execute those tasks, and return results to the driver.

  ---

  More info: Running Spark: an overview of Spark's runtime architecture - Manning (https://bit.ly/2RPmJn9)

• Question 108:

  Which of the following statements about the differences between actions and transformations is correct?

  A. Actions are evaluated lazily, while transformations are not evaluated lazily.
  B. Actions generate RDDs, while transformations do not.
  C. Actions do not send results to the driver, while transformations do.
  D. Actions can be queued for delayed execution, while transformations can only be processed immediately.
  E. Actions can trigger Adaptive Query Execution, while transformation cannot.
  E. Actions can trigger Adaptive Query Execution, while transformation cannot.

• Question 109:

  Which of the following code blocks performs an inner join of DataFrames transactionsDf and itemsDf on columns productId and itemId, respectively, excluding columns value and storeId from DataFrame transactionsDf and column attributes from DataFrame itemsDf?

  A. transactionsDf.drop('value', 'storeId').join(itemsDf.select('attributes'), transactionsDf.productId==itemsDf.itemId)
  B. 1.transactionsDf.createOrReplaceTempView('transactionsDf') 2.itemsDf.createOrReplaceTempView('itemsDf') 3.spark.sql("SELECT -value, -storeId FROM transactionsDf INNER JOIN itemsDf ON productId==itemId").drop("attributes")
  C. transactionsDf.drop("value", "storeId").join(itemsDf.drop("attributes"), "transactionsDf.productId==itemsDf.itemId")
  D. 1.transactionsDf \ 2. .drop(col('value'), col('storeId')) \ 3. .join(itemsDf.drop(col('attributes')), col('productId')==col('itemId'))
  E. 1.transactionsDf.createOrReplaceTempView('transactionsDf') 2.itemsDf.createOrReplaceTempView('itemsDf') 3.statement = """ 4.SELECT * FROM transactionsDf 5.INNER JOIN itemsDf 6.ON transactionsDf.productId==itemsDf.itemId 7.""" 8.spark.sql(statement).drop("value", "storeId", "attributes")
  E. 1.transactionsDf.createOrReplaceTempView('transactionsDf') 2.itemsDf.createOrReplaceTempView('itemsDf') 3.statement = """ 4.SELECT * FROM transactionsDf 5.INNER JOIN itemsDf 6.ON transactionsDf.productId==itemsDf.itemId 7.""" 8.spark.sql(statement).drop("value", "storeId", "attributes")

• Question 110:

  Which of the following code blocks immediately removes the previously cached DataFrame transactionsDf from memory and disk?

  A. array_remove(transactionsDf, "*")
  B. transactionsDf.unpersist()
  C. del transactionsDf
  D. transactionsDf.clearCache()
  E. transactionsDf.persist()
  B. transactionsDf.unpersist()

  ---

  transactionsDf.unpersist()

  Correct. The DataFrame.unpersist() command does exactly what the asks for - it removes all cached parts of the DataFrame from memory and disk.

  del transactionsDf

  False. While this option can help remove the DataFrame from memory and disk, it does not do so immediately. The reason is that this command just notifies the Python garbage collector that the

  transactionsDf now may be deleted from memory. However, the garbage collector does not do so immediately and, if you wanted it to run immediately, would need to be specifically triggered to do

  so. Find more information linked below.

  array_remove(transactionsDf, "*")

  Incorrect. The array_remove method from pyspark.sql.functions is used for removing elements from arrays in columns that match a specific condition. Also, the first argument would be a column, and

  not a DataFrame as shown in the code block.

  transactionsDf.persist()

  No. This code block does exactly the opposite of what is asked for: It caches (writes) DataFrame transactionsDf to memory and disk. Note that even though you do not pass in a specific storage

  level here, Spark will use the default storage level (MEMORY_AND_DISK).

  transactionsDf.clearCache()

  Wrong. Spark's DataFrame does not have a clearCache() method.

  More info: pyspark.sql.DataFrame.unpersist -- PySpark 3.1.2 documentation, python - How to delete an RDD in PySpark for the purpose of releasing resources? - Stack Overflow

  Static notebook | Dynamic notebook: See test 3, 40 (Databricks import instructions)