Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Questions & Answers

• Question 171:

  Which of the following code blocks reads all CSV files in directory filePath into a single DataFrame, with column names defined in the CSV file headers?

  Content of directory filePath:

  1._SUCCESS

  2._committed_2754546451699747124

  3._started_2754546451699747124

  4.part-00000-tid-2754546451699747124-10eb85bf-8d91-4dd0-b60b-2f3c02eeecaa-298-1- c000.csv.gz

  5.part-00001-tid-2754546451699747124-10eb85bf-8d91-4dd0-b60b-2f3c02eeecaa-299-1- c000.csv.gz

  6.part-00002-tid-2754546451699747124-10eb85bf-8d91-4dd0-b60b-2f3c02eeecaa-300-1- c000.csv.gz

  7.part-00003-tid-2754546451699747124-10eb85bf-8d91-4dd0-b60b-2f3c02eeecaa-301-1- c000.csv.gz

  spark.option("header",True).csv(filePath)

  A. spark.read.format("csv").option("header",True).option("compression","zip").load(filePath)
  B. spark.read().option("header",True).load(filePath)
  C. spark.read.format("csv").option("header",True).load(filePath)
  D. spark.read.load(filePath)
  C. spark.read.format("csv").option("header",True).load(filePath)

• Question 172:

  Which of the following code blocks reads JSON file imports.json into a DataFrame?

  A. spark.read().mode("json").path("/FileStore/imports.json")
  B. spark.read.format("json").path("/FileStore/imports.json")
  C. spark.read("json", "/FileStore/imports.json")
  D. spark.read.json("/FileStore/imports.json")
  E. spark.read().json("/FileStore/imports.json")
  D. spark.read.json("/FileStore/imports.json")

  ---

  Static notebook | Dynamic notebook: See test 1, 25 (Databricks import instructions) (https://flrs.github.io/spark_practice_tests_code/#1/25.html , https://bit.ly/sparkpracticeexams_import_instructions)

• Question 173:

  The code block displayed below contains an error. The code block should return the average of rows in column value grouped by unique storeId. Find the error.

  Code block:

  transactionsDf.agg("storeId").avg("value")

  A. Instead of avg("value"), avg(col("value")) should be used.
  B. The avg("value") should be specified as a second argument to agg() instead of being appended to it.
  C. All column names should be wrapped in col() operators.
  D. agg should be replaced by groupBy.
  E. "storeId" and "value" should be swapped.
  D. agg should be replaced by groupBy.

• Question 174:

  The code block shown below should show information about the data type that column storeId of DataFrame transactionsDf contains. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  Code block: transactionsDf.__1__(__2__).__3__

  A. 1. select 2. "storeId" 3. print_schema()
  B. 1. limit 2. 1 3. columns
  C. 1. select 2. "storeId" 3. printSchema()
  D. 1. limit 2. "storeId" 3. printSchema()
  E. 1. select 2. storeId 3. dtypes
  B. 1. limit 2. 1 3. columns

  ---

  Correct code block:

  transactionsDf.select("storeId").printSchema()

  The difficulty of this is that it is hard to solve with the stepwise first-to-last- gap approach that has worked well for similar questions, since the answer options are so different from

  one

  another. Instead, you might want to eliminate answers by looking for patterns of frequently wrong answers.

  A first pattern that you may recognize by now is that column names are not expressed in quotes. For this reason, the answer that includes storeId should be eliminated. By now, you may have understood that the DataFrame.limit() is useful for

  returning a specified amount of rows. It has nothing to do with specific columns. For this reason, the answer that resolves to

  limit("storeId") can be eliminated.

  Given that we are interested in information about the data type, you should

• Question 175:

  Which of the following statements about DAGs is correct?

  A. DAGs help direct how Spark executors process tasks, but are a limitation to the proper execution of a query when an executor fails.
  B. DAG stands for "Directing Acyclic Graph".
  C. Spark strategically hides DAGs from developers, since the high degree of automation in Spark means that developers never need to consider DAG layouts.
  D. In contrast to transformations, DAGs are never lazily executed.
  E. DAGs can be decomposed into tasks that are executed in parallel.
  E. DAGs can be decomposed into tasks that are executed in parallel.

• Question 176:

  Which of the following statements about executors is correct, assuming that one can consider each of the JVMs working as executors as a pool of task execution slots?

  A. Slot is another name for executor.
  B. There must be less executors than tasks.
  C. An executor runs on a single core.
  D. There must be more slots than tasks.
  E. Tasks run in parallel via slots.
  E. Tasks run in parallel via slots.

• Question 177:

  Which of the following code blocks displays various aggregated statistics of all columns in DataFrame transactionsDf, including the standard deviation and minimum of values in each column?

  A. transactionsDf.summary()
  B. transactionsDf.agg("count", "mean", "stddev", "25%", "50%", "75%", "min")
  C. transactionsDf.summary("count", "mean", "stddev", "25%", "50%", "75%", "max").show()
  D. transactionsDf.agg("count", "mean", "stddev", "25%", "50%", "75%", "min").show()
  E. transactionsDf.summary().show()
  E. transactionsDf.summary().show()

  ---

  The DataFrame.summary() command is very practical for quickly calculating statistics of a DataFrame. You need to call .show() to display the results of the calculation. By default, the command calculates various statistics (see documentation linked below), including standard deviation and minimum. Note that the answer that lists many options in the summary() parentheses does not include the minimum, which is asked for in the question. Answer options that include agg() do not work here as shown, since DataFrame.agg() expects more complex, column-specific instructions on how to aggregate values.

  More info:

  -pyspark.sql.DataFrame.summary -- PySpark 3.1.2 documentation

  -pyspark.sql.DataFrame.agg -- PySpark 3.1.2 documentation

  Static notebook | Dynamic notebook: See test 3, 46 (Databricks import instructions)

• Question 178:

  The code block displayed below contains an error. The code block should return a DataFrame where all entries in column supplier contain the letter combination et in this order. Find the error.

  Code block:

  itemsDf.filter(Column('supplier').isin('et'))

  A. The Column operator should be replaced by the col operator and instead of isin, contains should be used.
  B. The expression inside the filter parenthesis is malformed and should be replaced by isin('et', 'supplier').
  C. Instead of isin, it should be checked whether column supplier contains the letters et, so isin should be replaced with contains. In addition, the column should be accessed using col['supplier'].
  D. The expression only returns a single column and filter should be replaced by select.
  B. The expression inside the filter parenthesis is malformed and should be replaced by isin('et', 'supplier').

  ---

  Correct code block:

  itemsDf.filter(col('supplier').contains('et'))

  A mixup can easily happen here between isin and contains. Since we want to check whether a column "contains" the values et, this is the operator we should use here. Note that both methods are

  methods of Spark's Column object. See below for documentation links. A specific Column object can be accessed through the col() method and not the Column() method or through col[], which is an essential thing to know here. In PySpark,

  Column references a generic

  column object. To use it for queries, you need to link the generic column object to a specific DataFrame. This can be achieved, for example, through the col() method.

  More info:

  -

  isin documentation: pyspark.sql.Column.isin -- PySpark 3.1.1 documentation

  -

  contains documentation: pyspark.sql.Column.contains -- PySpark 3.1.1 documentation

  Static notebook | Dynamic notebook: See test 1, 51 (Databricks import instructions)

• Question 179:

  Which of the following describes slots?

  A. Slots are dynamically created and destroyed in accordance with an executor's workload.
  B. To optimize I/O performance, Spark stores data on disk in multiple slots.
  C. A Java Virtual Machine (JVM) working as an executor can be considered as a pool of slots for task execution.
  D. A slot is always limited to a single core. Slots are the communication interface for executors and are used for receiving commands and sending results to the driver.
  C. A Java Virtual Machine (JVM) working as an executor can be considered as a pool of slots for task execution.

  ---

  Slots are the communication interface for executors and are used for receiving commands and sending results to the driver. Wrong, executors communicate with the driver directly. Slots are dynamically created and destroyed in accordance with an executor's workload. No, Spark does not actively create and destroy slots in accordance with the workload. Per executor, slots are made available in accordance with how many cores per executor (property spark.executor.cores) and how many CPUs per task (property spark.task.cpus) the Spark configuration calls for. A slot is always limited to a single core. No, a slot can span multiple cores. If a task would require multiple cores, it would have to be executed through a slot that spans multiple cores. In Spark documentation, "core" is often used interchangeably with "thread", although "thread" is the more accurate word. A single physical core may be able to make multiple threads available. So, it is better to say that a slot can span multiple threads. To optimize I/O performance, Spark stores data on disk in multiple slots. No ?Spark stores data on disk in multiple partitions, not slots. More info: Spark Architecture | Distributed Systems Architecture

• Question 180:

  Which of the following statements about lazy evaluation is incorrect?

  A. Predicate pushdown is a feature resulting from lazy evaluation.
  B. Execution is triggered by transformations.
  C. Spark will fail a job only during execution, but not during definition.
  D. Accumulators do not change the lazy evaluation model of Spark.
  E. Lineages allow Spark to coalesce transformations into stages
  B. Execution is triggered by transformations.