Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Questions & Answers

• Question 131:

  Which of the following code blocks stores DataFrame itemsDf in executor memory and, if insufficient memory is available, serializes it and saves it to disk?

  A. itemsDf.persist(StorageLevel.MEMORY_ONLY)
  B. itemsDf.cache(StorageLevel.MEMORY_AND_DISK)
  C. itemsDf.store()
  D. itemsDf.cache()
  E. itemsDf.write.option('destination', 'memory').save()
  D. itemsDf.cache()

• Question 132:

  Which of the following describes the conversion of a computational query into an execution plan in Spark?

  A. Spark uses the catalog to resolve the optimized logical plan.
  B. The catalog assigns specific resources to the optimized memory plan.
  C. The executed physical plan depends on a cost optimization from a previous stage.
  D. Depending on whether DataFrame API or SQL API are used, the physical plan may differ.
  E. The catalog assigns specific resources to the physical plan.
  C. The executed physical plan depends on a cost optimization from a previous stage.

  ---

  The executed physical plan depends on a cost optimization from a previous stage. Correct! Spark considers multiple physical plans on which it performs a cost analysis and selects the final physical plan in accordance with the lowest-cost

  outcome of that analysis.

  That final physical plan is then executed by Spark.

  Spark uses the catalog to resolve the optimized logical plan. No. Spark uses the catalog to resolve the unresolved logical plan, but not the optimized logical plan. Once the unresolved logical plan is resolved, it is then optimized using the

  Catalyst Optimizer.

  The optimized logical plan is the input for physical planning. The catalog assigns specific resources to the physical plan. No. The catalog stores metadata, such as a list of names of columns, data types, functions, and databases. Spark

  consults the catalog for resolving the references in a logical plan at the beginning of the conversion of the query into an execution plan. The result is then an optimized logical plan.

  Depending on whether DataFrame API or SQL API are used, the physical plan may differ. Wrong ?the physical plan is independent of which API was used. And this is one of the great strengths of Spark!

  The catalog assigns specific resources to the optimized memory plan. There is no specific "memory plan" on the journey of a Spark computation. More info: Spark's Logical and Physical plans ... When, Why, How and Beyond. | by Laurent

  Leturgez | datalex | Medium

• Question 133:

  Which of the following code blocks returns a copy of DataFrame transactionsDf that only includes columns transactionId, storeId, productId and f?

  Sample of DataFrame transactionsDf:

  1.+-------------+---------+-----+-------+---------+----+

  2.|transactionId|predError|value|storeId|productId| f|

  3.+-------------+---------+-----+-------+---------+----+

  4.| 1| 3| 4| 25| 1|null|

  5.| 2| 6| 7| 2| 2|null|

  6.| 3| 3| null| 25| 3|null|

  7.+-------------+---------+-----+-------+---------+----+

  A. transactionsDf.drop(col("value"), col("predError"))
  B. transactionsDf.drop("predError", "value")
  C. transactionsDf.drop(value, predError)
  D. transactionsDf.drop(["predError", "value"])
  E. transactionsDf.drop([col("predError"), col("value")])
  B. transactionsDf.drop("predError", "value")

• Question 134:

  Which of the following describes Spark's way of managing memory?

  A. Spark uses a subset of the reserved system memory.
  B. Storage memory is used for caching partitions derived from DataFrames.
  C. As a general rule for garbage collection, Spark performs better on many small objects than few big objects.
  D. Disabling serialization potentially greatly reduces the memory footprint of a Spark application.
  E. Spark's memory usage can be divided into three categories: Execution, transaction, and storage.
  B. Storage memory is used for caching partitions derived from DataFrames.

  ---

  Spark's memory usage can be divided into three categories: Execution, transaction, and storage.

  No, it is either execution or storage.

  As a general rule for garbage collection, Spark performs better on many small objects than few big objects.

  No, Spark's garbage collection runs faster on fewer big objects than many small objects. Disabling serialization potentially greatly reduces the memory footprint of a Spark application.

  The opposite is true ?serialization reduces the memory footprint, but may impact performance in a negative way.

  Spark uses a subset of the reserved system memory. No, the reserved system memory is separate from Spark memory. Reserved memory stores Spark's internal objects.

  More info: Tuning - Spark 3.1.2 Documentation, Spark Memory Management | Distributed Systems Architecture, Learning Spark, 2nd Edition, Chapter 7

• Question 135:

  Which of the following code blocks saves DataFrame transactionsDf in location /FileStore/transactions.csv as a CSV file and throws an error if a file already exists in the location?

  A. transactionsDf.write.save("/FileStore/transactions.csv")
  B. transactionsDf.write.format("csv").mode("error").path("/FileStore/transactions.csv")
  C. transactionsDf.write.format("csv").mode("ignore").path("/FileStore/transactions.csv")
  D. transactionsDf.write("csv").mode("error").save("/FileStore/transactions.csv")
  E. transactionsDf.write.format("csv").mode("error").save("/FileStore/transactions.csv")
  E. transactionsDf.write.format("csv").mode("error").save("/FileStore/transactions.csv")

  ---

  Static notebook | Dynamic notebook: See test 1, 28 (Databricks import instructions) (https://flrs.github.io/spark_practice_tests_code/#1/28.html , https://bit.ly/sparkpracticeexams_import_instructions)

• Question 136:

  Which of the following describes characteristics of the Spark UI?

  A. Via the Spark UI, workloads can be manually distributed across executors.
  B. Via the Spark UI, stage execution speed can be modified.
  C. The Scheduler tab shows how jobs that are run in parallel by multiple users are distributed across the cluster.
  D. There is a place in the Spark UI that shows the property spark.executor.memory.
  E. Some of the tabs in the Spark UI are named Jobs, Stages, Storage, DAGs, Executors, and SQL.
  D. There is a place in the Spark UI that shows the property spark.executor.memory.

• Question 137:

  In which order should the code blocks shown below be run in order to assign articlesDf a DataFrame that lists all items in column attributes ordered by the number of times these items occur, from most to least often? Sample of DataFrame articlesDf:

  1.

  articlesDf = articlesDf.groupby("col")

  2.

  articlesDf = articlesDf.select(explode(col("attributes")))

  3.

  articlesDf = articlesDf.orderBy("count").select("col")

  4.

  articlesDf = articlesDf.sort("count",ascending=False).select("col")

  5.

  articlesDf = articlesDf.groupby("col").count()

  A. 4, 5
  B. 2, 5, 3
  C. 5, 2
  D. 2, 3, 4
  E. 2, 5, 4
  E. 2, 5, 4

• Question 138:

  Which of the following code blocks produces the following output, given DataFrame transactionsDf?

  Output:

  1.root

  2.

  |-- transactionId: integer (nullable = true)

  3.

  |-- predError: integer (nullable = true)

  4.

  |-- value: integer (nullable = true)

  5.

  |-- storeId: integer (nullable = true)

  6.

  |-- productId: integer (nullable = true)

  7.

  |-- f: integer (nullable = true)

  DataFrame transactionsDf:

  1.+-------------+---------+-----+-------+---------+----+

  2.|transactionId|predError|value|storeId|productId| f|

  3.+-------------+---------+-----+-------+---------+----+

  4.| 1| 3| 4| 25| 1|null|

  5.| 2| 6| 7| 2| 2|null|

  6.| 3| 3| null| 25| 3|null|

  7.+-------------+---------+-----+-------+---------+----+

  A. transactionsDf.schema.print()
  B. transactionsDf.rdd.printSchema()
  C. transactionsDf.rdd.formatSchema()
  D. transactionsDf.printSchema()
  E. print(transactionsDf.schema)
  D. transactionsDf.printSchema()

  ---

  The output is the typical output of a DataFrame.printSchema() call. The DataFrame's RDD representation does not have a printSchema or formatSchema method (find available methods in the RDD documentation linked below). The output of print(transactionsDf.schema) is this: StructType(List(StructField(transactionId,IntegerType,true),StructField(predError,IntegerTy pe,true),StructField (value,IntegerType,true),StructField(storeId,IntegerType,true),StructField(productId,Integer Type,true),StructField(f,IntegerType,true))). It includes the same information as the nicely formatted original output, but is not nicely formatted itself. Lastly, the DataFrame's schema attribute does not have a print() method.

  More info:

  -pyspark.RDD: pyspark.RDD -- PySpark 3.1.2 documentation

  - DataFrame.printSchema(): pyspark.sql.DataFrame.printSchema -- PySpark 3.1.2 documentation

  Static notebook | Dynamic notebook: See test 2, 52 (Databricks import instructions)

• Question 139:

  Which of the following code blocks creates a new one-column, two-row DataFrame dfDates with column date of type timestamp?

  A. 1.dfDates = spark.createDataFrame(["23/01/2022 11:28:12","24/01/2022 10:58:34"], ["date"]) 2.dfDates = dfDates.withColumn("date", to_timestamp("dd/MM/yyyy HH:mm:ss", "date"))
  B. 1.dfDates = spark.createDataFrame([("23/01/2022 11:28:12",),("24/01/2022 10:58:34",)], ["date"]) 2.dfDates = dfDates.withColumnRenamed("date", to_timestamp("date", "yyyy-MM-ddHH:mm:ss"))
  C. 1.dfDates = spark.createDataFrame([("23/01/2022 11:28:12",),("24/01/2022 10:58:34",)], ["date"]) 2.dfDates = dfDates.withColumn("date", to_timestamp("date", "dd/MM/yyyy HH:mm:ss"))
  D. 1.dfDates = spark.createDataFrame(["23/01/2022 11:28:12","24/01/2022 10:58:34"], ["date"]) 2.dfDates = dfDates.withColumnRenamed("date", to_datetime("date", "yyyy-MM-ddHH:mm:ss"))
  E. 1.dfDates = spark.createDataFrame([("23/01/2022 11:28:12",),("24/01/2022 10:58:34",)], ["date"])
  C. 1.dfDates = spark.createDataFrame([("23/01/2022 11:28:12",),("24/01/2022 10:58:34",)], ["date"]) 2.dfDates = dfDates.withColumn("date", to_timestamp("date", "dd/MM/yyyy HH:mm:ss"))

• Question 140:

  Which of the following describes a difference between Spark's cluster and client execution modes?

  A. In cluster mode, the cluster manager resides on a worker node, while it resides on an edge node in client mode.
  B. In cluster mode, executor processes run on worker nodes, while they run on gateway nodes in client mode.
  C. In cluster mode, the driver resides on a worker node, while it resides on an edge node in client mode.
  D. In cluster mode, a gateway machine hosts the driver, while it is co-located with the executor in client mode.
  E. In cluster mode, the Spark driver is not co-located with the cluster manager, while it is co-located in client mode.
  C. In cluster mode, the driver resides on a worker node, while it resides on an edge node in client mode.