Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Questions & Answers

• Question 151:

  Which of the following code blocks adds a column predErrorSqrt to DataFrame transactionsDf that is the square root of column predError?

  A. transactionsDf.withColumn("predErrorSqrt", sqrt(predError))
  B. transactionsDf.select(sqrt(predError))
  C. transactionsDf.withColumn("predErrorSqrt", col("predError").sqrt())
  D. transactionsDf.withColumn("predErrorSqrt", sqrt(col("predError")))
  E. transactionsDf.select(sqrt("predError"))
  D. transactionsDf.withColumn("predErrorSqrt", sqrt(col("predError")))

  ---

  transactionsDf.withColumn("predErrorSqrt", sqrt(col("predError"))) Correct. The DataFrame.withColumn() operator is used to add a new column to a DataFrame. It takes two arguments: The name of the new column (here: predErrorSqrt) and a Column expression as the new column. In PySpark, a Column expression means referring to a column using the col("predError") command or by other means, for example by transactionsDf.predError, or even just using the column name as a string, "predError". The asks for the square root. sqrt() is a function in pyspark.sql.functions and calculates the square root. It takes a value or a Column as an input. Here it is the predError column of DataFrame transactionsDf expressed through col("predError"). transactionsDf.withColumn("predErrorSqrt", sqrt(predError)) Incorrect. In this expression, sqrt(predError) is incorrect syntax. You cannot refer to predError in this way ?to Spark it looks as if you are trying to refer to the non-existent Python variable predError. You could pass transactionsDf.predError, col("predError") (as in the correct solution), or even just "predError" instead.

  transactionsDf.select(sqrt(predError))

  Wrong. Here, the explanation just above this one about how to refer to predError applies.

  transactionsDf.select(sqrt("predError"))

  No. While this is correct syntax, it will return a single-column DataFrame only containing a column showing the square root of column predError. However, the asks for a column to

  be added to the original DataFrame transactionsDf. transactionsDf.withColumn("predErrorSqrt", col("predError").sqrt()) No. The issue with this statement is that column col("predError") has no sqrt() method. sqrt() is a member of

  pyspark.sql.functions, but not of pyspark.sql.Column. More info: pyspark.sql.DataFrame.withColumn -- PySpark 3.1.2 documentation and pyspark.sql.functions.sqrt -- PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See

  test 2, 31 (Databricks import instructions)

• Question 152:

  Which of the following code blocks reads in the JSON file stored at filePath as a DataFrame?

  A. spark.read.json(filePath)
  B. spark.read.path(filePath, source="json")
  C. spark.read().path(filePath)
  D. spark.read().json(filePath)
  E. spark.read.path(filePath)
  A. spark.read.json(filePath)

• Question 153:

  The code block displayed below contains an error. The code block should configure Spark to split data in 20 parts when exchanging data between executors for joins or aggregations.

  Find the error.

  Code block:

  spark.conf.set(spark.sql.shuffle.partitions, 20)

  A. The code block uses the wrong command for setting an option.
  B. The code block sets the wrong option.
  C. The code block expresses the option incorrectly.
  D. The code block sets the incorrect number of parts.
  E. The code block is missing a parameter.
  C. The code block expresses the option incorrectly.

• Question 154:

  The code block shown below should convert up to 5 rows in DataFrame transactionsDf that have the value 25 in column storeId into a Python list. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  Code block:

  transactionsDf.__1__(__2__).__3__(__4__)

  A. 1. filter 2. "storeId"==25 3. collect 4. 5
  B. 1. filter 2. col("storeId")==25 3. toLocalIterator 4. 5
  C. 1. select 2. storeId==25 3. head 4. 5
  D. 1. filter 2. col("storeId")==25 3. take 4. 5
  E. 1. filter 2. col("storeId")==25 3. collect 4. 5
  D. 1. filter 2. col("storeId")==25 3. take 4. 5

• Question 155:

  The code block shown below should return a DataFrame with all columns of DataFrame transactionsDf, but only maximum 2 rows in which column productId has at least the value 2. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  transactionsDf.__1__(__2__).__3__

  A. 1. where 2. "productId" > 2 3. max(2)
  B. 1. where 2. transactionsDf[productId] >= 2 3. limit(2)
  C. 1. filter 2. productId > 2 3. max(2)
  D. 1. filter 2. col("productId") >= 2 3. limit(2)
  E. 1. where 2. productId >= 2 3. limit(2)
  D. 1. filter 2. col("productId") >= 2 3. limit(2)

  ---

  Correct code block:

  transactionsDf.filter(col("productId") >= 2).limit(2) The filter and where operators in gap 1 are just aliases of one another, so you cannot use them to pick the right answer.

  The column definition in gap 2 is more helpful. The DataFrame.filter() method takes an argument of type Column or str. From all possible answers, only the one including col("productId") >= 2 fits

  this profile, since it returns a Column type.

  The answer option using "productId" > 2 is invalid, since Spark does not understand that "productId" refers to column productId. The answer option using transactionsDf[productId] >= 2 is wrong

  because you cannot refer to a column using square bracket notation in Spark (if you are coming from Python using Pandas, this is something to watch out for). In all other options, productId is being

  referred to as a Python variable, so they are relatively easy to eliminate. Also note that the asks for the value in column productId being at least 2. This translates to a "greater or equal" sign (>= 2), but not a "greater" sign (> 2). Another thing

  worth noting is that there is no DataFrame.max() method. If you picked any option including this, you may be confusing it with the pyspark.sql.functions.max method.

  The correct

  method to limit the amount of rows is the DataFrame.limit() method.

  More info:

  -pyspark.sql.DataFrame.filter -- PySpark 3.1.2 documentation

  -pyspark.sql.DataFrame.limit -- PySpark 3.1.2 documentation

  Static notebook | Dynamic notebook: See test 3, 54 (Databricks import instructions)

• Question 156:

  Which of the following code blocks removes all rows in the 6-column DataFrame transactionsDf that have missing data in at least 3 columns?

  A. transactionsDf.dropna("any")
  B. transactionsDf.dropna(thresh=4)
  C. transactionsDf.drop.na("",2)
  D. transactionsDf.dropna(thresh=2)
  E. transactionsDf.dropna("",4)
  B. transactionsDf.dropna(thresh=4)

• Question 157:

  The code block displayed below contains an error. The code block should display the schema of DataFrame transactionsDf. Find the error.

  Code block:

  transactionsDf.rdd.printSchema

  A. There is no way to print a schema directly in Spark, since the schema can be printed easily through using print(transactionsDf.columns), so that should be used instead.
  B. The code block should be wrapped into a print() operation.
  C. PrintSchema is only accessible through the spark session, so the code block should be rewritten as spark.printSchema(transactionsDf).
  D. PrintSchema is a method and should be written as printSchema(). It is also not callable through transactionsDf.rdd, but should be called directly from transactionsDf.
  E. PrintSchema is a not a method of transactionsDf.rdd. Instead, the schema should be printed via transactionsDf.print_schema().
  D. PrintSchema is a method and should be written as printSchema(). It is also not callable through transactionsDf.rdd, but should be called directly from transactionsDf.

  ---

  Correct code block:

  transactionsDf.printSchema()

  This is more of a knowledge that you should just memorize or look up in the provided documentation during the exam. You can get more info about DataFrame.printSchema() in the

  documentation (link below). However - it is a plain simple method without any arguments. One answer points to an alternative of printing the schema: You could also use print(transactionsDf.schema). This will give you readable, but not nicely

  formatted, description of the schema.

  More info: pyspark.sql.DataFrame.printSchema -- PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1, 54 (Databricks import instructions)

• Question 158:

  The code block displayed below contains one or more errors. The code block should load parquet files at location filePath into a DataFrame, only loading those files that have been modified before 2029-03-20 05:44:46. Spark should enforce a schema according to the schema shown below. Find the error.

  Schema:

  1.root

  2.

  |-- itemId: integer (nullable = true)

  3.

  |-- attributes: array (nullable = true)

  4.

  | |-- element: string (containsNull = true)

  5.

  |-- supplier: string (nullable = true)

  Code block:

  1.schema = StructType([

  2.

  StructType("itemId", IntegerType(), True),

  3.

  StructType("attributes", ArrayType(StringType(), True), True),

  4.

  StructType("supplier", StringType(), True)

  5.])

  6.

  7.spark.read.options("modifiedBefore", "2029-03-20T05:44:46").schema(schema).load(filePath)

  A. The attributes array is specified incorrectly, Spark cannot identify the file format, and the syntax of the call to Spark's DataFrameReader is incorrect.
  B. Columns in the schema definition use the wrong object type and the syntax of the call to Spark's DataFrameReader is incorrect.
  C. The data type of the schema is incompatible with the schema() operator and the modification date threshold is specified incorrectly.
  D. Columns in the schema definition use the wrong object type, the modification date threshold is specified incorrectly, and Spark cannot identify the file format.
  E. Columns in the schema are unable to handle empty values and the modification date threshold is specified incorrectly.
  D. Columns in the schema definition use the wrong object type, the modification date threshold is specified incorrectly, and Spark cannot identify the file format.

• Question 159:

  The code block displayed below contains an error. The code block should arrange the rows of DataFrame transactionsDf using information from two columns in an ordered fashion, arranging first by column value, showing smaller numbers at the top and greater numbers at the bottom, and then by column predError, for which all values should be arranged in the inverse way of the order of items in column value. Find the error.

  Code block:

  transactionsDf.orderBy('value', asc_nulls_first(col('predError')))

  A. Two orderBy statements with calls to the individual columns should be chained, instead of having both columns in one orderBy statement.
  B. Column value should be wrapped by the col() operator.
  C. Column predError should be sorted in a descending way, putting nulls last.
  D. Column predError should be sorted by desc_nulls_first() instead.
  E. Instead of orderBy, sort should be used.
  C. Column predError should be sorted in a descending way, putting nulls last.

• Question 160:

  The code block displayed below contains an error. The code block should return a new DataFrame that only contains rows from DataFrame transactionsDf in which the value in column predError is at least 5.

  Find the error.

  Code block:

  transactionsDf.where("col(predError) >= 5")

  A. The argument to the where method should be "predError >= 5".
  B. Instead of where(), filter() should be used.
  C. The expression returns the original DataFrame transactionsDf and not a new DataFrame. To avoid this, the code block should be transactionsDf.toNewDataFrame().where("col(predError) >= 5").
  D. The argument to the where method cannot be a string.
  E. Instead of >=, the SQL operator GEQ should be used.
  A. The argument to the where method should be "predError >= 5".