Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Questions & Answers

• Question 121:

  The code block displayed below contains an error. The code block should create DataFrame itemsAttributesDf which has columns itemId and attribute and lists every attribute from the attributes column in DataFrame itemsDf next to the itemId of the respective row in itemsDf. Find the error.

  A sample of DataFrame itemsDf is below.

  

  Code block:

  itemsAttributesDf = itemsDf.explode("attributes").alias("attribute").select("attribute", "itemId")

  A. Since itemId is the index, it does not need to be an argument to the select() method.
  B. The alias() method needs to be called after the select() method.
  C. The explode() method expects a Column object rather than a string.
  D. explode() is not a method of DataFrame. explode() should be used inside the select() method instead.
  E. The split() method should be used inside the select() method instead of the explode() method.
  D. explode() is not a method of DataFrame. explode() should be used inside the select() method instead.

  ---

  The correct code block looks like this:

  

  Then, the first couple of rows of itemAttributesDf look like this:

  

  C:\Users\Admin\Desktop\Data\Odt data\Untitled.jpg

  explode() is not a method of DataFrame. explode() should be used inside the select() method instead.

  This is correct.

  The split() method should be used inside the select() method instead of the explode() method.

  No, the split() method is used to split strings into parts. However, column attributs is an array of strings. In this case, the explode() method is appropriate. Since itemId is the index, it does not need to be an argument to the select() method. No,

  itemId still needs to be selected, whether it is used as an index or not. The explode() method expects a Column object rather than a string. No, a string works just fine here. This being said, there are some valid alternatives to passing in a

  string:

  

  The alias() method needs to be called after the select() method.

  No.

  More info: pyspark.sql.functions.explode -- PySpark 3.1.1 documentation (https://bit.ly/2QUZI1J)

  Static notebook | Dynamic notebook: See test 1, 22 (Databricks import instructions) (https://flrs.github.io/spark_practice_tests_code/#1/22.html , https://bit.ly/sparkpracticeexams_import_instructions)

• Question 122:

  Which of the following describes the difference between client and cluster execution modes?

  A. In cluster mode, the driver runs on the worker nodes, while the client mode runs the driver on the client machine.
  B. In cluster mode, the driver runs on the edge node, while the client mode runs the driver in a worker node.
  C. In cluster mode, each node will launch its own executor, while in client mode, executors will exclusively run on the client machine.
  D. In client mode, the cluster manager runs on the same host as the driver, while in cluster mode, the cluster manager runs on a separate node.
  E. In cluster mode, the driver runs on the master node, while in client mode, the driver runs on a virtual machine in the cloud.
  A. In cluster mode, the driver runs on the worker nodes, while the client mode runs the driver on the client machine.

• Question 123:

  Which of the following code blocks returns a copy of DataFrame transactionsDf in which column productId has been renamed to productNumber?

  A. transactionsDf.withColumnRenamed("productId", "productNumber")
  B. transactionsDf.withColumn("productId", "productNumber")
  C. transactionsDf.withColumnRenamed("productNumber", "productId")
  D. transactionsDf.withColumnRenamed(col(productId), col(productNumber))
  E. transactionsDf.withColumnRenamed(productId, productNumber)
  A. transactionsDf.withColumnRenamed("productId", "productNumber")

  ---

  More info: pyspark.sql.DataFrame.withColumnRenamed -- PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2, 35 (Databricks import instructions)

• Question 124:

  The code block shown below should return a single-column DataFrame with a column named consonant_ct that, for each row, shows the number of consonants in column itemName of DataFrame

  itemsDf. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  DataFrame itemsDf:

  1.+------+----------------------------------+-----------------------------+-------------------+

  2.|itemId|itemName |attributes |supplier |

  3.+------+----------------------------------+-----------------------------+-------------------+

  4.|1 |Thick Coat for Walking in the Snow|[blue, winter, cozy] |Sports Company Inc.|

  5.|2 |Elegant Outdoors Summer Dress |[red, summer, fresh, cooling]|YetiX |

  6.|3 |Outdoors Backpack |[green, summer, travel] |Sports Company Inc.|

  7.+------+----------------------------------+-----------------------------+-------------------+

  Code block:

  itemsDf.select(__1__(__2__(__3__(__4__), "a|e|i|o|u|\s", "")).__5__("consonant_ct"))

  A. 1. length 2. regexp_extract 3. upper 4. col("itemName") 5. as
  B. 1. size 2. regexp_replace 3. lower 4. "itemName" 5. alias
  C. 1. lower 2. regexp_replace 3. length 4. "itemName" 5. alias
  D. 1. length 2. regexp_replace 3. lower 4. col("itemName") 5. alias
  E. 1. size 2. regexp_extract 3. lower 4. col("itemName") 5. alias
  D. 1. length 2. regexp_replace 3. lower 4. col("itemName") 5. alias

  ---

  Correct code block:

  itemsDf.select(length(regexp_replace(lower(col("itemName")), "a|e|i|o|u|\s", "")).alias("consonant_ct"))

  Returned DataFrame:

  +------------+

  |consonant_ct|

  +------------+

  | 19|

  | 16|

  | 10|

  +------------+

  This tries to make you think about the string functions Spark provides and in which order they should be applied. Arguably the most difficult part, the regular expression "a|e|i|o|u|

  \s", is not a numbered blank. However, if you are not familiar with the string functions, it may be a good idea to review those before the exam. The size operator and the length operator can easily be confused. size works on arrays, while

  length works on strings. Luckily, this is something you can read up about in the documentation.

  The code block works by first converting all uppercase letters in column itemName into lowercase (the lower() part). Then, it replaces all vowels by "nothing" - an empty character "" (the

  regexp_replace() part). Now, only lowercase characters without spaces are included in the DataFrame. Then, per row, the length operator counts these remaining characters. Note that column

  itemName in itemsDf does not include any numbers or other characters, so we do not need to make any provisions for these. Finally, by using the alias() operator, we rename the resulting column to

  consonant_ct.

  More info:

  -

  lower: pyspark.sql.functions.lower -- PySpark 3.1.2 documentation

  -

  regexp_replace: pyspark.sql.functions.regexp_replace -- PySpark 3.1.2 documentation

  -

  length: pyspark.sql.functions.length -- PySpark 3.1.2 documentation

  -

  alias: pyspark.sql.Column.alias -- PySpark 3.1.2 documentation

  Static notebook | Dynamic notebook: See test 2, 51 (Databricks import instructions)

• Question 125:

  Which of the following statements about RDDs is incorrect?

  A. An RDD consists of a single partition.
  B. The high-level DataFrame API is built on top of the low-level RDD API.
  C. RDDs are immutable.
  D. RDD stands for Resilient Distributed Dataset.
  E. RDDs are great for precisely instructing Spark on how to do a query.
  A. An RDD consists of a single partition.

• Question 126:

  Which of the following code blocks returns a DataFrame that matches the multi-column DataFrame itemsDf, except that integer column itemId has been converted into a string column?

  A. itemsDf.withColumn("itemId", convert("itemId", "string"))
  B. itemsDf.withColumn("itemId", col("itemId").cast("string"))
  C. itemsDf.select(cast("itemId", "string"))
  D. itemsDf.withColumn("itemId", col("itemId").convert("string"))
  E. spark.cast(itemsDf, "itemId", "string")
  B. itemsDf.withColumn("itemId", col("itemId").cast("string"))

  ---

  itemsDf.withColumn("itemId", col("itemId").cast("string")) Correct. You can convert the data type of a column using the cast method of the Column class. Also note that you will have to use the withColumn method on itemsDf for replacing the existing itemId column with the new version that contains strings. itemsDf.withColumn("itemId", col("itemId").convert("string")) Incorrect. The Column object that col("itemId") returns does not have a convert method. itemsDf.withColumn("itemId", convert ("itemId", "string")) Wrong. Spark's spark.sql.functions module does not have a convert method. The is trying to mislead you by using the word "converted". Type conversion is also called "type casting". This may help you remember to look for a cast method instead of a convert method (see correct answer). itemsDf.select(astype("itemId", "string")) False. While astype is a method of Column (and an alias of Column.cast), it is not a method of pyspark.sql.functions (what the code block implies). In addition, the

• Question 127:

  The code block shown below should write DataFrame transactionsDf to disk at path csvPath as a single CSV file, using tabs (\t characters) as separators between columns, expressing missing

  values as string n/a, and omitting a header row with column names. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  transactionsDf.__1__.write.__2__(__3__, " ").__4__.__5__(csvPath)

  A. 1. coalesce(1) 2. option 3. "sep" 4. option("header", True) 5. path
  B. 1. coalesce(1) 2. option 3. "colsep" 4. option("nullValue", "n/a") 5. path
  C. 1. repartition(1) 2. option 3. "sep" 4. option("nullValue", "n/a") 5. csv
  D. 1. csv 2. option 3. "sep" 4. option("emptyValue", "n/a") 5. path ? 1. repartition(1) 2. mode 3. "sep" 4. mode("nullValue", "n/a") 5. csv
  C. 1. repartition(1) 2. option 3. "sep" 4. option("nullValue", "n/a") 5. csv

  ---

  Correct code block:

  transactionsDf.repartition(1).write.option("sep", "\t").option("nullValue", "n/a").csv(csvPath) It is important here to understand that the specifically asks for writing the DataFrame as a single CSV file. This should trigger you to think about

  partitions. By default, every partition

  is written as a separate file, so you need to include repatition(1) into your call. coalesce(1) works here, too!

  Secondly, the is very much an invitation to search through the parameters in the Spark documentation that work with DataFrameWriter.csv (link below). You will also need to know

  that you need an option() statement to apply these parameters. The final concern is about the general call structure. Once you have called accessed write of your DataFrame, options follow and then you write the DataFrame with csv. Instead

  of csv(csvPath),

  you could also use save(csvPath, format='csv') here. More info: pyspark.sql.DataFrameWriter.csv -- PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1, 52 (Databricks import instructions)

• Question 128:

  The code block shown below should return only the average prediction error (column predError) of a random subset, without replacement, of approximately 15% of rows in DataFrame transactionsDf. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  transactionsDf.__1__(__2__, __3__).__4__(avg('predError'))

  A. 1. sample 2. True 3. 0.15 4. filter
  B. 1. sample 2. False 3. 0.15 4. select
  C. 1. sample 2. 0.85 3. False 4. select
  D. 1. fraction 2. 0.15 3. True 4. where
  E. 1. fraction 2. False 3. 0.85 4. select
  B. 1. sample 2. False 3. 0.15 4. select

  ---

  Correct code block: transactionsDf.sample(withReplacement=False, fraction=0.15).select(avg('predError')) You should remember that getting a random subset of rows means sampling. This, in turn should point you to the DataFrame.sample() method. Once you know this, you can look up the correct order of arguments in the documentation (link below). Lastly, you have to decide whether to use filter, where or select. where is just an alias for filter(). filter() is not the correct method to use here, since it would only allow you to filter rows based on some condition. However, the asks to return only the average prediction error. You can control the columns that a query returns with the select() method ?so this is the correct method to use here. More info: pyspark.sql.DataFrame.sample -- PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2, 28 (Databricks import instructions)

• Question 129:

  Which of the following code blocks returns a one-column DataFrame for which every row contains an array of all integer numbers from 0 up to and including the number given in column predError of DataFrame transactionsDf, and null if predError is null?

  Sample of DataFrame transactionsDf:

  1.+-------------+---------+-----+-------+---------+----+

  2.|transactionId|predError|value|storeId|productId| f|

  3.+-------------+---------+-----+-------+---------+----+

  4.| 1| 3| 4| 25| 1|null|

  5.| 2| 6| 7| 2| 2|null|

  6.| 3| 3| null| 25| 3|null|

  7.| 4| null| null| 3| 2|null|

  8.| 5| null| null| null| 2|null|

  9.| 6| 3| 2| 25| 2|null|

  10.+-------------+---------+-----+-------+---------+----+

  A. 1.def count_to_target(target): 2. if target is None: 3. return 4. 5. result = [range(target)] 6. return result 7. 8.count_to_target_udf = udf(count_to_target, ArrayType[IntegerType]) 9. 10.transactionsDf.select(count_to_target_udf(col('predError')))
  B. 1.def count_to_target(target): 2. if target is None: 3. return 4. 5. result = list(range(target)) 6. return result 7. 8.transactionsDf.select(count_to_target(col('predError')))
  C. 1.def count_to_target(target): 2. if target is None: 3. return 4. 5. result = list(range(target)) 6. return result 7. 8.count_to_target_udf = udf(count_to_target, ArrayType(IntegerType())) 9. 10.transactionsDf.select(count_to_target_udf('predError')) (Correct)
  D. 1.def count_to_target(target): 2. result = list(range(target)) 3. return result 4. 5.count_to_target_udf = udf(count_to_target, ArrayType(IntegerType())) 6. 7.df = transactionsDf.select(count_to_target_udf('predError'))
  E. 1.def count_to_target(target): 2. if target is None: 3. return 4. 5. result = list(range(target)) 6. return result 7. 8.count_to_target_udf = udf(count_to_target) 9. 10.transactionsDf.select(count_to_target_udf('predError'))
  C. 1.def count_to_target(target): 2. if target is None: 3. return 4. 5. result = list(range(target)) 6. return result 7. 8.count_to_target_udf = udf(count_to_target, ArrayType(IntegerType())) 9. 10.transactionsDf.select(count_to_target_udf('predError')) (Correct)

• Question 130:

  Which of the following code blocks returns a copy of DataFrame itemsDf where the column supplier has been renamed to manufacturer?

  A. itemsDf.withColumn(["supplier", "manufacturer"])
  B. itemsDf.withColumn("supplier").alias("manufacturer")
  C. itemsDf.withColumnRenamed("supplier", "manufacturer")
  D. itemsDf.withColumnRenamed(col("manufacturer"), col("supplier"))
  E. itemsDf.withColumnsRenamed("supplier", "manufacturer")
  C. itemsDf.withColumnRenamed("supplier", "manufacturer")

  ---

  itemsDf.withColumnRenamed("supplier", "manufacturer") Correct! This uses the relatively trivial DataFrame method withColumnRenamed for renaming column supplier to column manufacturer.

  Note that the asks for "a copy of DataFrame itemsDf". This may be confusing if you are not familiar with Spark yet. RDDs (Resilient Distributed Datasets) are the foundation of

  Spark DataFrames and are immutable. As such, DataFrames are immutable, too. Any command that changes anything in the DataFrame therefore necessarily returns a copy, or a new version, of it

  that has the changes applied.

  itemsDf.withColumnsRenamed("supplier", "manufacturer") Incorrect. Spark's DataFrame API does not have a withColumnsRenamed() method. itemsDf.withColumnRenamed(col("manufacturer"), col("supplier")) No. Watch out ?although the

  col() method works for many methods of the DataFrame API, withColumnRenamed is not one of them. As outlined in the documentation linked below, withColumnRenamed expects strings.

  itemsDf.withColumn(["supplier", "manufacturer"])

  Wrong. While DataFrame.withColumn() exists in Spark, it has a different purpose than renaming columns. withColumn is typically used to add columns to DataFrames, taking the name of the new

  column as a first, and a Column as a second argument. Learn more via the documentation that is linked below.

  itemsDf.withColumn("supplier").alias("manufacturer") No. While DataFrame.withColumn() exists, it requires 2 arguments. Furthermore, the alias() method on DataFrames would not help the cause of renaming a column much.

  DataFrame.alias() can be

  useful in addressing the input of join statements. However, this is far outside of the scope of this question. If you are curious nevertheless, check out the link below. More info: pyspark.sql.DataFrame.withColumnRenamed -- PySpark 3.1.1

  documentation, pyspark.sql.DataFrame.withColumn -- PySpark 3.1.1 documentation, and pyspark.sql.DataFrame.alias --

  PySpark 3.1.2 documentation (https://bit.ly/3aSB5tm , https://bit.ly/2Tv4rbE , https://bit.ly/2RbhBd2)

  Static notebook | Dynamic notebook: See test 1, 31 (Databricks import instructions) (https://flrs.github.io/spark_practice_tests_code/#1/31.html , https://bit.ly/sparkpracticeexams_import_instructions)