Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Online Questions & Answers

• Question 81:

  The code block displayed below contains an error. The code block should produce a DataFrame with color as the only column and three rows with color values of red, blue, and green, respectively.

  Find the error.

  Code block:

  1.spark.createDataFrame([("red",), ("blue",), ("green",)], "color")

  2.Instead of calling spark.createDataFrame, just DataFrame should be called.

  A. The commas in the tuples with the colors should be eliminated.
  B. The colors red, blue, and green should be expressed as a simple Python list, and not a list of tuples.
  C. Instead of color, a data type should be specified.
  D. The "color" expression needs to be wrapped in brackets, so it reads ["color"].
  D. The "color" expression needs to be wrapped in brackets, so it reads ["color"].

• Question 82:

  The code block displayed below contains an error. The code block should return a DataFrame in which column predErrorAdded contains the results of Python function add_2_if_geq_3 as applied to numeric and nullable column predError in DataFrame transactionsDf.

  Find the error.

  Code block:

  1.def add_2_if_geq_3(x):

  2.

  if x is None:

  3.

  return x

  4.

  elif x >= 3:

  5.

  return x+2

  6.

  return x

  7.

  8.add_2_if_geq_3_udf = udf(add_2_if_geq_3)

  9.

  10.transactionsDf.withColumnRenamed("predErrorAdded", add_2_if_geq_3_udf(col("predError")))

  A. The operator used to adding the column does not add column predErrorAdded to the DataFrame.
  B. Instead of col("predError"), the actual DataFrame with the column needs to be passed, like so transactionsDf.predError.
  C. The udf() method does not declare a return type.
  D. UDFs are only available through the SQL API, but not in the Python API as shown in the code block.
  E. The Python function is unable to handle null values, resulting in the code block crashing on execution.
  A. The operator used to adding the column does not add column predErrorAdded to the DataFrame.

  ---

  Correct code block:

  def add_2_if_geq_3(x):

  if x is None:

  return x

  elif x >= 3:

  return x+2

  return x

  add_2_if_geq_3_udf = udf(add_2_if_geq_3)

  transactionsDf.withColumn("predErrorAdded",

  add_2_if_geq_3_udf(col("predError"))).show()

  Instead of withColumnRenamed, you should use the withColumn operator.

  The udf() method does not declare a return type.

  It is fine that the udf() method does not declare a return type, this is not a required argument. However, the default return type is StringType. This may not be the ideal return type for numeric,

  nullable data ?but the code will run without specified return type nevertheless. The Python function is unable to handle null values, resulting in the code block crashing on execution.

  The Python function is able to handle null values, this is what the statement if x is None does.

  UDFs are only available through the SQL API, but not in the Python API as shown in the code block.

  No, they are available through the Python API. The code in the code block that concerns UDFs is correct.

  Instead of col("predError"), the actual DataFrame with the column needs to be passed, like so transactionsDf.predError.

  You may choose to use the transactionsDf.predError syntax, but the col("predError") syntax is fine.

• Question 83:

  The code block shown below should add column transactionDateForm to DataFrame transactionsDf. The column should express the unix-format timestamps in column transactionDate as string type like Apr 26 (Sunday). Choose the answer that correctly fills the blanks in the code block to accomplish this.

  transactionsDf.__1__(__2__, from_unixtime(__3__, __4__))

  A. 1. withColumn 2. "transactionDateForm" 3. "MMM d (EEEE)" 4. "transactionDate"
  B. 1. select 2. "transactionDate" 3. "transactionDateForm" 4. "MMM d (EEEE)"
  C. 1. withColumn 2. "transactionDateForm" 3. "transactionDate" 4. "MMM d (EEEE)"
  D. 1. withColumn 2. "transactionDateForm" 3. "transactionDate" 4. "MM d (EEE)"
  E. 1. withColumnRenamed 2. "transactionDate" 3. "transactionDateForm" 4. "MM d (EEE)"
  C. 1. withColumn 2. "transactionDateForm" 3. "transactionDate" 4. "MMM d (EEEE)"

• Question 84:

  Which of the following code blocks returns a single-column DataFrame of all entries in Python list throughputRates which contains only float-type values ?

  A. spark.createDataFrame((throughputRates), FloatType)
  B. spark.createDataFrame(throughputRates, FloatType)
  C. spark.DataFrame(throughputRates, FloatType)
  D. spark.createDataFrame(throughputRates)
  E. spark.createDataFrame(throughputRates, FloatType())
  E. spark.createDataFrame(throughputRates, FloatType())

  ---

  spark.createDataFrame(throughputRates, FloatType()) Correct! spark.createDataFrame is the correct operator to use here and the type FloatType() which is passed in for the command's schema argument is correctly instantiated using the

  parentheses.

  Remember that it is essential in PySpark to instantiate types when passing them to SparkSession.createDataFrame. And, in Databricks, spark returns a SparkSession object. spark.createDataFrame((throughputRates), FloatType) No. While

  packing throughputRates in parentheses does not do anything to the execution of this command, not instantiating the FloatType with parentheses as in the previous answer will make this

  command fail.

  spark.createDataFrame(throughputRates, FloatType)

  Incorrect. Given that it does not matter whether you pass throughputRates in parentheses or not, see the explanation of the previous answer for further insights.

  spark.DataFrame(throughputRates, FloatType)

  Wrong. There is no SparkSession.DataFrame() method in Spark.

  spark.createDataFrame(throughputRates)

  False. Avoiding the schema argument will have PySpark try to infer the schema. However, as you can see in the documentation (linked below), the inference will only work if you pass in an "RDD of

  either Row, namedtuple, or dict" for data (the first argument to createDataFrame). But since you are passing a Python list, Spark's schema inference will fail.

  More info: pyspark.sql.SparkSession.createDataFrame -- PySpark 3.1.2 documentation

  Static notebook | Dynamic notebook: See test 3, 55 (Databricks import instructions)

• Question 85:

  Which of the following code blocks returns a DataFrame with approximately 1,000 rows from the 10,000-row DataFrame itemsDf, without any duplicates, returning the same rows even if the code block is run twice?

  A. itemsDf.sampleBy("row", fractions={0: 0.1}, seed=82371)
  B. itemsDf.sample(fraction=0.1, seed=87238)
  C. itemsDf.sample(fraction=1000, seed=98263)
  D. itemsDf.sample(withReplacement=True, fraction=0.1, seed=23536)
  E. itemsDf.sample(fraction=0.1)
  B. itemsDf.sample(fraction=0.1, seed=87238)

  ---

  itemsDf.sample(fraction=0.1, seed=87238)

  Correct. If itemsDf has 10,000 rows, this code block returns about 1,000, since DataFrame.sample() is never guaranteed to return an exact amount of rows. To ensure you are not returning

  duplicates, you should leave the withReplacement parameter at False, which is the default. Since the specifies that the same rows should be returned even if the code block is run

  twice,

  you need to specify a seed. The number passed in the seed does not matter as long as it is an integer.

  itemsDf.sample(withReplacement=True, fraction=0.1, seed=23536) Incorrect. While this code block fulfills almost all requirements, it may return duplicates.

  This is because withReplacement is set to True.

  Here is how to understand what replacement means: Imagine you have a bucket of 10,000 numbered balls and you need to take 1,000 balls at random from the bucket (similar to the problem in the

  question). Now, if you would take those balls with replacement, you would take a ball, note its number, and put it back into the bucket, meaning the next time you take a ball from the bucket there

  would be a chance you could take the exact same ball again. If you took the balls without replacement, you would leave the ball outside the bucket and not put it back in as you take the next 999

  balls.

  itemsDf.sample(fraction=1000, seed=98263)

  Wrong. The fraction parameter needs to have a value between 0 and 1. In this case, it should be 0.1, since 1,000/10,000 = 0.1.

  itemsDf.sampleBy("row", fractions={0: 0.1}, seed=82371) No, DataFrame.sampleBy() is meant for stratified sampling. This means that based on the values in a column in a DataFrame, you can draw a certain fraction of rows containing those

  values from

  the DataFrame (more details linked below). In the scenario at hand, sampleBy is not the right operator to use because you do not have any information about any column that the sampling should

  depend on.

  itemsDf.sample(fraction=0.1)

  Incorrect. This code block checks all the boxes except that it does not ensure that when you run it a second time, the exact same rows will be returned. In order to achieve this, you would have to

  specify a seed.

  More info:

  -pyspark.sql.DataFrame.sample -- PySpark 3.1.2 documentation

  -pyspark.sql.DataFrame.sampleBy -- PySpark 3.1.2 documentation

  - Types of Samplings in PySpark 3. The explanations of the sampling... | by Pinar Ersoy | Towards Data Science

• Question 86:

  Which of the following code blocks performs an inner join between DataFrame itemsDf and DataFrame transactionsDf, using columns itemId and transactionId as join keys, respectively?

  A. itemsDf.join(transactionsDf, "inner", itemsDf.itemId == transactionsDf.transactionId)
  B. itemsDf.join(transactionsDf, itemId == transactionId)
  C. itemsDf.join(transactionsDf, itemsDf.itemId == transactionsDf.transactionId, "inner")
  D. itemsDf.join(transactionsDf, "itemsDf.itemId == transactionsDf.transactionId", "inner")
  E. itemsDf.join(transactionsDf, col(itemsDf.itemId) == col(transactionsDf.transactionId))
  C. itemsDf.join(transactionsDf, itemsDf.itemId == transactionsDf.transactionId, "inner")

  ---

  More info: pyspark.sql.DataFrame.join -- PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2, 27 (Databricks import instructions)

• Question 87:

  Which of the following code blocks returns the number of unique values in column storeId of DataFrame transactionsDf?

  A. transactionsDf.select("storeId").dropDuplicates().count()
  B. transactionsDf.select(count("storeId")).dropDuplicates()
  C. transactionsDf.select(distinct("storeId")).count()
  D. transactionsDf.dropDuplicates().agg(count("storeId"))
  E. transactionsDf.distinct().select("storeId").count()
  A. transactionsDf.select("storeId").dropDuplicates().count()

  ---

  transactionsDf.select("storeId").dropDuplicates().count() Correct! After dropping all duplicates from column storeId, the remaining rows get counted, representing the number of unique values in the column. transactionsDf.select(count ("storeId")).dropDuplicates() No. transactionsDf.select(count("storeId")) just returns a single-row DataFrame showing the number of non-null rows. dropDuplicates() does not have any effect in this context.

  transactionsDf.dropDuplicates().agg(count("storeId")) Incorrect. While transactionsDf.dropDuplicates() removes duplicate rows from transactionsDf, it does not do so taking only column storeId into consideration, but eliminates full row duplicates instead. transactionsDf.distinct().select("storeId").count() Wrong. transactionsDf.distinct() identifies unique rows across all columns, but not only unique rows with respect to column storeId. This may leave duplicate values in the column, making the count not represent the number of unique values in that column. transactionsDf.select(distinct("storeId")).count() False. There is no distinct method in pyspark.sql.functions.

• Question 88:

  The code block shown below should return a one-column DataFrame where the column storeId is converted to string type. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  transactionsDf.__1__(__2__.__3__(__4__))

  A. 1. select 2. col("storeId") 3. cast 4. StringType
  B. 1. select 2. col("storeId") 3. as 4. StringType
  C. 1. cast 2. "storeId" 3. as 4. StringType()
  D. 1. select 2. col("storeId") 3. cast 4. StringType()
  E. 1. select 2. storeId 3. cast 4. StringType()
  D. 1. select 2. col("storeId") 3. cast 4. StringType()

  ---

  Correct code block: transactionsDf.select(col("storeId").cast(StringType())) Solving this involves understanding that, when using types from the pyspark.sql.types such as StringType, these types need to be instantiated when using them in Spark, or, in simple words, they need to be followed by parentheses like so: StringType(). You could also use .cast("string") instead, but that option is not given here. More info: pyspark.sql.Column.cast -- PySpark 3.1.2 documentation

  Static notebook | Dynamic notebook: See test 2, 37 (Databricks import instructions)

• Question 89:

  The code block shown below should return a two-column DataFrame with columns transactionId and supplier, with combined information from DataFrames itemsDf and transactionsDf. The code block should merge rows in which column productId of DataFrame transactionsDf matches the value of column itemId in DataFrame itemsDf, but only where column storeId of DataFrame

  transactionsDf does not match column itemId of DataFrame itemsDf. Choose the answer that correctly fills the blanks in the code block to accomplish this.

  Code block:

  transactionsDf.__1__(itemsDf, __2__).__3__(__4__)

  A. 1. join 2. transactionsDf.productId==itemsDf.itemId, how="inner" 3. select 4. "transactionId", "supplier"
  B. 1. select 2. "transactionId", "supplier" 3. join 4. [transactionsDf.storeId!=itemsDf.itemId, transactionsDf.productId==itemsDf.itemId]
  C. 1. join 2. [transactionsDf.productId==itemsDf.itemId, transactionsDf.storeId!=itemsDf.itemId] 3. select 4. "transactionId", "supplier"
  D. 1. filter 2. "transactionId", "supplier" 3. join 4. "transactionsDf.storeId!=itemsDf.itemId, transactionsDf.productId==itemsDf.itemId"
  E. 1. join 2. transactionsDf.productId==itemsDf.itemId, transactionsDf.storeId!=itemsDf.itemId 3. filter 4. "transactionId", "supplier"
  C. 1. join 2. [transactionsDf.productId==itemsDf.itemId, transactionsDf.storeId!=itemsDf.itemId] 3. select 4. "transactionId", "supplier"

  ---

  This is pretty complex and, in its complexity, is probably above what you would encounter in the exam. However, reading the carefully, you can use your logic skills to weed out the wrong answers here. First, you should examine the join statement which is common to all answers. The first argument of the join() operator (documentation linked below) is the DataFrame to be joined with. Where join is in gap 3, the first argument of gap 4 should therefore be another DataFrame. For none of the questions where join is in the third gap, this is the case. So you can immediately discard two answers. For all other answers, join is in gap 1, followed by .(itemsDf, according to the code block. Given how the join() operator is called, there are now three remaining candidates. Looking further at the join() statement, the second argument (on=) expects "a string for the join column name, a list of column names, a join expression (Column), or a list of Columns", according to the documentation. As one answer option includes a list of join expressions (transactionsDf.productId==itemsDf.itemId, transactionsDf.storeId!=itemsDf.itemId) which is unsupported according to the documentation, we can discard that answer, leaving us with two remaining candidates. Both candidates have valid syntax, but only one of them fulfills the condition in the "only where column storeId of DataFrame transactionsDf does not match column itemId of DataFrame itemsDf". So, this one remaining answer option has to be the correct one! As you can see, although sometimes overwhelming at first, even more complex questions can be figured out by rigorously applying the knowledge you can gain from the documentation during the exam.

  More info: pyspark.sql.DataFrame.join -- PySpark 3.1.2 documentation

  Static notebook | Dynamic notebook: See test 3, 47 (Databricks import instructions)

• Question 90:

  Which of the following code blocks reads in parquet file /FileStore/imports.parquet as a DataFrame?

  A. spark.mode("parquet").read("/FileStore/imports.parquet")
  B. spark.read.path("/FileStore/imports.parquet", source="parquet")
  C. spark.read().parquet("/FileStore/imports.parquet")
  D. spark.read.parquet("/FileStore/imports.parquet")
  E. spark.read().format('parquet').open("/FileStore/imports.parquet")
  D. spark.read.parquet("/FileStore/imports.parquet")