Free DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Web TestEngine Demo

Q51 Single choice

A data engineer is working with a dataset containing 2 billion rows distributed across 10 Spark partitions. The engineer needs to compute the approximate count of distinct users by their 'user_id' field and also calculate the average value of 'transaction_amount'. Both calculations must be done in a single transformation step to minimize shuffling.

Which set of Spark code will achieve this goal while ensuring performance?

A
from pyspark.sql.functions import approx_count_distinct, avg
df.agg (
approx_count_distinct ("user_id") .alias ("approx_count_distinct"), avg("transaction_amount").alias ("average_transaction")
).show()
B
from pyspark.sql.functions import count Distinct, avg
df.agg (
count Distinct ("user_id").alias ("approx_count_distinct"), avg("transaction_amount").alias ("average_transaction")
).show()
C
df.creareOrReplaceTempView("transactions")
spark.sql("""
SELECT -
COUNT(DISTINCT user_id) AS approx._count_distinct, AVG(transaction_amount) AS average_transaction
FROM transactions -
""").show()
D
from pyspark.sql.functions import col
approx._count = df.select ("user_id").distinct().count()
average_transaction = df.select("transaction_amount").groupBy().avg().collect() [0] [0] print(approx_count, average_transaction)

DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Web TestEngine demo