Databricks Databricks Certifications DATABRICKS-CERTIFIED-DATA-ENGINEER-ASSOCIATE Questions & Answers

• Question 1:

  Which of the following tools is used by Auto Loader process data incrementally?

  A. Checkpointing

  B. Spark Structured Streaming

  C. Data Explorer

  D. Unity Catalog

  E. Databricks SQL

  Correct Answer: B

  Explanation: The Auto Loader process in Databricks is typically used in conjunction with Spark Structured Streaming to process data incrementally. Spark Structured Streaming is a real-time data processing framework that allows you to process data streams incrementally as new data arrives. The Auto Loader is a feature in Databricks that works with Structured Streaming to automatically detect and process new data files as they are added to a specified data source location. It allows for incremental data processing without the need for manual intervention. How does Auto Loader track ingestion progress? As files are discovered, their metadata is persisted in a scalable key-value store (RocksDB) in the checkpoint location of your Auto Loader pipeline. This key-value store ensures that data is processed exactly once. In case of failures, Auto Loader can resume from where it left off by information stored in the checkpoint location and continue to provide exactly-once guarantees when writing data into Delta Lake. You don't need to maintain or manage any state yourself to achieve fault tolerance or exactly-once semantics.https://docs.databricks.com/ingestion/auto- loader/index.html

• Question 2:

  Which of the following statements regarding the relationship between Silver tables and Bronze tables is always true?

  A. Silver tables contain a less refined, less clean view of data than Bronze data.

  B. Silver tables contain aggregates while Bronze data is unaggregated.

  C. Silver tables contain more data than Bronze tables.

  D. Silver tables contain a more refined and cleaner view of data than Bronze tables.

  E. Silver tables contain less data than Bronze tables.

  Correct Answer: D

  Explanation: https://www.databricks.com/glossary/medallion-architecture

• Question 3:

  Which of the following benefits is provided by the array functions from Spark SQL?

  A. An ability to work with data in a variety of types at once

  B. An ability to work with data within certain partitions and windows

  C. An ability to work with time-related data in specified intervals

  D. An ability to work with complex, nested data ingested from JSON files

  E. An ability to work with an array of tables for procedural automation

  Correct Answer: D

  Explanation: Array functions in Spark SQL are primarily used for working with arrays and complex, nested data structures, such as those often encountered when ingesting JSON files. These functions allow you to manipulate and query nested arrays and structures within your data, making it easier to extract and work with specific elements or values within complex data formats. While some of the other options (such as option A for working with different data types) are features of Spark SQL or SQL in general, array functions specifically excel at handling complex, nested data structures like those found in JSON files.

• Question 4:

  Which of the following is a benefit of the Databricks Lakehouse Platform embracing open source technologies?

  A. Cloud-specific integrations

  B. Simplified governance

  C. Ability to scale storage

  D. Ability to scale workloads

  E. Avoiding vendor lock-in

  Correct Answer: E

  Explanation: https://double.cloud/blog/posts/2023/01/break-free-from-vendor-lock-in-with- open-source-tech/

• Question 5:

  Which of the following describes the storage organization of a Delta table?

  A. Delta tables are stored in a single file that contains data, history, metadata, and other attributes.

  B. Delta tables store their data in a single file and all metadata in a collection of files in a separate location.

  C. Delta tables are stored in a collection of files that contain data, history, metadata, and other attributes.

  D. Delta tables are stored in a collection of files that contain only the data stored within the table.

  E. Delta tables are stored in a single file that contains only the data stored within the table.

  Correct Answer: C

  Explanation: Delta tables store data in a structured manner using Parquet files, and they also maintain metadata and transaction logs in separate directories. This organization allows for versioning, transactional capabilities, and metadata tracking in Delta Lake. Thank you for pointing out the error, and I appreciate your understanding.

• Question 6:

  A data engineer only wants to execute the final block of a Python program if the Python variable day_of_week is equal to 1 and the Python variable review_period is True.

  Which of the following control flow statements should the data engineer use to begin this conditionally executed code block?

  A. if day_of_week = 1 and review_period:

  B. if day_of_week = 1 and review_period = "True":

  C. if day_of_week == 1 and review_period == "True":

  D. if day_of_week == 1 and review_period:

  E. if day_of_week = 1 and review_period: = "True":

  Correct Answer: D

  Explanation: This statement will check if the variable day_of_week is equal to 1 and if the variable review_period evaluates to a truthy value. The use of the double equal sign (==) in the comparison of day_of_week is important, as a single equal sign (=) would be used to assign a value to the variable instead of checking its value. The use of a single ampersand (and) instead of the keyword and is not valid syntax in Python. The use of quotes around True in options B and C will result in a string comparison, which will not evaluate to True even if the value of review_period is True.

• Question 7:

  Which of the following can be used to simplify and unify siloed data architectures that are specialized for specific use cases?

  A. None of these

  B. Data lake

  C. Data warehouse

  D. All of these

  E. Data lakehouse

  Correct Answer: E

• Question 8:

  A data engineering team has noticed that their Databricks SQL queries are running too slowly when they are submitted to a non-running SQL endpoint. The data engineering team wants this issue to be resolved.

  Which of the following approaches can the team use to reduce the time it takes to return results in this scenario?

  A. They can turn on the Serverless feature for the SQL endpoint and change the Spot Instance Policy to "Reliability Optimized."

  B. They can turn on the Auto Stop feature for the SQL endpoint.

  C. They can increase the cluster size of the SQL endpoint.

  D. They can turn on the Serverless feature for the SQL endpoint.

  E. They can increase the maximum bound of the SQL endpoint's scaling range

  Correct Answer: C

  Explanation: https://www.databricks.com/blog/2022/03/10/top-5-databricks-performance- tips.html

• Question 9:

  Which of the following data lakehouse features results in improved data quality over a traditional data lake?

  A. A data lakehouse provides storage solutions for structured and unstructured data.

  B. A data lakehouse supports ACID-compliant transactions.

  C. A data lakehouse allows the use of SQL queries to examine data.

  D. A data lakehouse stores data in open formats.

  E. A data lakehouse enables machine learning and artificial Intelligence workloads.

  Correct Answer: B

  Explanation: One of the key features of a data lakehouse that results in improved data quality over a traditional data lake is its support for ACID (Atomicity, Consistency, Isolation, Durability) transactions. ACID transactions provide data integrity and consistency guarantees, ensuring that operations on the data are reliable and that data is not left in an inconsistent state due to failures or concurrent access. In a traditional data lake, such transactional guarantees are often lacking, making it challenging to maintain data quality, especially in scenarios involving multiple data writes, updates, or complex transformations. A data lakehouse, by offering ACID compliance, helps maintain data quality by providing strong consistency and reliability, which is crucial for data pipelines and analytics.

• Question 10:

  A data engineer wants to schedule their Databricks SQL dashboard to refresh every hour, but they only want the associated SQL endpoint to be running when it is necessary. The dashboard has multiple queries on multiple datasets associated with it. The data that feeds the dashboard is automatically processed using a Databricks Job.

  Which of the following approaches can the data engineer use to minimize the total running time of the SQL endpoint used in the refresh schedule of their dashboard?

  A. They can turn on the Auto Stop feature for the SQL endpoint.

  B. They can ensure the dashboard's SQL endpoint is not one of the included query's SQL endpoint.

  C. They can reduce the cluster size of the SQL endpoint.

  D. They can ensure the dashboard's SQL endpoint matches each of the queries' SQL endpoints.

  E. They can set up the dashboard's SQL endpoint to be serverless.

  Correct Answer: A