Databricks DATABRICKS-CERTIFIED-DATA-ENGINEER-ASSOCIATE Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-DATA-ENGINEER-ASSOCIATE Online Questions & Answers

• Question 61:

  A dataset has been defined using Delta Live Tables and includes an expectations clause:

  CONSTRAINT valid_timestamp EXPECT (timestamp > '2020-01-01') ON VIOLATION DROP ROW

  What is the expected behavior when a batch of data containing data that violates these constraints is processed?

  A. Records that violate the expectation are dropped from the target dataset and loaded into a quarantine table.
  B. Records that violate the expectation are added to the target dataset and flagged as invalid in a field added to the target dataset.
  C. Records that violate the expectation are dropped from the target dataset and recorded as invalid in the event log.
  D. Records that violate the expectation are added to the target dataset and recorded as invalid in the event log.
  E. Records that violate the expectation cause the job to fail.
  C. Records that violate the expectation are dropped from the target dataset and recorded as invalid in the event log.

  ---

  Explanation

  Delta Live Tables expectations are optional clauses that apply data quality checks on each record passing through a query. An expectation consists of a description, a boolean statement, and an action to take when a record fails the expectation. The ON VIOLATION clause specifies the action to take, which can be one of the following: warn, drop, or fail. The drop action means that invalid records are dropped from the target dataset before data is written to the target. The failure is reported as a metric for the dataset, which can be viewed by querying the Delta Live Tables event log. The event log contains information such as the number of records that violate an expectation, the number of records dropped, and the number of records written to the target dataset.

  References: Manage data quality with Delta Live Tables Monitor Delta Live Tables pipelines Delta Live Tables SQL language reference

• Question 62:

  A data engineer is using the Databricks OPTIMIZE command on a Delta table. What happens when OPTIMIZE is run twice on the same table with the same data?

  A. It has no effect because it is idempotent.
  B. It changes the number of tuples per file significantly.
  C. It further reduces file sizes by re-clustering the data.
  D. It triggers a full liquid clustering process.
  A. It has no effect because it is idempotent.

  ---

  Explanation

  The OPTIMIZE command is idempotent - running it again on the same Delta table with unchanged data has no additional effect, since the files are already compacted.

• Question 63:

  A data engineer needs to load files incrementally from a cloud storage directory into a Delta table while ensuring schema evolution is handled automatically. Which of the following tools should be used?

  A. Databricks SQL
  B. Delta Live Tables
  C. Auto Loader
  D. Structured Streaming
  E. Unity Catalog
  C. Auto Loader

  ---

  Explanation

  Auto Loader is designed for incremental ingestion with schema inference and evolution, making it ideal for continuously processing new files arriving in cloud storage.

• Question 64:

  Which Databricks feature provides automatic data ingestion with schema inference?

  A. Auto Loader
  B. Delta Live Tables
  C. Unity Catalog
  D. Job Scheduler
  E. Data Explorer
  A. Auto Loader

  ---

  Explanation

  Auto Loader automatically ingests files and infers schema from input data.

• Question 65:

  A data engineer is working on a personal laptop and needs to perform complex transformations on data stored in a Delta Lake on cloud storage. The engineer decides to use Databricks Connect to interact with Databricks clusters and work in their local IDE.

  How does Databricks Connect enable the engineer to develop, test, and debug code seamlessly on their local machine while interacting with Databricks clusters?

  A. By providing a local environment that mimics the Databricks runtime, enabling the engineer to develop, test, and debug code using a specific IDE that is required by Databricks
  B. By providing a local environment that mimics the Databricks runtime, enabling the engineer to develop, test, and debug code only through Databricks' own web interface
  C. By allowing direct execution of Spark jobs from the local machine without needing a network connection
  D. By providing a local environment that mimics the Databricks runtime, enabling the engineer to develop, test, and debug code using their preferred IDE
  D. By providing a local environment that mimics the Databricks runtime, enabling the engineer to develop, test, and debug code using their preferred IDE

  ---

  Explanation

  Databricks Connect lets engineers use their preferred local IDE while connecting to Databricks clusters remotely. It mimics the Databricks runtime locally, enabling seamless development, testing, and debugging while executing workloads on the Databricks cluster.

• Question 66:

  In which of the following file formats is data from Delta Lake tables primarily stored?

  A. Delta
  B. CSV
  C. Parquet
  D. JSON
  E. A proprietary, optimized format specific to Databricks
  C. Parquet

  ---

  Explanation

  Delta Lake is an open source project that provides ACID transactions, time travel, and other features on top of Apache Parquet, a columnar file format that is widely used for big data analytics. Delta Lake uses versioned Parquet files to store your data in your cloud storage, along with JSON files as transaction logs and checkpoint files to track the changes and ensure data integrity. Delta Lake is compatible with any Apache Hive compatible file format, such as CSV, JSON, or AVRO, but it primarily stores data as Parquet files for better performance and compression.

  References: How to Create Delta Lake tables, 5 reasons to choose Delta Lake format (on Databricks), Parquet vs Delta format in Azure Data Lake Gen 2 store, What is Delta Lake? - Azure Databricks, Lakehouse and Delta tables - Microsoft Fabric

• Question 67:

  What happens when VACUUM is executed with RETENTION 0 HOURS on a Delta table?

  A. No effect on data.
  B. Deletes all unreferenced files immediately.
  C. Deletes all table data.
  D. Compacts small files.
  E. Refreshes the transaction log.
  B. Deletes all unreferenced files immediately.

  ---

  Explanation

  VACUUM RETENTION 0 HOURS forces immediate deletion of old unreferenced files (use carefully).

• Question 68:

  The Delta transaction log for the `students' tables is shown using the `DESCRIBE HISTORY students' command. A Data Engineer needs to query the table as it existed before the UPDATE operation listed in the log. Which command should the Data Engineer use to achieve this? (Choose two.)

  

  A. SELECT * FROM students@v4
  B. SELECT * FROM students TIMESTAMP AS OF `2024-04-22T 14:32:47.000+00:00'
  C. SELECT * FROM students FROM HISTORY VERSION AS OF 3
  D. SELECT * FROM students VERSION AS OF 5
  E. SELECT * FROM students TIMESTAMP AS OF `2024-04-22T 14:32:58.000+00:00'
  A. SELECT * FROM students@v4
  B. SELECT * FROM students TIMESTAMP AS OF `2024-04-22T 14:32:47.000+00:00'

  ---

  Explanation

• Question 69:

  What is the functionality of AutoLoader in Databricks?

  A. Auto Loader automatically ingests and processes new files from cloud storage, handling both batch and streaming data with support for schema evolution.
  B. Auto Loader automatically ingests and processes new files from cloud storage, handling batch and streaming data with no support for schema evolution.
  C. Auto Loader automatically ingests and processes new files from cloud storage, handling only streaming data with no support for schema evolution.
  D. Auto Loader automatically ingests and processes new files from cloud storage, handling batch data with support for schema evolution.
  A. Auto Loader automatically ingests and processes new files from cloud storage, handling both batch and streaming data with support for schema evolution.

  ---

  Explanation

  Auto Loader in Databricks automatically ingests new files from cloud storage, supports both batch and streaming ingestion, and handles schema evolution, making it ideal for scalable pipelines.

• Question 70:

  An organization needs to share a dataset stored in its Databricks Unity Catalog with an external partner who uses a different data platform that is not Databricks. The goal is to maintain data security and ensure the partner can access the

  data efficiently.

  .

  Which method should the data engineer use to securely share the dataset with the external partner?

  A. Using Delta Sharing with the open sharing protocol
  B. Exporting data as CSV files and emailing them
  C. Using a third-party API to access the Delta table
  D. Databricks-to-Databricks Sharing
  A. Using Delta Sharing with the open sharing protocol

  ---

  Explanation

  Delta Sharing with the open sharing protocol allows secure data sharing from Unity Catalog to external partners, even if they are not using Databricks, while maintaining governance and efficient access.