Databricks DATABRICKS-CERTIFIED-DATA-ENGINEER-ASSOCIATE Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-DATA-ENGINEER-ASSOCIATE Online Questions & Answers

• Question 41:

  In order for Structured Streaming to reliably track the exact progress of the processing so that it can handle any kind of failure by restarting and/or reprocessing, which of the following two approaches is used by Spark to record the offset range of the data being processed in each trigger?

  A. Checkpointing and Write-ahead Logs
  B. Structured Streaming cannot record the offset range of the data being processed in each trigger.
  C. Replayable Sources and Idempotent Sinks
  D. Write-ahead Logs and Idempotent Sinks
  E. Checkpointing and Idempotent Sinks
  A. Checkpointing and Write-ahead Logs

  ---

  Explanation

  Structured Streaming uses checkpointing and write-ahead logs to record the offset range of the data being processed in each trigger. This ensures that the engine can reliably track the exact progress of the processing and handle any kind of failure by restarting and/or reprocessing. Checkpointing is the mechanism of saving the state of a streaming query to fault-tolerant storage (such as HDFS) so that it can be recovered after a failure. Write-ahead logs are files that record the offset range of the data being processed in each trigger and are written to the checkpoint location before the processing starts. These logs are used to recover the query state and resume processing from the last processed offset range in case of a failure.

  References: Structured Streaming Programming Guide, Fault Tolerance Semantics

• Question 42:

  A new data engineering team team. has been assigned to an ELT project. The new data engineering team will need full privileges on the database customers to fully manage the project.

  Which of the following commands can be used to grant full permissions on the database to the new data engineering team?

  A. GRANT USAGE ON DATABASE customers TO team;
  B. GRANT ALL PRIVILEGES ON DATABASE team TO customers;
  C. GRANT SELECT PRIVILEGES ON DATABASE customers TO teams;
  D. GRANT SELECT CREATE MODIFY USAGE PRIVILEGES ON DATABASE customers TO team;
  E. GRANT ALL PRIVILEGES ON DATABASE customers TO team;
  E. GRANT ALL PRIVILEGES ON DATABASE customers TO team;

  ---

  Explanation

  To grant full permissions on a database to a user, group, or service principal, the GRANT ALL PRIVILEGES ON DATABASE command can be used. This command grants all the applicable privileges on the database, such as CREATE,

  SELECT, MODIFY, and USAGE. The other options are either incorrect or incomplete, as they do not grant all the privileges or specify the wrong database or principal.

  References:

  GRANT

  Privileges

• Question 43:

  A data engineer wants to schedule their Databricks SQL dashboard to refresh every hour, but they only want the associated SQL endpoint to be running when It is necessary. The dashboard has multiple queries on multiple datasets associated with it. The data that feeds the dashboard is automatically processed using a Databricks Job.

  Which approach can the data engineer use to minimize the total running time of the SQL endpoint used in the refresh schedule of their dashboard?

  A. They can reduce the cluster size of the SQL endpoint.
  B. They can turn on the Auto Stop feature for the SQL endpoint.
  C. They can set up the dashboard's SQL endpoint to be serverless.
  D. They can ensure the dashboard's SQL endpoint matches each of the queries' SQL endpoints.
  B. They can turn on the Auto Stop feature for the SQL endpoint.

  ---

  Explanation

  To minimize the total running time of the SQL endpoint used in the refresh schedule of a dashboard in Databricks, the most effective approach is to utilize the Auto Stop feature. This feature allows the SQL endpoint to automatically stop after a period of inactivity, ensuring that it only runs when necessary, such as during the dashboard refresh or when actively queried. This minimizes resource usage and associated costs by ensuring the SQL endpoint is not running idle outside of these operations.

  References:Databricks documentation on SQL endpoints: SQL Endpoints in Databricks

• Question 44:

  What is the maximum output supported by a job cluster to ensure a notebook does not fail?

  A. 25MBs
  B. 10MBs
  C. 30MBs
  D. 15MBs
  B. 10MBs

  ---

  Explanation

  The maximum output supported by a job cluster in Databricks is 10MB. If the output exceeds this limit, the notebook may fail.

• Question 45:

  Which of the following describes a scenario in which a data team will want to utilize cluster pools?

  A. An automated report needs to be refreshed as quickly as possible.
  B. An automated report needs to be made reproducible.
  C. An automated report needs to be tested to identify errors.
  D. An automated report needs to be version-controlled across multiple collaborators.
  E. An automated report needs to be runnable by all stakeholders.
  A. An automated report needs to be refreshed as quickly as possible.

  ---

  Explanation

  Databricks cluster pools are a set of idle, ready-to-use instances that can reduce cluster start and auto-scaling times. This is useful for scenarios where a data team needs to run an automated report as quickly as possible, without waiting for the cluster to launch or scale up. Cluster pools can also help save costs by reusing idle instances across different clusters and avoiding DBU charges for idle instances in the pool.

  References: Best practices: pools | Databricks on AWS, Best practices: pools - Azure Databricks | Microsoft Learn, Best practices: pools | Databricks on Google Cloud

• Question 46:

  Which languages are supported by Serverless compute clusters? (Choose two.)

  A. SQL
  B. Python
  C. R
  D. Scala
  E. Java
  A. SQL
  B. Python

  ---

  Explanation

  Serverless compute clusters in Databricks support SQL (via Serverless SQL Warehouses) and Python (for interactive and job-based execution). Other languages like R, Scala, and Java require regular clusters.

• Question 47:

  Which of the following can be used to simplify and unify siloed data architectures that are specialized for specific use cases?

  A. None of these
  B. Data lake
  C. Data warehouse
  D. All of these
  E. Data lakehouse
  E. Data lakehouse

  ---

  Explanation

  A data lakehouse is a new paradigm that can be used to simplify and unify siloed data architectures that are specialized for specific use cases. A data lakehouse combines the best of both data lakes and data warehouses, providing a single platform that supports diverse data types, open standards, low-cost storage, high-performance queries, ACID transactions, schema enforcement, and governance. A data lakehouse enables data engineers to build reliable and scalable data pipelines that can serve various downstream applications and users, such as data science, machine learning, analytics, and reporting. A data lakehouse leverages the power of Delta Lake, a storage layer that brings reliability and performance to data lakes.

  References: What is a data lakehouse?, Delta Lake, Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics

• Question 48:

  A data engineer has developed an ETL that produce a Delta managed table with liquid clustering feature activated as output. Several consumers are having issues regarding time delay when reading this table. How could the Data Engineer be sure about the OPTIMIZE command has been executed explicitly?

  A. Check the system table system.storage.predictive_optimization_operations_history
  B. Use SHOW TABLES EXTENDED to check the partitions columns used
  C. Use DESCRIBE DETAIL table to see the file size and number of files for the table
  D. Use DESCRIBE HISTORY table to check if exists any OPTIMIZE operation
  D. Use DESCRIBE HISTORY table to check if exists any OPTIMIZE operation

  ---

  Explanation

  To confirm whether the OPTIMIZE command has been explicitly executed on a Delta table, the data engineer should run DESCRIBE HISTORY table, which records all table operations, including OPTIMIZE runs.

• Question 49:

  A data engineer has a Job that has a complex run schedule, and they want to transfer that schedule to other Jobs.

  Rather than manually selecting each value in the scheduling form in Databricks, which of the following tools can the data engineer use to represent and submit the schedule programmatically?

  A. pyspark.sql.types.DateType
  B. datetime
  C. pyspark.sql.types.TimestampType
  D. Cron syntax
  E. There is no way to represent and submit this information programmatically
  D. Cron syntax

  ---

  Explanation

  Cron syntax is a tool that can be used to represent and submit a complex run schedule programmatically. Cron syntax is a string of six fields that specify the frequency, date, and time of a job run. For example, the cron expression 0 0 12 * * ?

  means run the job at 12:00 PM every day. The data engineer can use the Databricks REST API to create or update a job with a cron schedule. The data engineer can also use the Databricks CLI to create or update a job with a cron schedule

  by using a JSON file that contains the cron expression. The other tools are either invalid or not suitable for representing and submitting a complex run schedule programmatically.

  References: Schedule a job, Jobs API, Databricks CLI, Cron expressions

• Question 50:

  A table in Delta Lake has accumulated many small Parquet files, which is slowing down query performance. Which command should a data engineer run to improve performance?

  A. VACUUM
  B. MERGE
  C. OPTIMIZE
  D. REPARTITION
  E. CLEANUP
  C. OPTIMIZE

  ---

  Explanation

  OPTIMIZE compacts small files into larger ones for efficient reads, while VACUUM is for removing old unreferenced files.