Databricks Databricks Certifications DATABRICKS-CERTIFIED-PROFESSIONAL-DATA-ENGINEER Questions & Answers

• Question 21:

  What is a method of installing a Python package scoped at the notebook level to all nodes in the currently active cluster?

  A. Use andPip install in a notebook cell

  B. Run source env/bin/activate in a notebook setup script

  C. Install libraries from PyPi using the cluster UI

  D. Use andsh install in a notebook cell

  Correct Answer: C

  Installing a Python package scoped at the notebook level to all nodes in the currently active cluster in Databricks can be achieved by using the Libraries tab in the cluster UI. This interface allows you to install libraries across all nodes in the

  cluster. While the %pip command in a notebook cell would only affect the driver node, using the cluster UI ensures that the package is installed on all nodes.

  References:

  Databricks Documentation on Libraries: Libraries

• Question 22:

  When evaluating the Ganglia Metrics for a given cluster with 3 executor nodes, which indicator would signal proper utilization of the VM's resources?

  A. The five Minute Load Average remains consistent/flat

  B. Bytes Received never exceeds 80 million bytes per second

  C. Network I/O never spikes

  D. Total Disk Space remains constant

  E. CPU Utilization is around 75%

  Correct Answer: E

  In the context of cluster performance and resource utilization, a CPU utilization rate of around 75% is generally considered a good indicator of efficient resource usage. This level of CPU utilization suggests that the cluster is being effectively

  used without being overburdened or underutilized.

  A consistent 75% CPU utilization indicates that the cluster's processing power is being effectively employed while leaving some headroom to handle spikes in workload or additional tasks without maxing out the CPU, which could lead to

  performance degradation.

  A five Minute Load Average that remains consistent/flat (Option A) might indicate underutilization or a bottleneck elsewhere.

  Monitoring network I/O (Options B and C) is important, but these metrics alone don't provide a complete picture of resource utilization efficiency. Total Disk Space (Option D) remaining constant is not necessarily an indicator of proper

  resource utilization, as it's more related to storage rather than computational efficiency.

  References:

  Ganglia Monitoring System: Ganglia Documentation

  Databricks Documentation on Monitoring: Databricks Cluster Monitoring

• Question 23:

  A developer has successfully configured credential for Databricks Repos and cloned a remote Git repository. Hey don not have privileges to make changes to the main branch, which is the only branch currently visible in their workspace.

  Use Response to pull changes from the remote Git repository commit and push changes to a branch that appeared as a changes were pulled.

  A. Use Repos to merge all differences and make a pull request back to the remote repository.

  B. Use repos to merge all difference and make a pull request back to the remote repository.

  C. Use Repos to create a new branch commit all changes and push changes to the remote Git repertory.

  D. Use repos to create a fork of the remote repository commit all changes and make a pull request on the source repository

  Correct Answer: C

  In Databricks Repos, when a user does not have privileges to make changes directly to the main branch of a cloned remote Git repository, the recommended approach is to create a new branch within the Databricks workspace. The developer can then make changes in this new branch, commit those changes, and push the new branch to the remote Git repository. This workflow allows for isolated development without affecting the main branch, enabling the developer to propose changes via a pull request from the new branch to the main branch in the remote repository. This method adheres to common Git collaboration workflows, fostering code review and collaboration while ensuring the integrity of the main branch. References: Databricks documentation on using Repos with Git: https://docs.databricks.com/repos.html

• Question 24:

  To reduce storage and compute costs, the data engineering team has been tasked with curating a series of aggregate tables leveraged by business intelligence dashboards, customer-facing applications, production machine learning models, and ad hoc analytical queries.

  The data engineering team has been made aware of new requirements from a customer-facing application, which is the only downstream workload they manage entirely. As a result, an aggregate table used by numerous teams across the organization will need to have a number of fields renamed, and additional fields will also be added.

  Which of the solutions addresses the situation while minimally interrupting other teams in the organization without increasing the number of tables that need to be managed?

  A. Send all users notice that the schema for the table will be changing; include in the communication the logic necessary to revert the new table schema to match historic queries.

  B. Configure a new table with all the requisite fields and new names and use this as the source for the customer-facing application; create a view that maintains the original data schema and table name by aliasing select fields from the new table.

  C. Create a new table with the required schema and new fields and use Delta Lake's deep clone functionality to sync up changes committed to one table to the corresponding table.

  D. Replace the current table definition with a logical view defined with the query logic currently writing the aggregate table; create a new table to power the customer-facing application.

  E. Add a table comment warning all users that the table schema and field names will be changing on a given date; overwrite the table in place to the specifications of the customer-facing application.

  Correct Answer: B

  This is the correct answer because it addresses the situation while minimally interrupting other teams in the organization without increasing the number of tables that need to be managed. The situation is that an aggregate table used by numerous teams across the organization will need to have a number of fields renamed, and additional fields will also be added, due to new requirements from a customer-facing application. By configuring a new table with all the requisite fields and new names and using this as the source for the customer-facing application, the data engineering team can meet the new requirements without affecting other teams that rely on the existing table schema and name. By creating a view that maintains the original data schema and table name by aliasing select fields from the new table, the data engineering team can also avoid duplicating data or creating additional tables that need to be managed. Verified References: [Databricks Certified Data Engineer Professional], under "Lakehouse" section; Databricks Documentation, under "CREATE VIEW" section.

• Question 25:

  A Delta Lake table representing metadata about content posts from users has the following schema:

  user_id LONG, post_text STRING, post_id STRING, longitude FLOAT, latitude FLOAT, post_time TIMESTAMP, date DATE

  This table is partitioned by the date column. A query is run with the following filter:

  longitude < 20 and longitude >-20

  Which statement describes how data will be filtered?

  A. Statistics in the Delta Log will be used to identify partitions that might Include files in the filtered range.

  B. No file skipping will occur because the optimizer does not know the relationship between the partition column and the longitude.

  C. The Delta Engine will use row-level statistics in the transaction log to identify the flies that meet the filter criteria.

  D. Statistics in the Delta Log will be used to identify data files that might include records in the filtered range.

  E. The Delta Engine will scan the parquet file footers to identify each row that meets the filter criteria.

  Correct Answer: D

  This is the correct answer because it describes how data will be filtered when a query is run with the following filter: longitude < 20 and longitude >-20. The query is run on a Delta Lake table that has the following schema: user_id LONG, post_text STRING, post_id STRING, longitude FLOAT, latitude FLOAT, post_time TIMESTAMP, date DATE. This table is partitioned by the date column. When a query is run on a partitioned Delta Lake table, Delta Lake uses statistics in the Delta Log to identify data files that might include records in the filtered range. The statistics include information such as min and max values for each column in each data file. By using these statistics, Delta Lake can skip reading data files that do not match the filter condition, which can improve query performance and reduce I/O costs. Verified References: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Data skipping" section.

• Question 26:

  A junior data engineer is working to implement logic for a Lakehouse table named silver_device_recordings. The source data contains 100 unique fields in a highly nested JSON structure.

  The silver_device_recordings table will be used downstream for highly selective joins on a number of fields, and will also be leveraged by the machine learning team to filter on a handful of relevant fields, in total, 15 fields have been identified

  that will often be used for filter and join logic.

  The data engineer is trying to determine the best approach for dealing with these nested fields before declaring the table schema.

  Which of the following accurately presents information about Delta Lake and Databricks that may Impact their decision-making process?

  A. Because Delta Lake uses Parquet for data storage, Dremel encoding information for nesting can be directly referenced by the Delta transaction log.

  B. Tungsten encoding used by Databricks is optimized for storing string data: newly-added native support for querying JSON strings means that string types are always most efficient.

  C. Schema inference and evolution on Databricks ensure that inferred types will always accurately match the data types used by downstream systems.

  D. By default Delta Lake collects statistics on the first 32 columns in a table; these statistics are leveraged for data skipping when executing selective queries.

  Correct Answer: D

  Delta Lake, built on top of Parquet, enhances query performance through data skipping, which is based on the statistics collected for each file in a table. For tables with a large number of columns, Delta Lake by default collects and stores statistics only for the first 32 columns. These statistics include min/max values and null counts, which are used to optimize query execution by skipping irrelevant data files. When dealing with highly nested JSON structures, understanding this behavior is crucial for schema design, especially when determining which fields should be flattened or prioritized in the table structure to leverage data skipping efficiently for performance optimization.References: Databricks documentation on Delta Lake optimization techniques, including data skipping and statistics collection (https://docs.databricks.com/delta/optimizations/index.html).

• Question 27:

  A junior member of the data engineering team is exploring the language interoperability of Databricks notebooks. The intended outcome of the below code is to register a view of all sales that occurred in countries on the continent of Africa that appear in the geo_lookup table.

  Before executing the code, running SHOW TABLES on the current database indicates the database contains only two tables: geo_lookup and sales.

  

  Which statement correctly describes the outcome of executing these command cells in order in an interactive notebook?

  A. Both commands will succeed. Executing show tables will show that countries at and sales at have been registered as views.

  B. Cmd 1 will succeed. Cmd 2 will search all accessible databases for a table or view named countries af: if this entity exists, Cmd 2 will succeed.

  C. Cmd 1 will succeed and Cmd 2 will fail, countries at will be a Python variable representing a PySpark DataFrame.

  D. Both commands will fail. No new variables, tables, or views will be created.

  E. Cmd 1 will succeed and Cmd 2 will fail, countries at will be a Python variable containing a list of strings.

  Correct Answer: E

  This is the correct answer because Cmd 1 is written in Python and uses a list comprehension to extract the country names from the geo_lookup table and store them in a Python variable named countries af. This variable will contain a list of

  strings, not a PySpark DataFrame or a SQL view. Cmd 2 is written in SQL and tries to create a view named sales af by selecting from the sales table where city is in countries af. However, this command will fail because countries af is not a

  valid SQL entity and cannot be used in a SQL query. To fix this, a better approach would be to use spark.sql() to execute a SQL query in Python and pass the countries af variable as a parameter. Verified References:

  [Databricks Certified Data Engineer Professional], under "Language Interoperability" section; Databricks Documentation, under "Mix languages" section.

• Question 28:

  What statement is true regarding the retention of job run history?

  A. It is retained until you export or delete job run logs

  B. It is retained for 30 days, during which time you can deliver job run logs to DBFS or S3

  C. t is retained for 60 days, during which you can export notebook run results to HTML

  D. It is retained for 60 days, after which logs are archived

  E. It is retained for 90 days or until the run-id is re-used through custom run configuration

  Correct Answer: C

• Question 29:

  The business reporting tem requires that data for their dashboards be updated every hour. The total processing time for the pipeline that extracts transforms and load the data for their pipeline runs in 10 minutes. Assuming normal operating conditions, which configuration will meet their service-level agreement requirements with the lowest cost?

  A. Schedule a jo to execute the pipeline once and hour on a dedicated interactive cluster.

  B. Schedule a Structured Streaming job with a trigger interval of 60 minutes.

  C. Schedule a job to execute the pipeline once hour on a new job cluster.

  D. Configure a job that executes every time new data lands in a given directory.

  Correct Answer: C

  Scheduling a job to execute the data processing pipeline once an hour on a new job cluster is the most cost-effective solution given the scenario. Job clusters are ephemeral in nature; they are spun up just before the job execution and

  terminated upon completion, which means you only incur costs for the time the cluster is active. Since the total processing time is only 10 minutes, a new job cluster created for each hourly execution minimizes the running time and thus the

  cost, while also fulfilling the requirement for hourly data updates for the business reporting team's dashboards.

  References:

  Databricks documentation on jobs and job clusters:

  https://docs.databricks.com/jobs.html

• Question 30:

  A Delta Lake table was created with the below query:

  

  Consider the following query:

  DROP TABLE prod.sales_by_store-

  If this statement is executed by a workspace admin, which result will occur?

  A. Nothing will occur until a COMMIT command is executed.

  B. The table will be removed from the catalog but the data will remain in storage.

  C. The table will be removed from the catalog and the data will be deleted.

  D. An error will occur because Delta Lake prevents the deletion of production data.

  E. Data will be marked as deleted but still recoverable with Time Travel.

  Correct Answer: C

  When a table is dropped in Delta Lake, the table is removed from the catalog and the data is deleted. This is because Delta Lake is a transactional storage layer that provides ACID guarantees. When a table is dropped, the transaction log is updated to reflect the deletion of the table and the data is deleted from the underlying storage. References: https://docs.databricks.com/delta/quick-start.html#drop-a-table https://docs.databricks.com/delta/delta-batch.html#drop-table