Databricks Certified Data Engineer Professional

A standard view stores query logic rather than materialized join results, so the filtering and join execute when recent_orders is queried. Delta Lake supplies a consistent snapshot of each source table for that query. The returned rows therefore come from the valid table versions at the time the query began, even if concurrent updates finish before it ends.

Changing the schema and ingestion logic affects records processed after the change; it does not retroactively recalculate values for rows already stored. Historical records therefore lack the newly derived Spark ingestion timestamp and the added Kafka metadata unless a separate backfill rewrites them. This limits comparisons of past and current processing delays.

Table tags are table metadata, so they are added by altering the existing table and setting key-value tag pairs. The valid form is ALTER TABLE table_name SET TAGS ('key1' = 'value1', 'key2' = 'value2'). This directly applies both tags programmatically; comments and the proposed standalone tag commands do not perform that table alteration.

Job run history remains available for 60 days. During that retention window, notebook run results can be exported to HTML for preservation or review outside the retained run record. The key distinction is that the history is not kept indefinitely and is not described as being archived automatically after the retention period.

A streaming table is appropriate for continuously ingesting and retaining the raw Auto Loader telemetry. Another streaming table can incrementally maintain the latest location, speed, and fuel level per truck for near-real-time operations. The daily distance and fuel-efficiency aggregates are derived reporting results, so a materialized view provides the appropriate maintained batch-style representation.

DESCRIBE EXTENDED dev.pii_test returns the table's detailed metadata in one result. That metadata includes the column definitions and their comments, the table-level comment, and table properties such as contains_pii. SHOW TBLPROPERTIES would confirm the custom property but would not also confirm both kinds of comments.

The query performs a stateful aggregation with groupBy, so Structured Streaming stores aggregation state and schema information in the checkpoint. Adding new_member_promo changes the state schema of the aggregation. An existing checkpoint cannot safely be reused after this type of stateful query change. A new checkpoint location is required so the stream can initialize state using the new aggregation schema. mergeSchema only handles schema evolution for the Delta output table; it does not make an incompatible streaming checkpoint state schema valid.

Disk spill indicates that execution data cannot remain in available memory. Selecting compute with a more favorable core-to-memory ratio gives concurrent tasks more memory for operations such as joins and shuffles. Reducing spark.sql.files.maxPartitionBytes also creates smaller input partitions, lowering each task's working-set requirement. Both changes directly reduce memory pressure that causes spilling.

Common Python testing frameworks work naturally with importable Python modules rather than functions scattered across notebook execution contexts. Moving the functions into Files in Repos lets production code and test code import the same modules, organize dependencies, and run through standard test tooling. Tests can then exercise representative non-production data without coupling test execution to production notebooks.

The dashboard requires one refresh per hour, while the pipeline completes in only 10 minutes. Scheduling the pipeline as an hourly job on a new job cluster directly meets that interval and allows the compute to terminate after each run. An always-running interactive or streaming cluster would consume resources between the required updates.