Q29
Single choice
Given the following PySpark code snippet in a Databricks notebook:
filtered_df=spark.read.format("delta").load("/mnt/daya/large_table") \ .filter ("event_date> '2024-01-01'")
filtered_df.count ()
The data engineer notices from the Query Profile that the scan operator for filtered_df is reading almost all files, despite a filter being applied.
What is the probable reason for poor data skipping?