Databricks Databricks Certifications DATABRICKS-MACHINE-LEARNING-ASSOCIATE Questions & Answers

• Question 61:

  A data scientist is working with a feature set with the following schema:

  

  Thecustomer_idcolumn is the primary key in the feature set. Each of the columns in the feature set has missing values. They want to replace the missing values by imputing a common value for each feature.

  Which of the following lists all of the columns in the feature set that need to be imputed using the most common value of the column?

  A. customer_id, loyalty_tier

  B. loyalty_tier

  C. units

  D. spend

  E. customer_id

  Correct Answer: B

  For the feature set schema provided, the columns that need to be imputed using the most common value (mode) are typically the categorical columns. In this case, loyalty_tieris the only categorical column that should be imputed using the

  most common value.customer_idis a unique identifier and should not be imputed, whilespendandunits are numerical columns that should typically be imputed using the mean or median values, not the mode.

  References:

  Databricks documentation on missing value imputation: Handling Missing Data If you need any further clarification or additional questions answered, please let me know!

• Question 62:

  A data scientist has replaced missing values in their feature set with each respective feature variable's median value. A colleague suggests that the data scientist is throwing away valuable information by doing this.

  Which of the following approaches can they take to include as much information as possible in the feature set?

  A. Impute the missing values using each respective feature variable's mean value instead of the median value

  B. Refrain from imputing the missing values in favor of letting the machine learning algorithm determine how to handle them

  C. Remove all feature variables that originally contained missing values from the feature set

  D. Create a binary feature variable for each feature that contained missing values indicating whether each row's value has been imputed

  E. Create a constant feature variable for each feature that contained missing values indicating the percentage of rows from the feature that was originally missing

  Correct Answer: D

  By creating a binary feature variable for each feature with missing values to indicate whether a value has been imputed, the data scientist can preserve information about the original state of the data. This approach maintains the integrity of

  the dataset by marking which values are original and which are synthetic (imputed). Here are the steps to implement this approach:

  Identify Missing Values:Determine which features contain missing values. Impute Missing Values:Continue with median imputation or choose another method (mean, mode, regression, etc.) to fill missing values. Create Indicator Variables:For

  each feature that had missing values, add a new binary feature. This feature should be '1' if the original value was missing and imputed, and '0' otherwise.

  Data Integration:Integrate these new binary features into the existing dataset. This maintains a record of where data imputation occurred, allowing models to potentially weight these observations differently. Model Adjustment:Adjust machine

  learning models to account for these new features, which might involve considering interactions between these binary indicators and other features.

  References:

  "Feature Engineering for Machine Learning" by Alice Zheng and Amanda Casari (O'Reilly Media, 2018), especially the sections on handling missing data. Scikit-learn documentation on imputing missing values: https://scikit-learn.org/stable/

  modules/impute.html

• Question 63:

  A data scientist is using Spark ML to engineer features for an exploratory machine learning project.

  They decide they want to standardize their features using the following code block: Upon code review, a colleague expressed concern with the features being standardized prior to splitting the data into a training set and a test set.

  

  Which of the following changes can the data scientist make to address the concern?

  A. Utilize the MinMaxScaler object to standardize the training data according to global minimum and maximum values

  B. Utilize the MinMaxScaler object to standardize the test data according to global minimum and maximum values

  C. Utilize a cross-validation process rather than a train-test split process to remove the need for standardizing data

  D. Utilize the Pipeline API to standardize the training data according to the test data's summary statistics

  E. Utilize the Pipeline API to standardize the test data according to the training data's summary statistics

  Correct Answer: E

  To address the concern about standardizing features prior to splitting the data, the correct approach is to use the Pipeline API to ensure that only the training data's summary statistics are used to standardize the test data. This is achieved by

  fitting the StandardScaler (or any scaler) on the training data and then transforming both the training and test data using the fitted scaler. This approach prevents information leakage from the test data into the model training process and

  ensures that the model is evaluated fairly.

  References:

  Best Practices in Preprocessing in Spark ML (Handling Data Splits and Feature Standardization).

• Question 64:

  Which of the following describes the relationship between native Spark DataFrames and pandas API on Spark DataFrames?

  A. pandas API on Spark DataFrames are single-node versions of Spark DataFrames with additional metadata

  B. pandas API on Spark DataFrames are more performant than Spark DataFrames

  C. pandas API on Spark DataFrames are made up of Spark DataFrames and additional metadata

  D. pandas API on Spark DataFrames are less mutable versions of Spark DataFrames

  Correct Answer: C

  The pandas API on Spark DataFrames are made up of Spark DataFrames with additional metadata. The pandas API on Spark aims to provide the pandas-like experience with the scalability and distributed nature of Spark. It allows users to

  work with pandas functions on large datasets by leveraging Spark's underlying capabilities.

  References:

  Databricks documentation on pandas API on Spark: pandas API on Spark

• Question 65:

  A data scientist has developed a random forest regressor rfr and included it as the final stage in a Spark MLPipeline pipeline. They then set up a cross-validation process with pipeline as the estimator in the following code block:

  

  Which of the following is a negative consequence of includingpipelineas the estimator in the cross-validation process rather thanrfras the estimator?

  A. The process will have a longer runtime because all stages of pipeline need to be refit or retransformed with each mode

  B. The process will leak data from the training set to the test set during the evaluation phase

  C. The process will be unable to parallelize tuning due to the distributed nature of pipeline

  D. The process will leak data prep information from the validation sets to the training sets for each model

  Correct Answer: A

  Including the entire pipeline as the estimator in the cross-validation process means that all stages of the pipeline, including data preprocessing steps like string indexing and vector assembling, will be refit or retransformed for each fold of the cross-validation. This results in a longer runtime because each fold requires re-execution of these preprocessing steps, which can be computationally expensive. If only the random forest regressor (rfr) were included as the estimator, the preprocessing steps would be performed once, and only the model fitting would be repeated for each fold, significantly reducing the computational overhead. References: Databricks documentation on cross-validation: Cross Validation

• Question 66:

  A machine learning engineer would like to develop a linear regression model with Spark ML to predict the price of a hotel room. They are using the Spark DataFrametrain_dfto train the model.

  The Spark DataFrametrain_dfhas the following schema:

  

  The machine learning engineer shares the following code block:

  

  Which of the following changes does the machine learning engineer need to make to complete the task?

  A. They need to call the transform method on train df

  B. They need to convert the features column to be a vector

  C. They do not need to make any changes

  D. They need to utilize a Pipeline to fit the model

  E. They need to split thefeaturescolumn out into one column for each feature

  Correct Answer: B

  In Spark ML, the linear regression model expects the feature column to be a vector type. However, if thefeaturescolumn in the DataFrametrain_dfis not already in this format (such as being a column of type UDT or a non-vectorized type), the engineerneeds to convert it to a vector column using a transformer likeVectorAssembler. This is a critical step in preparing the data for modeling as Spark ML models require input features to be combined into a single vector column. References: Spark MLlib documentation forLinearRegression:https://spark.apache.org/docs/latest/ml-classification-regression.html#linear-regression

• Question 67:

  Which of the Spark operations can be used to randomly split a Spark DataFrame into a training DataFrame and a test DataFrame for downstream use?

  A. TrainValidationSplit

  B. DataFrame.where

  C. CrossValidator

  D. TrainValidationSplitModel

  E. DataFrame.randomSplit

  Correct Answer: E

  The correct method to randomly split a Spark DataFrame into training and test sets is by using therandomSplitmethod. This method allows you to specify the proportions for the split as a list of weights and returns multiple DataFrames

  according to those weights. This is directly intended for splitting DataFrames randomly and is the appropriate choice for preparing data for training and testing in machine learning workflows.References:

  Apache Spark DataFrame API documentation (DataFrame Operations:randomSplit).

• Question 68:

  A data scientist has written a data cleaning notebook that utilizes the pandas library, but their colleague has suggested that they refactor their notebook to scale with big data.

  Which of the following approaches can the data scientist take to spend the least amount of time refactoring their notebook to scale with big data?

  A. They can refactor their notebook to process the data in parallel.

  B. They can refactor their notebook to use the PySpark DataFrame API.

  C. They can refactor their notebook to use the Scala Dataset API.

  D. They can refactor their notebook to use Spark SQL.

  E. They can refactor their notebook to utilize the pandas API on Spark.

  Correct Answer: E

  The data scientist can refactor their notebook to utilize the pandas API on Spark (now known aspandas on Spark, formerlyKoalas). This allows for the least amount of changes to the existing pandas-based code while scaling to handle big

  data using Spark's distributed computing capabilities.pandas on Sparkprovides a similar API to pandas, making the transition smoother and faster compared to completely rewriting the code to use PySpark DataFrame API, Scala Dataset API,

  or Spark SQL.References:

  Databricks documentation on pandas API on Spark (formerly Koalas).

• Question 69:

  The implementation of linear regression in Spark ML first attempts to solve the linear regression problem using matrix decomposition, but this method does not scale well to large datasets with a large number of variables.

  Which of the following approaches does Spark ML use to distribute the training of a linear regression model for large data?

  A. Logistic regression

  B. Spark ML cannot distribute linear regression training

  C. Iterative optimization

  D. Least-squares method

  E. Singular value decomposition

  Correct Answer: C

  For large datasets with many variables, Spark ML distributes the training of a linear regression model using iterative optimization methods. Specifically, Spark ML employs algorithms such as Gradient Descent or L-BFGS (Limited-memory

  Broyden璅letcher璆oldfarb璖hanno) to iteratively minimize the loss function. These iterative methods are suitable for distributed computing environments and can handle large-scale data efficiently by partitioning the data across nodes in a

  cluster and performing parallel updates.References:

  Spark MLlib Documentation (Linear Regression with Iterative Optimization).

• Question 70:

  A data scientist has created two linear regression models. The first model uses price as a label variable and the second model uses log(price) as a label variable. When evaluating the RMSE of each model bycomparing the label predictions to the actual price values, the data scientist notices that the RMSE for the second model is much larger than the RMSE of the first model.

  Which of the following possible explanations for this difference is invalid?

  A. The second model is much more accurate than the first model

  B. The data scientist failed to exponentiate the predictions in the second model prior tocomputingthe RMSE

  C. The datascientist failed to take the logof the predictions in the first model prior to computingthe RMSE

  D. The first model is much more accurate than the second model

  E. The RMSE is an invalid evaluation metric for regression problems

  Correct Answer: E

  The Root Mean Squared Error (RMSE) is a standard and widely used metric for evaluating the accuracy of regression models. The statement that it is invalid is incorrect. Here's a breakdown of why the other statements are or are not valid:

  Transformations and RMSE Calculation:If the model predictions were transformed (e.g., using log), they should be converted back to their original scale before calculating RMSE to ensure accuracy in the evaluation. Missteps in this

  conversion process can lead to misleading RMSE values. Accuracy of Models:Without additional information, we can't definitively say which model is more accurate without considering their RMSE values properly scaled back to the original

  price scale.

  Appropriateness of RMSE:RMSE is entirely valid for regression problems as it provides a measure of how accurately a model predicts the outcome, expressed in the same units as the dependent variable.

  References:

  "Applied Predictive Modeling" by Max Kuhn and Kjell Johnson (Springer, 2013), particularly the chapters discussing model evaluation metrics.