Databricks DATABRICKS-MACHINE-LEARNING-ASSOCIATE Online Practice Questions and Exam Preparation

Databricks DATABRICKS-MACHINE-LEARNING-ASSOCIATE Online Questions & Answers

• Question 51:

  A data scientist is using the following code block to tune hyperparameters for a machine learning model:

  

  Which change can they make the above code block to improve the likelihood of a more accurate model?

  A. Increase num_evals to 100
  B. Change fmin() to fmax()
  C. Change sparkTrials() to Trials()
  D. Change tpe.suggest to random.suggest
  A. Increase num_evals to 100

  ---

  Explanation/Reference:

  To improve the likelihood of a more accurate model, the data scientist can increasenum_evalsto 100. Increasing the number of evaluations allows the hyperparameter tuning process to explore a larger search space and evaluate more

  combinations of hyperparameters, which increases the chance of finding a more optimal set of hyperparameters for the model.

  References:

  Databricks documentation on hyperparameter tuning: Hyperparameter Tuning

• Question 52:

  A machine learning engineer wants to parallelize the training of group-specific models using the Pandas Function API. They have developed thetrain_modelfunction, and they want to apply it to each group of DataFramedf.

  They have written the following incomplete code block:

  

  Which of the following pieces of code can be used to fill in the above blank to complete the task?

  A. applyInPandas
  B. mapInPandas
  C. predict
  D. train_model
  E. groupedApplyIn
  B. mapInPandas

  ---

  Explanation/Reference:

  The functionmapInPandasin the PySpark DataFrame API allows for applying a function to each partition of the DataFrame. When working with grouped data,groupbyfollowed by applyInPandasis the correct approach to apply a function to each group as a separate Pandas DataFrame. However, if the function should apply across each partition of the grouped data rather than on each individual group,mapInPandaswould be utilized. Since the code snippet indicates the use ofgroupby, the intent seems to be to applytrain_model on each group specifically, which aligns withapplyInPandas. Thus,applyInPandasis a better fit to ensure that each group generated bygroupbyis processed through the train_modelfunction, preserving the partitioning and grouping integrity. References: PySpark Documentation on applying functions to grouped data:https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.Gro upedData.applyInPandas.html

• Question 53:

  A data scientist wants to use Spark ML to one-hot encode the categorical features in their PySpark DataFramefeatures_df. A list of the names of the string columns is assigned to theinput_columnsvariable.

  They have developed this code block to accomplish this task:

  

  The code block is returning an error.

  Which of the following adjustments does the data scientist need to make to accomplish this task?

  A. They need to specify the method parameter to the OneHotEncoder.
  B. They need to remove the line with the fit operation.
  C. They need to use Stringlndexer prior to one-hot encodinq the features.
  D. They need to useVectorAssemblerprior to one-hot encoding the features.
  C. They need to use Stringlndexer prior to one-hot encodinq the features.

  ---

  Explanation/Reference:

  TheOneHotEncoderin Spark ML requires numerical indices as inputs rather than string labels. Therefore, you need to first convert the string columns to numerical indices usingStringIndexer. After that, you can applyOneHotEncoderto these

  indices.

  Corrected code:

  frompyspark.ml.featureimportStringIndexer, OneHotEncoder# Convert string column to indexindexers = [StringIndexer(inputCol=col,

  outputCol=col+"_index")forcolininput_columns] indexer_model = Pipeline(stages=indexers).fit(features_df) indexed_features_df = indexer_model.transform(features_df)# One-hot encode the indexed columnsohe = OneHotEncoder

  (inputCols=[col+"_index"forcolininput_columns], outputCols=output_columns) ohe_model = ohe.fit(indexed_features_df) ohe_features_df = ohe_model.transform(indexed_features_df)

  References:

  PySpark ML Documentation

• Question 54:

  A data scientist uses 3-fold cross-validation and the following hyperparameter grid when optimizing model hyperparameters via grid search for a classification problem:

  Hyperparameter 1: [2, 5, 10] Hyperparameter 2: [50, 100]

  Which of the following represents the number of machine learning models that can be trained in parallel during this process?

  A. 3
  B. 5
  C. 6
  D. 18
  D. 18

  ---

  Explanation/Reference:

  To determine the number of machine learning models that can be trained in parallel, we need to calculate the total number of combinations of hyperparameters. The given hyperparameter grid includes:

  Hyperparameter 1: [2, 5, 10] (3 values)

  Hyperparameter 2: [50, 100] (2 values)

  The total number of combinations is the product of the number of values for each hyperparameter:3 (values of Hyperparameter 1)? (values of Hyperparameter 2)=63 (value s of Hyperparameter 1)? (values of Hyperparameter 2)=6 With 3-fold

  cross-validation, each combination of hyperparameters will be evaluated 3 times. Thus, the total number of models trained will

  be:6 (combinations)? (folds)=186 (combinations)? (folds)=18 However, the number of models that can be trained in parallel is equal to the number of hyperparameter combinations, not the total number of models considering cross-validation.

  Therefore, 6 models can be trained in parallel.

  References:

  Databricks documentation on hyperparameter tuning: Hyperparameter Tuning

• Question 55:

  A data scientist has developed a random forest regressor rfr and included it as the final stage in a Spark MLPipeline pipeline. They then set up a cross-validation process with pipeline as the estimator in the following code block:

  

  Which of the following is a negative consequence of includingpipelineas the estimator in the cross-validation process rather thanrfras the estimator?

  A. The process will have a longer runtime because all stages of pipeline need to be refit or retransformed with each mode
  B. The process will leak data from the training set to the test set during the evaluation phase
  C. The process will be unable to parallelize tuning due to the distributed nature of pipeline
  D. The process will leak data prep information from the validation sets to the training sets for each model
  A. The process will have a longer runtime because all stages of pipeline need to be refit or retransformed with each mode

  ---

  Explanation/Reference:

  Including the entire pipeline as the estimator in the cross-validation process means that all stages of the pipeline, including data preprocessing steps like string indexing and vector assembling, will be refit or retransformed for each fold of the cross-validation. This results in a longer runtime because each fold requires re-execution of these preprocessing steps, which can be computationally expensive. If only the random forest regressor (rfr) were included as the estimator, the preprocessing steps would be performed once, and only the model fitting would be repeated for each fold, significantly reducing the computational overhead. References: Databricks documentation on cross-validation: Cross Validation

• Question 56:

  A machine learning engineer would like to develop a linear regression model with Spark ML to predict the price of a hotel room. They are using the Spark DataFrametrain_dfto train the model.

  The Spark DataFrametrain_dfhas the following schema:

  

  The machine learning engineer shares the following code block:

  

  Which of the following changes does the machine learning engineer need to make to complete the task?

  A. They need to call the transform method on train df
  B. They need to convert the features column to be a vector
  C. They do not need to make any changes
  D. They need to utilize a Pipeline to fit the model
  E. They need to split thefeaturescolumn out into one column for each feature
  B. They need to convert the features column to be a vector

  ---

  Explanation/Reference:

  In Spark ML, the linear regression model expects the feature column to be a vector type. However, if thefeaturescolumn in the DataFrametrain_dfis not already in this format (such as being a column of type UDT or a non-vectorized type), the engineerneeds to convert it to a vector column using a transformer likeVectorAssembler. This is a critical step in preparing the data for modeling as Spark ML models require input features to be combined into a single vector column. References: Spark MLlib documentation forLinearRegression:https://spark.apache.org/docs/latest/ml-classification-regression.html#linear-regression

• Question 57:

  A data scientist has produced three new models for a single machine learning problem. In the past, the solution used just one model. All four models have nearly the same prediction latency, but a machine learning engineer suggests that the new solution will be less time efficient during inference.

  In which situation will the machine learning engineer be correct?

  A. When the new solution requires if-else logic determining which model to use to compute each prediction
  B. When the new solution's models have an average latency that is larger than the size of the original model
  C. When the new solution requires the use of fewer feature variables than the original model
  D. When the new solution requires that each model computes a prediction for every record
  E. When the new solution's models have an average size that is larger than the size of the original model
  D. When the new solution requires that each model computes a prediction for every record

  ---

  Explanation/Reference:

  If the new solution requires that each of the three models computes a prediction for every record, the time efficiency during inference will be reduced. This is because the inference process now involves running multiple models instead of a single model, thereby increasing the overall computation time for each record. In scenarios where inference must be done by multiple models for each record, the latency accumulates, making the process less time efficient compared to using a single model. References: Model Ensemble Techniques

• Question 58:

  A machine learning engineer has created a Feature Table new_table using Feature Store Client fs. When creating the table, they specified a metadata description with key information about the Feature Table. They now want to retrieve that metadata programmatically.

  Which of the following lines of code will return the metadata description?

  A. There is no way to return the metadata description programmatically.
  B. fs.create_training_set("new_table")
  C. fs.get_table("new_table").description
  D. fs.get_table("new_table").load_df()
  E. fs.get_table("new_table")
  C. fs.get_table("new_table").description

  ---

  Explanation/Reference:

  To retrieve the metadata description of a feature table created using the Feature Store Client (referred here asfs), the correct method involves callingget_tableon thefsclient with the table name as an argument, followed by accessing

  thedescription attribute of the returned object. The code snippetfs.get_table("new_table").description correctly achieves this by fetching the table object for "new_table" and then accessing its description attribute, where the metadata is stored.

  The other options do not correctly focus on retrieving the metadata description.References:

  Databricks Feature Store documentation (Accessing Feature Table Metadata).

• Question 59:

  A data scientist has defined a Pandas UDF function predict to parallelize the inference process for a single-node model:

  

  They have written the following incomplete code block to use predict to score each record of Spark DataFramespark_df:

  

  Which of the following lines of code can be used to complete the code block to successfully complete the task?

  A. predict(*spark_df.columns)
  B. mapInPandas(predict)
  C. predict(Iterator(spark_df))
  D. mapInPandas(predict(spark_df.columns))
  E. predict(spark_df.columns)
  B. mapInPandas(predict)

  ---

  Explanation/Reference:

  To apply the Pandas UDFpredictto each record of a Spark DataFrame, you use themapInPandasmethod. This method allows the Pandas UDF to operate on partitions of the DataFrame as pandas DataFrames, applying the specified function

  (predictin this case) to each partition. The correct code completion to execute this is simply mapInPandas(predict), which specifies the UDF to use without additional arguments orincorrect function calls.References:

  PySpark DataFrame documentation (Using mapInPandas with UDFs).

• Question 60:

  A data scientist has developed a linear regression model using Spark ML and computed the predictions in a Spark DataFrame preds_df with the following schema:

  prediction DOUBLE actual DOUBLE

  Which of the following code blocks can be used to compute the root mean-squared-error of the model according to the data in preds_df and assign it to the rmse variable?

  

  A. Option A
  B. Option B
  C. Option C
  D. Option D
  C. Option C

  ---

  Explanation/Reference:

  To compute the root mean-squared-error (RMSE) of a linear regression model using Spark ML, theRegressionEvaluatorclass is used. TheRegressionEvaluator is specifically designed for regression tasks and can calculate various metrics,

  including RMSE, based on the columns containing predictions and actual values. The correct code block to compute RMSE from thepreds_dfDataFrame is:

  regression_evaluator = RegressionEvaluator( predictionCol="prediction", labelCol="actual", metricName="rmse") rmse = regression_evaluator.evaluate(preds_df) This code creates an instance ofRegressionEvaluator, specifying the prediction

  and label columns, as well as the metric to be computed ("rmse"). It then evaluates the predictions in preds_dfand assigns the resulting RMSE value to thermsevariable. Options A and B incorrectly useBinaryClassificationEvaluator, which is

  not suitable for regression tasks. Option D also incorrectly usesBinaryClassificationEvaluator.

  References:

  PySpark ML Documentation