Databricks Databricks Certifications DATABRICKS-MACHINE-LEARNING-ASSOCIATE Questions & Answers

• Question 21:

  A data scientist has produced three new models for a single machine learning problem. In the past, the solution used just one model. All four models have nearly the same prediction latency, but a machine learning engineer suggests that the new solution will be less time efficient during inference.

  In which situation will the machine learning engineer be correct?

  A. When the new solution requires if-else logic determining which model to use to compute each prediction

  B. When the new solution's models have an average latency that is larger than the size of the original model

  C. When the new solution requires the use of fewer feature variables than the original model

  D. When the new solution requires that each model computes a prediction for every record

  E. When the new solution's models have an average size that is larger than the size of the original model

  Correct Answer: D

  If the new solution requires that each of the three models computes a prediction for every record, the time efficiency during inference will be reduced. This is because the inference process now involves running multiple models instead of a single model, thereby increasing the overall computation time for each record. In scenarios where inference must be done by multiple models for each record, the latency accumulates, making the process less time efficient compared to using a single model. References: Model Ensemble Techniques

• Question 22:

  A data scientist has developed a machine learning pipeline with a static input data set using Spark ML, but the pipeline is taking too long to process. They increase the number of workers in the cluster to get the pipeline to run more efficiently. They notice that the number of rows in the training set after reconfiguring the cluster is different from the number of rows in the training set prior to reconfiguring the cluster.

  Which of the following approaches will guarantee a reproducible training and test set for each model?

  A. Manually configure the cluster

  B. Write out the split data sets to persistent storage

  C. Set a speed in the data splitting operation

  D. Manually partition the input data

  Correct Answer: B

  To ensure reproducible training and test sets, writing the split data sets to persistent storage is a reliable approach. This allows you to consistently load the same training and test data for each model run, regardless of cluster reconfiguration

  or other changes in the environment.

  Correct approach:

  Split the data.

  Write the split data to persistent storage (e.g., HDFS, S3). Load the data from storage for each model training session. train_df, test_df = spark_df.randomSplit([0.8,0.2], seed=42) train_df.write.parquet("path/to/train_df.parquet")

  test_df.write.parquet("path/to/test_df.parquet")# Later, load the datatrain_df = spark.read.parquet("path/to/train_df.parquet") test_df = spark.read.parquet("path/to/test_df.parquet")

  References:

  Spark DataFrameWriter Documentation

• Question 23:

  A data scientist has defined a Pandas UDF function predict to parallelize the inference process for a single-node model:

  

  They have written the following incomplete code block to use predict to score each record of Spark DataFramespark_df:

  

  Which of the following lines of code can be used to complete the code block to successfully complete the task?

  A. predict(*spark_df.columns)

  B. mapInPandas(predict)

  C. predict(Iterator(spark_df))

  D. mapInPandas(predict(spark_df.columns))

  E. predict(spark_df.columns)

  Correct Answer: B

  To apply the Pandas UDFpredictto each record of a Spark DataFrame, you use themapInPandasmethod. This method allows the Pandas UDF to operate on partitions of the DataFrame as pandas DataFrames, applying the specified function

  (predictin this case) to each partition. The correct code completion to execute this is simply mapInPandas(predict), which specifies the UDF to use without additional arguments orincorrect function calls.References:

  PySpark DataFrame documentation (Using mapInPandas with UDFs).

• Question 24:

  Which of the following machine learning algorithms typically uses bagging?

  A. IGradient boosted trees

  B. K-means

  C. Random forest

  D. Decision tree

  Correct Answer: C

  Random Forest is a machine learning algorithm that typically uses bagging (Bootstrap Aggregating). Bagging is a technique that involves training multiple base models (such as decision trees) on different subsets of the data and then combining their predictions to improve overall model performance. Each subset is created by randomly sampling with replacement from the original dataset. The Random Forest algorithm builds multiple decision trees and merges them to get a more accurate and stable prediction. References: Databricks documentation on Random Forest: Random Forest in Spark ML

• Question 25:

  A machine learning engineer is converting a decision tree from sklearn to Spark ML. They notice that they are receiving different results despite all of their data and manually specified hyperparameter values being identical.

  Which of the following describes a reason that the single-node sklearn decision tree and the Spark ML decision tree can differ?

  A. Spark ML decision trees test every feature variable in the splitting algorithm

  B. Spark ML decision trees automatically prune overfit trees

  C. Spark ML decision trees test more split candidates in the splitting algorithm

  D. Spark ML decision trees test a random sample of feature variables in the splitting algorithm

  E. Spark ML decision trees test binned features values as representative split candidates

  Correct Answer: E

  One reason that results can differ between sklearn and Spark ML decision trees, despite identical data and hyperparameters, is that Spark ML decision trees test binned feature values as representative split candidates. Spark ML uses a

  method called "quantile binning" to reduce the number of potential split points by grouping continuous features into bins. This binning process can lead to different splits compared to sklearn, which tests all possible split points directly. This

  difference in the splitting algorithm can cause variations in the resulting trees.References:

  Spark MLlib Documentation (Decision Trees and Quantile Binning).

• Question 26:

  A machine learning engineer wants to parallelize the training of group-specific models using the Pandas Function API. They have developed thetrain_modelfunction, and they want to apply it to each group of DataFramedf.

  They have written the following incomplete code block:

  

  Which of the following pieces of code can be used to fill in the above blank to complete the task?

  A. applyInPandas

  B. mapInPandas

  C. predict

  D. train_model

  E. groupedApplyIn

  Correct Answer: B

  The functionmapInPandasin the PySpark DataFrame API allows for applying a function to each partition of the DataFrame. When working with grouped data,groupbyfollowed by applyInPandasis the correct approach to apply a function to each group as a separate Pandas DataFrame. However, if the function should apply across each partition of the grouped data rather than on each individual group,mapInPandaswould be utilized. Since the code snippet indicates the use ofgroupby, the intent seems to be to applytrain_model on each group specifically, which aligns withapplyInPandas. Thus,applyInPandasis a better fit to ensure that each group generated bygroupbyis processed through the train_modelfunction, preserving the partitioning and grouping integrity. References: PySpark Documentation on applying functions to grouped data:https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.Gro upedData.applyInPandas.html

• Question 27:

  A data scientist is developing a single-node machine learning model. They have a large number of model configurations to test as a part of their experiment. As a result, the model tuning process takes too long to complete. Which of the following approaches can be used to speed up the model tuning process?

  A. Implement MLflow Experiment Tracking

  B. Scale up with Spark ML

  C. Enable autoscaling clusters

  D. Parallelize with Hyperopt

  Correct Answer: D

  To speed up the model tuning process when dealing with a large number of model configurations, parallelizing the hyperparameter search using Hyperopt is an effective approach. Hyperopt provides tools likeSparkTrialswhich can run

  hyperparameter optimization in parallel across a Spark cluster.

  Example:

  fromhyperoptimportfmin, tpe, hp, SparkTrials search_space = {'x': hp.uniform('x',0,1),'y':

  hp.uniform('y',0,1) }defobjective(params):returnparams['x'] **2+ params['y'] **2spark_trials = SparkTrials(parallelism=4) best = fmin(fn=objective, space=search_space, algo=tpe.suggest, max_evals=100, trials=spark_trials) References:

  Hyperopt Documentation

• Question 28:

  A data scientist is using the following code block to tune hyperparameters for a machine learning model:

  

  Which change can they make the above code block to improve the likelihood of a more accurate model?

  A. Increase num_evals to 100

  B. Change fmin() to fmax()

  C. Change sparkTrials() to Trials()

  D. Change tpe.suggest to random.suggest

  Correct Answer: A

  To improve the likelihood of a more accurate model, the data scientist can increasenum_evalsto 100. Increasing the number of evaluations allows the hyperparameter tuning process to explore a larger search space and evaluate more

  combinations of hyperparameters, which increases the chance of finding a more optimal set of hyperparameters for the model.

  References:

  Databricks documentation on hyperparameter tuning: Hyperparameter Tuning

• Question 29:

  A data scientist is using MLflow to track their machine learning experiment. As a part of each of their MLflow runs, they are performing hyperparameter tuning. The data scientist would like to have one parent run for the tuning process with a child run for each unique combination of hyperparameter values. All parent and child runs are being manually started with mlflow.start_run.

  Which of the following approaches can the data scientist use to accomplish this MLflow run organization?

  A. Theycan turn on Databricks Autologging

  B. Theycan specify nested=True when startingthe child run for each unique combination of hyperparameter values

  C. Theycan start each child run inside the parentrun's indented code block usingmlflow.start runO

  D. They can start each child run with the same experiment ID as the parent run

  E. They can specify nested=True when starting the parent run for the tuningprocess

  Correct Answer: B

  To organize MLflow runs with one parent run for the tuning process and a child run for each unique combination of hyperparameter values, the data scientist can specifynested=Truewhen starting the child run. This approach ensures that

  each child run is properly nested under the parent run, maintaining a clear hierarchical structure for the experiment. This nesting helps in tracking and comparing different hyperparameter combinations within the same tuning

  process.References:

  MLflow Documentation (Managing Nested Runs).

• Question 30:

  An organization is developing a feature repository and is electing to one-hot encode all categorical feature variables. A data scientist suggests that the categorical feature variables should not be one-hot encoded within the feature repository.

  Which of the following explanations justifies this suggestion?

  A. One-hot encoding is a potentially problematic categorical variable strategy for some machine learning algorithms.

  B. One-hot encoding is dependent on the target variable's values which differ for each apaplication.

  C. One-hot encoding is computationally intensive and should only be performed on small samples of training sets for individual machine learning problems.

  D. One-hot encoding is not a common strategy for representing categorical feature variables numerically.

  Correct Answer: A

  The suggestion not to one-hot encode categorical feature variables within the feature repository is justified because one-hot encoding can be problematic for some machine learning algorithms. Specifically, one-hot encoding increases the dimensionality of the data, which can be computationally expensive and may lead to issues such as multicollinearity and overfitting. Additionally, some algorithms, such as tree-based methods, can handle categorical variables directly without requiring one-hot encoding. References: Databricks documentation on feature engineering: Feature Engineering