Databricks Databricks Certifications DATABRICKS-MACHINE-LEARNING-ASSOCIATE Questions & Answers

• Question 31:

  A data scientist wants to parallelize the training of trees in a gradient boosted tree to speed up the training process. A colleague suggests that parallelizing a boosted tree algorithm can be difficult.

  Which of the following describes why?

  A. Gradient boosting is not a linear algebra-based algorithm which is required for parallelization

  B. Gradient boosting requires access to all data at once which cannot happen during parallelization.

  C. Gradient boosting calculates gradients in evaluation metrics using all cores which prevents parallelization.

  D. Gradient boosting is an iterative algorithm that requires information from the previous iteration to perform the next step.

  Correct Answer: D

  Gradient boosting is fundamentally an iterative algorithm where each new tree is built based on the errors of the previous ones. This sequential dependency makes it difficult to parallelize the training of trees in gradient boosting, as each step relies on the results from the preceding step. Parallelization in this context would undermine the core methodology of the algorithm, which depends on sequentially improving the model'sperformance with each iteration.References: Machine Learning Algorithms (Challenges with Parallelizing Gradient Boosting).

  Gradient boosting is an ensemble learning technique that builds models in a sequential manner. Each new model corrects the errors made by the previous ones. This sequential dependency means that each iteration requires the results of the previous iteration to make corrections. Here is a step-by-step explanation of why this makes parallelization challenging: Sequential Nature: Gradient boosting builds one tree at a time. Each tree is trained to correct the residual errors of the previous trees. This requires the model to complete one iteration before starting the next. Dependence on Previous Iterations: The gradient calculation at each step depends on the predictions made by the previous models. Therefore, the model must wait until the previous tree has been fully trained and evaluated before starting to train the next tree. Difficulty in Parallelization: Because of this dependency, it is challenging to parallelize the training process. Unlike algorithms that process data independently in each step (e.g., random forests), gradient boosting cannot easily distribute the work across multiple processors or cores for simultaneous execution. This iterative and dependent nature of the gradient boosting process makes it difficult to parallelize effectively. References: Gradient Boosting Machine Learning Algorithm Understanding Gradient Boosting Machines

• Question 32:

  A data scientist wants to efficiently tune the hyperparameters of a scikit-learn model. They elect to use the Hyperopt library'sfminoperation to facilitate this process. Unfortunately, the final model is not very accurate. The data scientist suspects that there is an issue with theobjective_functionbeing passed as an argument tofmin.

  They use the following code block to create theobjective_function:

  

  Which of the following changes does the data scientist need to make to theirobjective_functionin order to produce a more accurate model?

  A. Add test set validation process

  B. Add a random_state argument to the RandomForestRegressor operation

  C. Remove the mean operation that is wrapping the cross_val_score operation

  D. Replace the r2 return value with-r2

  E. Replace the fmin operation with the fmax operation

  Correct Answer: D

  When using theHyperoptlibrary withfmin, the goal is to find the minimum of the objective function. Since you are usingcross_val_scoreto calculate the R2 score which is a measure of the proportion of the variance for a dependent variable

  that's explained by an independent variable(s) in a regression model, higher values are better. However,fmin seeks to minimize the objective function, so to align withfmin's goal, you should return the negative of the R2 score (-r2). This way,

  by minimizing the negative R2,fminis effectively maximizing the R2 score, which can lead to a more accurate model.

  References:

  Hyperopt Documentation: http://hyperopt.github.io/hyperopt/ Scikit-Learn documentation on model evaluation: https://scikit-learn.org/stable/modules/model_evaluation.html

• Question 33:

  A data scientist wants to tune a set of hyperparameters for a machine learning model. They have wrapped a Spark ML model in the objective functionobjective_functionand they have defined the search spacesearch_space.

  As a result, they have the following code block: Which of the following changes do they need to make to the above code block in order to accomplish the task?

  

  A. Change SparkTrials() to Trials()

  B. Reduce num_evals to be less than 10

  C. Change fmin() to fmax()

  D. Remove the trials=trials argument

  E. Remove the algo=tpe.suggest argument

  Correct Answer: A

  TheSparkTrials()is used to distribute trials of hyperparameter tuning across a Spark cluster. If the environment does not support Spark or if the user prefers not to usedistributed computing for this purpose, switching toTrials()would be

  appropriate.Trials() is the standard class for managing search trials in Hyperopt but does not distribute the computation. If the user is encountering issues withSparkTrials()possibly due to an unsupported configuration or an error in the cluster

  setup, usingTrials()can be a suitable change for running the optimization locally or in a non-distributed manner.

  References:

  Hyperopt documentation: http://hyperopt.github.io/hyperopt/

• Question 34:

  A data scientist is wanting to explore the Spark DataFrame spark_df. The data scientist wants visual histograms displaying the distribution of numeric features to be included in the exploration.

  Which of the following lines of code can the data scientist run to accomplish the task?

  A. spark_df.describe()

  B. dbutils.data(spark_df).summarize()

  C. This task cannot be accomplished in a single line of code.

  D. spark_df.summary()

  E. dbutils.data.summarize (spark_df)

  Correct Answer: E

  To display visual histograms and summaries of the numeric features in a Spark DataFrame, the Databricks utility functiondbutils.data.summarizecan be used. This function provides a comprehensive summary, including visual histograms.

  Correct code:

  dbutils.data.summarize(spark_df)

  Other options likespark_df.describe()andspark_df.summary()provide textual statistical summaries but do not include visual histograms.

  References:

  Databricks Utilities Documentation

• Question 35:

  A machine learning engineer has grown tired of needing to install the MLflow Python library on each of their clusters. They ask a senior machine learning engineer how their notebooks can load the MLflow library without installing it each time. The senior machine learning engineer suggests that they use Databricks Runtime for Machine Learning.

  Which of the following approaches describes how the machine learning engineer can begin using Databricks Runtime for Machine Learning?

  A. They can add a line enabling Databricks Runtime ML in their init script when creating their clusters.

  B. They can check the Databricks Runtime ML box when creating their clusters.

  C. They can select a Databricks Runtime ML version from the Databricks Runtime Version dropdown when creating their clusters.

  D. They can set the runtime-version variable in their Spark session to "ml".

  Correct Answer: C

  The Databricks Runtime for Machine Learning includes pre-installed packages and libraries essential for machine learning and deep learning, including MLflow. To use it, the machine learning engineer can simply select an appropriate

  Databricks Runtime ML version from the "Databricks Runtime Version" dropdown menu while creating their cluster. This selection ensures that all necessary machine learning libraries, including MLflow, are pre-installed and ready for use,

  avoiding the need to manually install them each time.

  References:

  Databricks documentation on creating clusters:

  https://docs.databricks.com/clusters/create.html

• Question 36:

  A health organization is developing a classification model to determine whether or not a patient currently has a specific type of infection. The organization's leaders want to maximize the number of positive cases identified by the model.

  Which of the following classification metrics should be used to evaluate the model?

  A. RMSE

  B. Precision

  C. Area under the residual operating curve

  D. Accuracy

  E. Recall

  Correct Answer: E

  When the goal is to maximize the identification of positive cases in a classification task, the metric of interest isRecall. Recall, also known as sensitivity, measures the proportion of actual positives that are correctly identified by the model (i.e.,

  the true positive rate). It is crucial for scenarios where missing a positive case (false negative) has serious implications, such as in medical diagnostics. The other metrics like Precision, RMSE, and Accuracy serve different aspects of

  performance measurement and are not specifically focused on maximizing the detection of positive cases alone.

  References:

  Classification Metrics in Machine Learning (Understanding Recall).

• Question 37:

  Which of the following approaches can be used to view the notebook that was run to create an MLflow run?

  A. Open the MLmodel artifact in the MLflow run paqe

  B. Click the "Models" link in the row corresponding to the run in the MLflow experiment paqe

  C. Click the "Source" link in the row corresponding to the run in the MLflow experiment page

  D. Click the "Start Time" link in the row corresponding to the run in the MLflow experiment page

  Correct Answer: C

  To view the notebook that was run to create an MLflow run, you can click the "Source" link in the row corresponding to the run in the MLflow experiment page. The "Source" link provides a direct reference to the source notebook or script that

  initiated the run, allowing you to review the code and methodology used in the experiment. This feature is particularly useful for reproducibility and for understanding the context of the experiment.

  References:

  MLflow Documentation (Viewing Run Sources and Notebooks).

• Question 38:

  A data scientist wants to efficiently tune the hyperparameters of a scikit-learn model in parallel. They elect to use the Hyperopt library to facilitate this process.

  Which of the following Hyperopt tools provides the ability to optimize hyperparameters in parallel?

  A. fmin

  B. SparkTrials

  C. quniform

  D. search_space

  E. objective_function

  Correct Answer: B

  TheSparkTrialsclass in the Hyperopt library allows for parallel hyperparameter optimization on a Spark cluster. This enables efficient tuning of hyperparameters by distributing the optimization process across multiple nodes in a cluster.

  fromhyperoptimportfmin, tpe, hp, SparkTrials search_space = {'x': hp.uniform('x',0,1),'y':

  hp.uniform('y',0,1) }defobjective(params):returnparams['x'] **2+ params['y'] **2spark_trials = SparkTrials(parallelism=4) best = fmin(fn=objective, space=search_space, algo=tpe.suggest, max_evals=100, trials=spark_trials) References:

  Hyperopt Documentation

• Question 39:

  A data scientist has a Spark DataFrame spark_df. They want to create a new Spark DataFrame that contains only the rows from spark_df where the value in column price is greater than 0.

  Which of the following code blocks will accomplish this task?

  A. spark_df[spark_df["price"] > 0]

  B. spark_df.filter(col("price") > 0)

  C. SELECT * FROM spark_df WHERE price > 0

  D. spark_df.loc[spark_df["price"] > 0,:]

  E. spark_df.loc[:,spark_df["price"] > 0]

  Correct Answer: B

  To filter rows in a Spark DataFrame based on a condition, you use thefilter method along with a column condition. The correct syntax in PySpark to accomplish this task isspark_df.filter(col("price") > 0), which filters the DataFrame to include

  only those rows where the value in the "price" column is greater than 0. Thecolfunction is used to specify column-based operations. The other options provided either do not use correct Spark DataFrame syntax or are intended for different

  types of data manipulation frameworks like pandas.References:

  PySpark DataFrame API documentation (Filtering DataFrames).

• Question 40:

  A data scientist has produced two models for a single machine learning problem. One of the models performs well when one of the features has a value of less than 5, and the other model performs well when the value of that feature is greater than or equal to 5. The data scientist decides to combine the two models into a single machine learning solution.

  Which of the following terms is used to describe this combination of models?

  A. Bootstrap aggregation

  B. Support vector machines

  C. Bucketing

  D. Ensemble learning

  E. Stacking

  Correct Answer: D

  Ensemble learning is a machine learning technique that involves combining several models to solve a particular problem. The scenario described fits the concept of ensemble learning, where two models, each performing well under different

  conditions, are combined to create a more robust model. This approach often leads to better performance as it combines the strengths of multiple models.

  References:

  Introduction to Ensemble Learning:

  https://machinelearningmastery.com/ensemble-machine-learning-algorithms-python-scikit-learn/