Databricks DATABRICKS-MACHINE-LEARNING-ASSOCIATE Online Practice Questions and Exam Preparation

Databricks DATABRICKS-MACHINE-LEARNING-ASSOCIATE Online Questions & Answers

• Question 21:

  A machine learning engineer has grown tired of needing to install the MLflow Python library on each of their clusters. They ask a senior machine learning engineer how their notebooks can load the MLflow library without installing it each time. The senior machine learning engineer suggests that they use Databricks Runtime for Machine Learning.

  Which of the following approaches describes how the machine learning engineer can begin using Databricks Runtime for Machine Learning?

  A. They can add a line enabling Databricks Runtime ML in their init script when creating their clusters.
  B. They can check the Databricks Runtime ML box when creating their clusters.
  C. They can select a Databricks Runtime ML version from the Databricks Runtime Version dropdown when creating their clusters.
  D. They can set the runtime-version variable in their Spark session to "ml".
  C. They can select a Databricks Runtime ML version from the Databricks Runtime Version dropdown when creating their clusters.

  ---

  Explanation/Reference:

  The Databricks Runtime for Machine Learning includes pre-installed packages and libraries essential for machine learning and deep learning, including MLflow. To use it, the machine learning engineer can simply select an appropriate

  Databricks Runtime ML version from the "Databricks Runtime Version" dropdown menu while creating their cluster. This selection ensures that all necessary machine learning libraries, including MLflow, are pre-installed and ready for use,

  avoiding the need to manually install them each time.

  References:

  Databricks documentation on creating clusters:

  https://docs.databricks.com/clusters/create.html

• Question 22:

  The implementation of linear regression in Spark ML first attempts to solve the linear regression problem using matrix decomposition, but this method does not scale well to large datasets with a large number of variables.

  Which of the following approaches does Spark ML use to distribute the training of a linear regression model for large data?

  A. Logistic regression
  B. Spark ML cannot distribute linear regression training
  C. Iterative optimization
  D. Least-squares method
  E. Singular value decomposition
  C. Iterative optimization

  ---

  Explanation/Reference:

  For large datasets with many variables, Spark ML distributes the training of a linear regression model using iterative optimization methods. Specifically, Spark ML employs algorithms such as Gradient Descent or L-BFGS (Limited-memory

  Broyden璅letcher璆oldfarb璖hanno) to iteratively minimize the loss function. These iterative methods are suitable for distributed computing environments and can handle large-scale data efficiently by partitioning the data across nodes in a

  cluster and performing parallel updates.References:

  Spark MLlib Documentation (Linear Regression with Iterative Optimization).

• Question 23:

  A health organization is developing a classification model to determine whether or not a patient currently has a specific type of infection. The organization's leaders want to maximize the number of positive cases identified by the model.

  Which of the following classification metrics should be used to evaluate the model?

  A. RMSE
  B. Precision
  C. Area under the residual operating curve
  D. Accuracy
  E. Recall
  E. Recall

  ---

  Explanation/Reference:

  When the goal is to maximize the identification of positive cases in a classification task, the metric of interest isRecall. Recall, also known as sensitivity, measures the proportion of actual positives that are correctly identified by the model (i.e.,

  the true positive rate). It is crucial for scenarios where missing a positive case (false negative) has serious implications, such as in medical diagnostics. The other metrics like Precision, RMSE, and Accuracy serve different aspects of

  performance measurement and are not specifically focused on maximizing the detection of positive cases alone.

  References:

  Classification Metrics in Machine Learning (Understanding Recall).

• Question 24:

  A data scientist is using Spark ML to engineer features for an exploratory machine learning project.

  They decide they want to standardize their features using the following code block: Upon code review, a colleague expressed concern with the features being standardized prior to splitting the data into a training set and a test set.

  

  Which of the following changes can the data scientist make to address the concern?

  A. Utilize the MinMaxScaler object to standardize the training data according to global minimum and maximum values
  B. Utilize the MinMaxScaler object to standardize the test data according to global minimum and maximum values
  C. Utilize a cross-validation process rather than a train-test split process to remove the need for standardizing data
  D. Utilize the Pipeline API to standardize the training data according to the test data's summary statistics
  E. Utilize the Pipeline API to standardize the test data according to the training data's summary statistics
  E. Utilize the Pipeline API to standardize the test data according to the training data's summary statistics

  ---

  Explanation/Reference:

  To address the concern about standardizing features prior to splitting the data, the correct approach is to use the Pipeline API to ensure that only the training data's summary statistics are used to standardize the test data. This is achieved by

  fitting the StandardScaler (or any scaler) on the training data and then transforming both the training and test data using the fitted scaler. This approach prevents information leakage from the test data into the model training process and

  ensures that the model is evaluated fairly.

  References:

  Best Practices in Preprocessing in Spark ML (Handling Data Splits and Feature Standardization).

• Question 25:

  A data scientist is developing a single-node machine learning model. They have a large number of model configurations to test as a part of their experiment. As a result, the model tuning process takes too long to complete. Which of the following approaches can be used to speed up the model tuning process?

  A. Implement MLflow Experiment Tracking
  B. Scale up with Spark ML
  C. Enable autoscaling clusters
  D. Parallelize with Hyperopt
  D. Parallelize with Hyperopt

  ---

  Explanation/Reference:

  To speed up the model tuning process when dealing with a large number of model configurations, parallelizing the hyperparameter search using Hyperopt is an effective approach. Hyperopt provides tools likeSparkTrialswhich can run

  hyperparameter optimization in parallel across a Spark cluster.

  Example:

  fromhyperoptimportfmin, tpe, hp, SparkTrials search_space = {'x': hp.uniform('x',0,1),'y':

  hp.uniform('y',0,1) }defobjective(params):returnparams['x'] **2+ params['y'] **2spark_trials = SparkTrials(parallelism=4) best = fmin(fn=objective, space=search_space, algo=tpe.suggest, max_evals=100, trials=spark_trials) References:

  Hyperopt Documentation

• Question 26:

  A data scientist has developed a machine learning pipeline with a static input data set using Spark ML, but the pipeline is taking too long to process. They increase the number of workers in the cluster to get the pipeline to run more efficiently. They notice that the number of rows in the training set after reconfiguring the cluster is different from the number of rows in the training set prior to reconfiguring the cluster.

  Which of the following approaches will guarantee a reproducible training and test set for each model?

  A. Manually configure the cluster
  B. Write out the split data sets to persistent storage
  C. Set a speed in the data splitting operation
  D. Manually partition the input data
  B. Write out the split data sets to persistent storage

  ---

  Explanation/Reference:

  To ensure reproducible training and test sets, writing the split data sets to persistent storage is a reliable approach. This allows you to consistently load the same training and test data for each model run, regardless of cluster reconfiguration

  or other changes in the environment.

  Correct approach:

  Split the data.

  Write the split data to persistent storage (e.g., HDFS, S3). Load the data from storage for each model training session. train_df, test_df = spark_df.randomSplit([0.8,0.2], seed=42) train_df.write.parquet("path/to/train_df.parquet")

  test_df.write.parquet("path/to/test_df.parquet")# Later, load the datatrain_df = spark.read.parquet("path/to/train_df.parquet") test_df = spark.read.parquet("path/to/test_df.parquet")

  References:

  Spark DataFrameWriter Documentation

• Question 27:

  A data scientist has written a data cleaning notebook that utilizes the pandas library, but their colleague has suggested that they refactor their notebook to scale with big data.

  Which of the following approaches can the data scientist take to spend the least amount of time refactoring their notebook to scale with big data?

  A. They can refactor their notebook to process the data in parallel.
  B. They can refactor their notebook to use the PySpark DataFrame API.
  C. They can refactor their notebook to use the Scala Dataset API.
  D. They can refactor their notebook to use Spark SQL.
  E. They can refactor their notebook to utilize the pandas API on Spark.
  E. They can refactor their notebook to utilize the pandas API on Spark.

  ---

  Explanation/Reference:

  The data scientist can refactor their notebook to utilize the pandas API on Spark (now known aspandas on Spark, formerlyKoalas). This allows for the least amount of changes to the existing pandas-based code while scaling to handle big

  data using Spark's distributed computing capabilities.pandas on Sparkprovides a similar API to pandas, making the transition smoother and faster compared to completely rewriting the code to use PySpark DataFrame API, Scala Dataset API,

  or Spark SQL.References:

  Databricks documentation on pandas API on Spark (formerly Koalas).

• Question 28:

  A machine learning engineer wants to parallelize the inference of group-specific models using the Pandas Function API. They have developed theapply_modelfunction that will look up and load the correct model for each group, and they want to apply it to each group of DataFramedf.

  They have written the following incomplete code block:

  

  Which piece of code can be used to fill in the above blank to complete the task?

  A. applyInPandas
  B. groupedApplyInPandas
  C. mapInPandas
  D. predict
  A. applyInPandas

  ---

  Explanation/Reference:

  To parallelize the inference of group-specific models using the Pandas Function API in PySpark, you can use theapplyInPandasfunction. This function allows you to apply a Python function on each group of a DataFrame and return a

  DataFrame, leveraging the power of pandas UDFs (user-defined functions) for better performance.

  prediction_df = ( df.groupby("device_id") .applyInPandas(apply_model, schema=apply_return_schema) )

  In this code:

  groupby("device_id"): Groups the DataFrame by the "device_id" column. applyInPandas(apply_model, schema=apply_return_schema): Applies theapply_modelfunction to each group and specifies the schema of the return DataFrame.

  References:

  PySpark Pandas UDFs Documentation

• Question 29:

  Which of the following describes the relationship between native Spark DataFrames and pandas API on Spark DataFrames?

  A. pandas API on Spark DataFrames are single-node versions of Spark DataFrames with additional metadata
  B. pandas API on Spark DataFrames are more performant than Spark DataFrames
  C. pandas API on Spark DataFrames are made up of Spark DataFrames and additional metadata
  D. pandas API on Spark DataFrames are less mutable versions of Spark DataFrames
  E. pandas API on Spark DataFrames are unrelated to Spark DataFrames
  C. pandas API on Spark DataFrames are made up of Spark DataFrames and additional metadata

  ---

  Explanation/Reference:

  Pandas API on Spark (previously known as Koalas) provides a pandas-like API on top of Apache Spark. It allows users to perform pandas operations on large datasets using Spark's distributed compute capabilities. Internally, it uses Spark DataFrames and adds metadata that facilitates handling operations in a pandas-like manner, ensuring compatibility and leveraging Spark's performance and scalability.

  References: pandas API on Spark documentation:https://spark.apache.org/docs/latest/api/python/user_guide/pandas _on_spark/index.html

• Question 30:

  A machine learning engineer is converting a decision tree from sklearn to Spark ML. They notice that they are receiving different results despite all of their data and manually specified hyperparameter values being identical.

  Which of the following describes a reason that the single-node sklearn decision tree and the Spark ML decision tree can differ?

  A. Spark ML decision trees test every feature variable in the splitting algorithm
  B. Spark ML decision trees automatically prune overfit trees
  C. Spark ML decision trees test more split candidates in the splitting algorithm
  D. Spark ML decision trees test a random sample of feature variables in the splitting algorithm
  E. Spark ML decision trees test binned features values as representative split candidates
  E. Spark ML decision trees test binned features values as representative split candidates

  ---

  Explanation/Reference:

  One reason that results can differ between sklearn and Spark ML decision trees, despite identical data and hyperparameters, is that Spark ML decision trees test binned feature values as representative split candidates. Spark ML uses a

  method called "quantile binning" to reduce the number of potential split points by grouping continuous features into bins. This binning process can lead to different splits compared to sklearn, which tests all possible split points directly. This

  difference in the splitting algorithm can cause variations in the resulting trees.References:

  Spark MLlib Documentation (Decision Trees and Quantile Binning).