Databricks DATABRICKS-MACHINE-LEARNING-ASSOCIATE Online Practice Questions and Exam Preparation

Databricks DATABRICKS-MACHINE-LEARNING-ASSOCIATE Online Questions & Answers

• Question 61:

  A data scientist has a Spark DataFrame spark_df. They want to create a new Spark DataFrame that contains only the rows from spark_df where the value in column price is greater than 0.

  Which of the following code blocks will accomplish this task?

  A. spark_df[spark_df["price"] > 0]
  B. spark_df.filter(col("price") > 0)
  C. SELECT * FROM spark_df WHERE price > 0
  D. spark_df.loc[spark_df["price"] > 0,:]
  E. spark_df.loc[:,spark_df["price"] > 0]
  B. spark_df.filter(col("price") > 0)

  ---

  Explanation/Reference:

  To filter rows in a Spark DataFrame based on a condition, you use thefilter method along with a column condition. The correct syntax in PySpark to accomplish this task isspark_df.filter(col("price") > 0), which filters the DataFrame to include

  only those rows where the value in the "price" column is greater than 0. Thecolfunction is used to specify column-based operations. The other options provided either do not use correct Spark DataFrame syntax or are intended for different

  types of data manipulation frameworks like pandas.References:

  PySpark DataFrame API documentation (Filtering DataFrames).

• Question 62:

  Which statement describes a Spark ML transformer?

  A. A transformer is an algorithm which can transform one DataFrame into another DataFrame
  B. A transformer is a hyperparameter grid that can be used to train a model
  C. A transformer chains multiple algorithms together to transform an ML workflow
  D. A transformer is a learning algorithm that can use a DataFrame to train a model
  A. A transformer is an algorithm which can transform one DataFrame into another DataFrame

  ---

  Explanation/Reference:

  In Spark ML, a transformer is an algorithm that can transform one DataFrame into another DataFrame. It takes a DataFrame as input and produces a new DataFrame as output. This transformation can involve adding new columns, modifying

  existing ones, or applying feature transformations. Examples of transformers in Spark MLlib include feature transformers likeStringIndexer,VectorAssembler, andStandardScaler.

  References:

  Databricks documentation on transformers: Transformers in Spark ML

• Question 63:

  A new data scientist has started working on an existing machine learning project. The project is a scheduled Job that retrains every day. The project currently exists in a Repo in Databricks. The data scientist has been tasked with improving the feature engineering of the pipeline's preprocessing stage. The data scientist wants to make necessary updates to the code that can be easily adopted into the project without changing what is being run each day.

  Which approach should the data scientist take to complete this task?

  A. They can create a new branch in Databricks, commit their changes, and push those changes to the Git provider.
  B. They can clone the notebooks in the repository into a Databricks Workspace folder and make the necessary changes.
  C. They can create a new Git repository, import it into Databricks, and copy and paste the existing code from the original repository before making changes.
  D. They can clone the notebooks in the repository into a new Databricks Repo and make the necessary changes.
  A. They can create a new branch in Databricks, commit their changes, and push those changes to the Git provider.

  ---

  Explanation/Reference:

  The best approach for the data scientist to take in this scenario is to create a new branch in Databricks, commit their changes, and push those changes to the Git provider. This approach allows the data scientist to make updates and

  improvements to the feature engineering part of the preprocessing pipeline without affecting the main codebase that runs daily. By creating a new branch, they can work on their changes in isolation. Once the changes are ready and tested,

  they can be merged back into the main branch through a pull request, ensuring a smooth integration process and allowing for code review and collaboration with other team members.

  References:

  Databricks documentation on Git integration: Databricks Repos

• Question 64:

  A data scientist uses 3-fold cross-validation when optimizing model hyperparameters for a regression problem. The following root-mean-squared-error values are calculated on each of the validation folds:

  1.

  10.0

  2.

  12.0

  3.

  17.0

  Which of the following values represents the overall cross-validation root-mean-squared error?

  A. 13.0
  B. 17.0
  C. 12.0
  D. 39.0
  E. 10.0
  A. 13.0

  ---

  Explanation/Reference:

  To calculate the overall cross-validation root-mean-squared error (RMSE), you average the RMSE values obtained from each validation fold. Given the RMSE values of 10.0, 12.0, and 17.0 for the three folds, the overall cross-validation

  RMSE is calculated as the average of these three values:

  Overall CV RMSE=10.0+12.0+17.03=39.03=13.0Overall CV RMSE=310.0+12.0+17.0 =339.0=13.0

  Thus, the correct answer is 13.0, which accurately represents the average RMSE across all folds.References:

  Cross-validation in Regression (Understanding Cross-Validation Metrics).

• Question 65:

  A data scientist wants to parallelize the training of trees in a gradient boosted tree to speed up the training process. A colleague suggests that parallelizing a boosted tree algorithm can be difficult.

  Which of the following describes why?

  A. Gradient boosting is not a linear algebra-based algorithm which is required for parallelization
  B. Gradient boosting requires access to all data at once which cannot happen during parallelization.
  C. Gradient boosting calculates gradients in evaluation metrics using all cores which prevents parallelization.
  D. Gradient boosting is an iterative algorithm that requires information from the previous iteration to perform the next step.
  D. Gradient boosting is an iterative algorithm that requires information from the previous iteration to perform the next step.

  ---

  Explanation/Reference:

  Gradient boosting is fundamentally an iterative algorithm where each new tree is built based on the errors of the previous ones. This sequential dependency makes it difficult to parallelize the training of trees in gradient boosting, as each step relies on the results from the preceding step. Parallelization in this context would undermine the core methodology of the algorithm, which depends on sequentially improving the model'sperformance with each iteration.References: Machine Learning Algorithms (Challenges with Parallelizing Gradient Boosting).

  Gradient boosting is an ensemble learning technique that builds models in a sequential manner. Each new model corrects the errors made by the previous ones. This sequential dependency means that each iteration requires the results of the previous iteration to make corrections. Here is a step-by-step explanation of why this makes parallelization challenging: Sequential Nature: Gradient boosting builds one tree at a time. Each tree is trained to correct the residual errors of the previous trees. This requires the model to complete one iteration before starting the next. Dependence on Previous Iterations: The gradient calculation at each step depends on the predictions made by the previous models. Therefore, the model must wait until the previous tree has been fully trained and evaluated before starting to train the next tree. Difficulty in Parallelization: Because of this dependency, it is challenging to parallelize the training process. Unlike algorithms that process data independently in each step (e.g., random forests), gradient boosting cannot easily distribute the work across multiple processors or cores for simultaneous execution. This iterative and dependent nature of the gradient boosting process makes it difficult to parallelize effectively. References: Gradient Boosting Machine Learning Algorithm Understanding Gradient Boosting Machines

• Question 66:

  Which of the following statements describes a Spark ML estimator?

  A. An estimator is a hyperparameter arid that can be used to train a model
  B. An estimator chains multiple alqorithms toqether to specify an ML workflow
  C. An estimator is a trained ML model which turns a DataFrame with features into a DataFrame with predictions
  D. An estimator is an alqorithm which can be fit on a DataFrame to produce a Transformer
  E. An estimator is an evaluation tool to assess to the quality of a model
  D. An estimator is an alqorithm which can be fit on a DataFrame to produce a Transformer

  ---

  Explanation/Reference:

  In the context of Spark MLlib, an estimator refers to an algorithm which can be "fit" on a DataFrame to produce a model (referred to as a Transformer), which can then be used to transform one DataFrame into another, typically adding

  predictions or model scores. This is a fundamental concept in machine learning pipelines in Spark, where the workflow includes fitting estimators to data to produce transformers.

  References:

  Spark MLlib Documentation:https://spark.apache.org/docs/latest/ml-pipeline.html#estimators

• Question 67:

  In which of the following situations is it preferable to impute missing feature values with their median value over the mean value?

  A. When the features are of the categorical type
  B. When the features are of the boolean type
  C. When the features contain a lot of extreme outliers
  D. When the features contain no outliers
  E. When the features contain no missingno values
  C. When the features contain a lot of extreme outliers

  ---

  Explanation/Reference:

  Imputing missing values with the median is often preferred over the mean in scenarios where the data contains a lot of extreme outliers. The median is a more robust measure of central tendency in such cases, as it is not as heavily

  influenced by outliers as the mean. Using the median ensures that the imputed values are more representative of the typical data point, thus preserving the integrity of the dataset's distribution. The other options are not specifically relevant to

  the question of handling outliers in numerical data.

  References:

  Data Imputation Techniques (Dealing with Outliers).

• Question 68:

  Which of the following machine learning algorithms typically uses bagging?

  A. Gradient boosted trees B. K-means
  C. Random forest
  D. Linear regression
  E. Decision tree
  C. Random forest

  ---

  Explanation/Reference:

  Random Forest is a machine learning algorithm that typically uses bagging (Bootstrap Aggregating). Bagging involves training multiple models independently on different random subsets of the data and then combining their predictions.

  Random Forests consist of many decision trees trained on random subsets of the training data and features, and their predictions are averaged to improve accuracy and control overfitting. This method enhances model robustness and

  predictive performance.References:

  Ensemble Methods in Machine Learning (Understanding Bagging and Random Forests).

• Question 69:

  A data scientist learned during their training to always use 5-fold cross-validation in their model development workflow. A colleague suggests that there are cases where a train-validation split could be preferred over k-fold cross-validation

  when k > 2.

  Which of the following describes a potential benefit of using a train-validation split over k-fold cross-validation in this scenario?

  A. A holdout set is not necessary when using a train-validation split
  B. Reproducibility is achievable when using a train-validation split
  C. Fewer hyperparameter values need to be tested when usinga train-validation split
  D. Bias is avoidable when using a train-validation split
  E. Fewer models need to be trained when using a train-validation split
  E. Fewer models need to be trained when using a train-validation split

  ---

  Explanation/Reference:

  A train-validation split is often preferred over k-fold cross-validation (with k >

  2) when computational efficiency is a concern. With a train-validation split, only two models (one on the training set and one on the validation set) are trained, whereas k-fold cross-validation requires training k models (one for each fold). This

  reduction in the number of models trained can save significant computational resources and time, especially when dealing with large datasets or complex models.

  References:

  Model Evaluation with Train-Test Split

• Question 70:

  A data scientist is wanting to explore summary statistics for Spark DataFrame spark_df. The data scientist wants to see the count, mean, standard deviation, minimum, maximum, and interquartile range (IQR) for each numerical feature.

  Which of the following lines of code can the data scientist run to accomplish the task?

  A. spark_df.summary ()
  B. spark_df.stats()
  C. spark_df.describe().head()
  D. spark_df.printSchema()
  E. spark_df.toPandas()
  A. spark_df.summary ()

  ---

  Explanation/Reference:

  Thesummary()function in PySpark's DataFrame API provides descriptive statistics which include count, mean, standard deviation, min, max, and quantiles for numeric columns. Here are the steps on how it can be used:

  Import PySpark:Ensure PySpark is installed and correctly configured in the Databricks environment.

  Load Data:Load the data into a Spark DataFrame.

  Apply Summary:Usespark_df.summary()to generate summary statistics. View Results:The output from thesummary()function includes the statistics specified in the query (count, mean, standard deviation, min, max, and potentially quartiles

  which approximate the interquartile range).

  References:

  PySpark

  Documentation:https://spark.apache.org/docs/latest/api/python/reference/api/pysp ark.sql.DataFrame.summary.html