Databricks Databricks Certifications DATABRICKS-MACHINE-LEARNING-ASSOCIATE Questions & Answers

• Question 1:

  Which of the following tools can be used to distribute large-scale feature engineering without the use of a UDF or pandas Function API for machine learning pipelines?

  A. Keras

  B. pandas

  C. PvTorch

  D. Spark ML

  E. Scikit-learn

  Correct Answer: D

  Spark ML (Machine Learning Library) is designed specifically for handling large-scale data processing and machine learning tasks directly within Apache Spark. It provides tools and APIs for large-scale feature engineering without the need to

  rely on user-defined functions (UDFs) or pandas Function API, allowing for more scalable and efficient data transformations directly distributed across a Spark cluster. Unlike Keras, pandas, PyTorch, and scikit-learn, Spark ML operates

  natively in a distributed environment suitable for big data scenarios.References:

  Spark MLlib documentation (Feature Engineering with Spark ML).

• Question 2:

  A data scientist is developing a machine learning pipeline using AutoML on Databricks Machine Learning.

  Which of the following steps will the data scientist need to perform outside of their AutoML experiment?

  A. Model tuning

  B. Model evaluation

  C. Model deployment

  D. Exploratory data analysis

  Correct Answer: D

  AutoML platforms, such as the one available in Databricks Machine Learning, streamline various stages of the machine learning pipeline including feature engineering, model selection, hyperparameter tuning, and model evaluation. However,

  exploratory data analysis (EDA) is typically performed outside the AutoML process. EDA involves understanding the dataset, visualizing distributions, identifying anomalies, and gaining insights into data before feeding it into a machine

  learning pipeline. This step is crucial for ensuring that the data is clean and suitable for model training but is generally done manually by the data scientist.

  References:

  Databricks documentation on AutoML:

  https://docs.databricks.com/applications/machine-learning/automl.html

• Question 3:

  What is the name of the method that transforms categorical features into a series of binary indicator feature variables?

  A. Leave-one-out encoding

  B. Target encoding

  C. One-hot encoding

  D. Categorical

  E. String indexing

  Correct Answer: C

  The method that transforms categorical features into a series of binary indicator variables is known as one-hot encoding. This technique converts each categorical value into a new binary column, which is essential for models that require

  numerical input. One-hot encoding is widely used because it helps to handle categorical data without introducing a false ordinal relationship among categories.References:

  Feature Engineering Techniques (One-Hot Encoding).

• Question 4:

  A new data scientist has started working on an existing machine learning project. The project is a scheduled Job that retrains every day. The project currently exists in a Repo in Databricks. The data scientist has been tasked with improving the feature engineering of the pipeline's preprocessing stage. The data scientist wants to make necessary updates to the code that can be easily adopted into the project without changing what is being run each day.

  Which approach should the data scientist take to complete this task?

  A. They can create a new branch in Databricks, commit their changes, and push those changes to the Git provider.

  B. They can clone the notebooks in the repository into a Databricks Workspace folder and make the necessary changes.

  C. They can create a new Git repository, import it into Databricks, and copy and paste the existing code from the original repository before making changes.

  D. They can clone the notebooks in the repository into a new Databricks Repo and make the necessary changes.

  Correct Answer: A

  The best approach for the data scientist to take in this scenario is to create a new branch in Databricks, commit their changes, and push those changes to the Git provider. This approach allows the data scientist to make updates and

  improvements to the feature engineering part of the preprocessing pipeline without affecting the main codebase that runs daily. By creating a new branch, they can work on their changes in isolation. Once the changes are ready and tested,

  they can be merged back into the main branch through a pull request, ensuring a smooth integration process and allowing for code review and collaboration with other team members.

  References:

  Databricks documentation on Git integration: Databricks Repos

• Question 5:

  Which of the following tools can be used to distribute large-scale feature engineering without the use of a UDF or pandas Function API for machine learning pipelines?

  A. Keras

  B. Scikit-learn

  C. PyTorch

  D. Spark ML

  Correct Answer: D

  Spark MLlib is a machine learning library within Apache Spark that provides scalable and distributed machine learning algorithms. It is designed to work with Spark DataFrames and leverages Spark's distributed computing capabilities to perform large-scale feature engineering and model training without the need for user-defined functions (UDFs) or the pandas Function API. Spark MLlib provides built-in transformations and algorithms that can be applied directly to large datasets. References: Databricks documentation on Spark MLlib: Spark MLlib

• Question 6:

  A machine learning engineer is trying to scale a machine learning pipeline by distributing its feature engineering process.

  Which of the following feature engineering tasks will be the least efficient to distribute?

  A. One-hot encoding categorical features

  B. Target encoding categorical features

  C. Imputing missing feature values with the mean

  D. Imputing missing feature values with the true median

  E. Creating binary indicator features for missing values

  Correct Answer: D

  Among the options listed, calculating the true median for imputing missing feature values is the least efficient to distribute. This is because the true median requires knowledge of the entire data distribution, which can be computationally

  expensive in a distributed environment. Unlike mean or mode, finding the median requires sorting the data or maintaining a full distribution, which is more intensive and often requires shuffling the data across partitions.

  References:

  Challenges in parallel processing and distributed computing for data aggregation like median calculation:https://www.apache.org

• Question 7:

  In which of the following situations is it preferable to impute missing feature values with their median value over the mean value?

  A. When the features are of the categorical type

  B. When the features are of the boolean type

  C. When the features contain a lot of extreme outliers

  D. When the features contain no outliers

  E. When the features contain no missingno values

  Correct Answer: C

  Imputing missing values with the median is often preferred over the mean in scenarios where the data contains a lot of extreme outliers. The median is a more robust measure of central tendency in such cases, as it is not as heavily

  influenced by outliers as the mean. Using the median ensures that the imputed values are more representative of the typical data point, thus preserving the integrity of the dataset's distribution. The other options are not specifically relevant to

  the question of handling outliers in numerical data.

  References:

  Data Imputation Techniques (Dealing with Outliers).

• Question 8:

  A data scientist is attempting to tune a logistic regression model logistic using scikit-learn. They want to specify a search space for two hyperparameters and let the tuning process randomly select values for each evaluation.

  They attempt to run the following code block, but it does not accomplish the desired task:

  

  Which of the following changes can the data scientist make to accomplish the task?

  A. Replace the GridSearchCV operation with RandomizedSearchCV

  B. Replace the GridSearchCV operation with cross_validate

  C. Replace the GridSearchCV operation with ParameterGrid

  D. Replace the random_state=0 argument with random_state=1

  E. Replace the penalty= ['12', '11'] argument with penalty=uniform ('12', '11')

  Correct Answer: A

  The user wants to specify a search space for hyperparameters and let the tuning process randomly select values.GridSearchCVsystematically tries every combination of the provided hyperparameter values, which can be computationally expensive and time-consuming.RandomizedSearchCV, on the other hand, samples hyperparameters from a distribution for a fixed number of iterations. This approach is usually faster and still can find very good parameters, especially when the search space is large or includes distributions. References: Scikit-Learn documentation on hyperparameter tuning: https://scikit-learn.org/stable/modules/grid_search.html#randomized-parameter-optimization

• Question 9:

  A team is developing guidelines on when to use various evaluation metrics for classification problems. The team needs to provide input on when to use the F1 score over accuracy.

  

  Which of the following suggestions should the team include in their guidelines?

  A. The F1 score should be utilized over accuracy when the number of actual positive cases is identical to the number of actual negative cases.

  B. The F1 score should be utilized over accuracy when there are greater than two classes in the target variable.

  C. The F1 score should be utilized over accuracy when there is significant imbalance between positive and negative classes and avoiding false negatives is a priority.

  D. The F1 score should be utilized over accuracy when identifying true positives and true negatives are equally important to the business problem.

  Correct Answer: C

  The F1 score is the harmonic mean of precision and recall and is particularly useful in situations where there is a significant imbalance between positive and negative classes. When there is a class imbalance, accuracy can be misleading

  because a model can achieve high accuracy by simply predicting the majority class. The F1 score, however, provides a better measure of the test's accuracy in terms of both false positives and false negatives.

  Specifically, the F1 score should be used over accuracy when:

  There is a significant imbalance between positive and negative classes. Avoiding false negatives is a priority, meaning recall (the ability to detect all positive instances) is crucial.

  In this scenario, the F1 score balances both precision (the ability to avoid false positives) and recall, providing a more meaningful measure of a model's performance under these conditions.

  References:

  Databricks documentation on classification metrics: Classification Metrics

• Question 10:

  A data scientist has a Spark DataFrame spark_df. They want to create a new Spark DataFrame that contains only the rows from spark_df where the value in column discount is less than or equal 0.

  Which of the following code blocks will accomplish this task?

  A. spark_df.loc[:,spark_df["discount"] <= 0]

  B. spark_df[spark_df["discount"] <= 0]

  C. spark_df.filter (col("discount") <= 0)

  D. spark_df.loc(spark_df["discount"] <= 0, :]

  Correct Answer: C

  To filter rows in a Spark DataFrame based on a condition, thefiltermethod is used. In this case, the condition is that the value in the "discount" column should be less than or equal to 0. The correct syntax uses thefiltermethod along with

  thecolfunction from pyspark.sql.functions.

  Correct code:

  frompyspark.sql.functionsimportcol filtered_df = spark_df.filter(col("discount") <=0) Option A and D use Pandas syntax, which is not applicable in PySpark. Option B is closer but misses the use of thecolfunction.

  References:

  PySpark SQL Documentation