Databricks DATABRICKS-CERTIFIED-GENERATIVE-AI-ENGINEER-ASSOCIATE Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-GENERATIVE-AI-ENGINEER-ASSOCIATE Online Questions & Answers

• Question 11:

  A Generative AI Engineer is creating an LLM-powered application that will need access to up-to-date news articles and stock prices.

  The design requires the use of stock prices which are stored in Delta tables and finding the latest relevant news articles by searching the internet. How should the Generative AI Engineer architect their LLM system?

  A. Use an LLM to summarize the latest news articles and lookup stock tickers from the summaries to find stock prices.
  B. Query the Delta table for volatile stock prices and use an LLM to generate a search query to investigate potential causes of the stock volatility.
  C. Download and store news articles and stock price information in a vector store. Use a RAG architecture to retrieve and generate at runtime.
  D. Create an agent with tools for SQL querying of Delta tables and web searching, provide retrieved values to an LLM for generation of response.
  D. Create an agent with tools for SQL querying of Delta tables and web searching, provide retrieved values to an LLM for generation of response.

  ---

  Explanation

  To build an LLM-powered system that accesses up-to-date news articles and stock prices, the best approach is to create an agent that has access to specific tools (option D).

  Agent with SQL and Web Search Capabilities:By using an agent-based architecture, the LLM can interact with external tools. The agent can query Delta tables (for up-to-date stock prices) via SQL and perform web searches to retrieve the

  latest news articles. This modular approach ensures the system can access both structured (stock prices) and unstructured (news) data sources dynamically.

  Why This Approach Works:

  SQL Queries for Stock Prices: Delta tables store stock prices, which the agent can query directly for the latest data.

  Web Search for News: For news articles, the agent can generate search queries and retrieve the most relevant and recent articles, then pass them to the LLM for processing.

  Why Other Options Are Less Suitable:

  A (Summarizing News for Stock Prices): This convoluted approach would not ensure accuracy when retrieving stock prices, which are already structured and stored in Delta tables.

  B (Stock Price Volatility Queries): While this could retrieve relevant information, it doesn't address how to obtain the most up-to-date news articles.

  C (Vector Store): Storing news articles and stock prices in a vector store might not capture the real-time nature of stock data and news updates, as it relies on pre-existing data rather than dynamic querying.

  Thus, using an agent with access to both SQL for querying stock prices and web search for retrieving news articles is the best approach for ensuring up-to-date and accurate responses.

• Question 12:

  A Generative Al Engineer is building a system that will answer questions on currently unfolding news topics. As such, it pulls information from a variety of sources including articles and social media posts. They are concerned about toxic posts on social media causing toxic outputs from their system.

  Which guardrail will limit toxic outputs?

  A. Use only approved social media and news accounts to prevent unexpected toxic data from getting to the LLM.
  B. Implement rate limiting
  C. Reduce the amount of context Items the system will Include in consideration for its response.
  D. Log all LLM system responses and perform a batch toxicity analysis monthly.
  A. Use only approved social media and news accounts to prevent unexpected toxic data from getting to the LLM.

  ---

  Explanation

• Question 13:

  A Generative Al Engineer is using an LLM to classify species of edible mushrooms based on text descriptions of certain features. The model is returning accurate responses in testing and the Generative Al Engineer is confident they have the correct list of possible labels, but the output frequently contains additional reasoning in the answer when the Generative Al Engineer only wants to return the label with no additional text.

  Which action should they take to elicit the desired behavior from this LLM?

  A. Use few snot prompting to instruct the model on expected output format
  B. Use zero shot prompting to instruct the model on expected output format
  C. Use zero shot chain-of-thought prompting to prevent a verbose output format
  D. Use a system prompt to instruct the model to be succinct in its answer
  D. Use a system prompt to instruct the model to be succinct in its answer

  ---

  Explanation

• Question 14:

  A Generative Al Engineer has already trained an LLM on Databricks and it is now ready to be deployed. Which of the following steps correctly outlines the easiest process for deploying a model on Databricks?

  A. Log the model as a pickle object, upload the object to Unity Catalog Volume, register it to Unity Catalog using MLflow, and start a serving endpoint
  B. Log the model using MLflow during training, directly register the model to Unity Catalog using the MLflow API, and start a serving endpoint
  C. Save the model along with its dependencies in a local directory, build the Docker image, and run the Docker container
  D. Wrap the LLM's prediction function into a Flask application and serve using Gunicorn
  B. Log the model using MLflow during training, directly register the model to Unity Catalog using the MLflow API, and start a serving endpoint

  ---

  Explanation

  Problem Context: The goal is to deploy a trained LLM on Databricks in the simplest and most integrated manner.

  Explanation of Options:

  Option A: This method involves unnecessary steps like logging the model as a pickle object, which is not the most efficient path in a Databricks environment.

  Option B: Logging the model with MLflow during training and then using MLflow's API to register and start serving the model is straightforward and leverages Databricks' built-in functionalities for seamless model deployment.

  Option C: Building and running a Docker container is a complex and less integrated approach within the Databricks ecosystem.

  Option D: Using Flask and Gunicorn is a more manual approach and less integrated compared to the native capabilities of Databricks and MLflow.

  Option provides the most straightforward and efficient process, utilizing Databricks' ecosystem to its full B advantage for deploying models.

• Question 15:

  A Generative Al Engineer is tasked with developing a RAG application that will help a small internal group of experts at their company answer specific questions, augmented by an internal knowledge base. They want the best possible quality in the answers, and neither latency nor throughput is a huge concern given that the user group is small and they're willing to wait for the best answer. The topics are sensitive in nature and the data is highly confidential and so, due to regulatory requirements, none of the information is allowed to be transmitted to third parties.

  Which model meets all the Generative Al Engineer's needs in this situation?

  A. Dolly 1.5B
  B. OpenAI GPT-4
  C. BGE-large
  D. Llama2-70B
  C. BGE-large

  ---

  Explanation

  Problem Context : The Generative AI Engineer needs a model for a Retrieval-Augmented Generation (RAG) application that provides high-quality answers, where latency and throughput are not major concerns. The key factors are

  confidentiality and sensitivity of the data, as well as the requirement for all processing to be confined to internal resources without external data transmission.

  Explanation of Options :

  Option A: Dolly 1.5B: This model does not typically support RAG applications as it's more focused on image generation tasks.

  Option B: OpenAI GPT-4: While GPT-4 is powerful for generating responses, its standard deployment involves cloud-based processing, which could violate the confidentiality requirements due to external data transmission.

  Option C: BGE-large: The BGE (Big Green Engine) large model is a suitable choice if it is configured to operate on-premises or within a secure internal environment that meets regulatory requirements. Assuming this setup, BGE-large can

  provide high-quality answers while ensuring that data is not transmitted to third parties, thus aligning with the project's sensitivity and confidentiality needs.

  Option D: Llama2-70B: Similar to GPT-4, unless specifically set up for on-premises use, it generally relies on cloud-based services, which might risk confidential data exposure.

  Given the sensitivity and confidentiality concerns, BGE-large is assumed to be configurable for secure internal use, making it the optimal choice for this scenario.

• Question 16:

  A Generative Al Engineer has developed an LLM application to answer questions about internal company policies. The Generative AI Engineer must ensure that the application doesn't hallucinate or leak confidential data. Which approach should NOT be used to mitigate hallucination or confidential data leakage?

  A. Add guardrails to filter outputs from the LLM before it is shown to the user
  B. Fine-tune the model on your data, hoping it will learn what is appropriate and not
  C. Limit the data available based on the user's access level
  D. Use a strong system prompt to ensure the model aligns with your needs.
  B. Fine-tune the model on your data, hoping it will learn what is appropriate and not

  ---

  Explanation

  When addressing concerns of hallucination and data leakage in an LLM application for internal company policies, fine-tuning the model on internal data with the hope it learns data boundaries can be problematic:

  Risk of Data Leakage: Fine-tuning on sensitive or confidential data does not guarantee that the model will not inadvertently include or reference this data in its outputs. There's a risk of overfitting to the specific data details, which might lead to unintended leakage.

  Hallucination: Fine-tuning does not necessarily mitigate the model's tendency to hallucinate; in fact, it might exacerbate it if the training data is not comprehensive or representative of all potential queries. Better Approaches:

  A, , and involve setting up operational safeguards and constraints that directly address data leakage C D and ensure responses are aligned with specific user needs and security levels. Fine-tuning lacks the targeted control needed for such sensitive applications and can introduce new risks, making it an unsuitable approach in this context.

• Question 17:

  A Generative Al Engineer is building a system which will answer questions on latest stock news articles.

  Which will NOT help with ensuring the outputs are relevant to financial news?

  A. Implement a comprehensive guardrail framework that includes policies for content filters tailored to the finance sector.
  B. Increase the compute to improve processing speed of questions to allow greater relevancy analysis C Implement a profanity filter to screen out offensive language
  C. Incorporate manual reviews to correct any problematic outputs prior to sending to the users
  B. Increase the compute to improve processing speed of questions to allow greater relevancy analysis C Implement a profanity filter to screen out offensive language

  ---

  Explanation

  In the context of ensuring that outputs are relevant to financial news, increasing compute power (option B) does not directly improve the relevance of the LLM-generated outputs. Here's why:

  Compute Power and Relevancy:Increasing compute power can help the model process inputs faster, but it does not inherentlyimprove the relevance of the answers. Relevancy depends on the data sources, the retrieval method, and the filtering mechanisms in place, not on how quickly the model processes the query. What Actually Helps with Relevance:Other methods, like content filtering, guardrails, or manual review, can directly impact the relevance of the model's responses by ensuring the model focuses on pertinent financial content. These methods

  help tailor the LLM's responses to the financial domain and avoid irrelevant or harmful outputs.

  Why Other Options Are More Relevant:

  A (Comprehensive Guardrail Framework): This will ensure that the model avoids generating content that is irrelevant or inappropriate in the finance sector.

  C (Profanity Filter): While not directly related to financial relevancy, ensuring the output is clean and professional is still important in maintaining the quality of responses.

  D (Manual Review): Incorporating human oversight to catch and correct issues with the LLM's output ensures the final answers are aligned with financial content expectations.

  Thus, increasing compute power does not help with ensuring the outputs are more relevant to financial news, making option B the correct answer.

• Question 18:

  A Generative AI Engineer is developing a patient-facing healthcare-focused chatbot. If the patient's question is not a medical emergency, the chatbot should solicit more information from the patient to pass to the doctor' s office and suggest a

  few relevant pre-approved medical articles for reading. If the patient's question is urgent, direct the patient to calling their local emergency services.

  Given the following user input:

  "I have been experiencing severe headaches and dizziness for the past two days."

  Which response is most appropriate for the chatbot to generate?

  A. Here are a few relevant articles for your browsing. Let me know if you have questions after reading them.
  B. Please call your local emergency services.
  C. Headaches can be tough. Hope you feel better soon!
  D. Please provide your age, recent activities, and any other symptoms you have noticed along with your headaches and dizziness.
  B. Please call your local emergency services.

  ---

  Explanation

  Problem Context: The task is to design responses for a healthcare-focused chatbot that appropriately addresses the urgency of a patient's symptoms.

  Explanation of Options:

  Option A: Suggesting articles might be suitable for less urgent inquiries but is inappropriate for symptoms that could indicate a serious condition.

  Option B: Given the description of severe symptoms like headaches and dizziness, directing the patient to emergency services is prudent. This aligns with medical guidelines that recommend immediate professional attention for such severe

  symptoms.

  Option C: Offering well-wishes does not address the potential seriousness of the symptoms and lacks appropriate action.

  Option D: While gathering more information is part of a detailed assessment, the immediate need here suggests a more urgent response.

  Given the potential severity of the described symptoms, Option B is the most appropriate, ensuring the chatbot directs patients to seek urgent care when needed, potentially saving lives.

• Question 19:

  A Generative Al Engineer is building a production-ready LLM system which replies directly to customers. The solution makes use of the Foundation Model API via provisioned throughput. They are concerned that the LLM could potentially

  respond in a toxic or otherwise unsafe way. They also wish to perform this with the least amount of effort.

  Which approach will do this?

  A. Host Llama Guard on Foundation Model API and use it to detect unsafe responses
  B. Add some LLM calls to their chain to detect unsafe content before returning text
  C. Add a regex expression on inputs and outputs to detect unsafe responses.
  D. Ask users to report unsafe responses
  A. Host Llama Guard on Foundation Model API and use it to detect unsafe responses

  ---

  Explanation

• Question 20:

  Generative AI Engineer at an electronics company just deployed a RAG application for customers to ask questions about products that the company carries. However, they received feedback that the RAG response often returns information about an irrelevant product.

  What can the engineer do to improve the relevance of the RAG's response?

  A. Assess the quality of the retrieved context
  B. Implement caching for frequently asked questions
  C. Use a different LLM to improve the generated response
  D. Use a different semantic similarity search algorithm
  A. Assess the quality of the retrieved context

  ---

  Explanation

  In a Retrieval-Augmented Generation (RAG) system, the key to providing relevant responses lies in the quality of the retrieved context. Here's why option A is the most appropriate solution:

  Context Relevance:The RAG model generates answers based on retrieved documents or context. If the retrieved information is about an irrelevant product, it suggests that the retrieval step is failing to select the right context. The Generative

  AI Engineer must first assess the quality of what is being retrieved and ensure it is pertinent to the query.

  Vector Search and Embedding Similarity:RAG typically uses vector search for retrieval, where embeddings of the query are matched against embeddings of product descriptions. Assessing the semantic similarity search process ensures that

  the closest matches are actually relevant to the query.

  Fine-tuning the Retrieval Process:By improving the retrieval quality , such as tuning the embeddings or adjusting the retrieval strategy, the system can return more accurate and relevant product information.

  Why Other Options Are Less Suitable:

  B (Caching FAQs): Caching can speed up responses for frequently asked questions but won't improve the relevance of the retrieved content for less frequent or new queries.

  C (Use a Different LLM): Changing the LLM only affects the generation step, not the retrieval process, which is the core issue here.

  D (Different Semantic Search Algorithm): This could help, but the first step is to evaluate the current retrieval context before replacing the search algorithm.

  Therefore, improving and assessing the quality of the retrieved context (option A) is the first step to fixing the issue of irrelevant product information.