Databricks DATABRICKS-CERTIFIED-GENERATIVE-AI-ENGINEER-ASSOCIATE Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-GENERATIVE-AI-ENGINEER-ASSOCIATE Online Questions & Answers

• Question 41:

  A Generative AI Engineer is developing a chatbot designed to assist users with insurance-related queries. The chatbot is built on a large language model (LLM) and is conversational. However, to maintain the chatbot's focus and to comply

  with company policy, it must not provide responses to questions about politics. Instead, when presented with political inquiries, the chatbot should respond with a standard message:

  "Sorry, I cannot answer that. I am a chatbot that can only answer questions around insurance."

  Which framework type should be implemented to solve this?

  A. Safety Guardrail
  B. Security Guardrail
  C. Contextual Guardrail
  D. Compliance Guardrail
  A. Safety Guardrail

  ---

  Explanation

  In this scenario, the chatbot must avoid answering political questions and instead provide a standard message for such inquiries. Implementing a Safety Guardrail is the appropriate solution for this:

  What is a Safety Guardrail?Safety guardrails are mechanisms implemented in Generative AI systems to ensure the model behaves within specific bounds. In this case, it ensures the chatbot does not answer politically sensitive or irrelevant

  questions, which aligns with the business rules.

  Preventing Responses to Political Questions:The Safety Guardrail is programmed to detect specific types of inquiries (like political questions) and prevent the model from generating responses outside its intended domain. When such queries

  are detected, the guardrail intervenes and provides a pre-defined response: "Sorry, I cannot answer that. I am a chatbot that can only answer questions around insurance."

  How It Works in Practice:The LLM system can include a classification layer or trigger rules based on specific keywords related to politics. When such terms are detected, the Safety Guardrail blocks the normal generation flow and responds

  with the fixed message.

  Why Other Options Are Less Suitable:

  B (Security Guardrail): This is more focused on protecting the system from security vulnerabilities or data breaches, not controlling the conversational focus.

  C (Contextual Guardrail): While context guardrails can limit responses based on context, safety guardrails are specifically about ensuring the chatbot stays within a safe conversational scope.

  D (Compliance Guardrail): Compliance guardrails are often related to legal and regulatory adherence, which is not directly relevant here.

  Therefore, a Safety Guardrail is the right framework to ensure the chatbot only answers insurance-related queries and avoids political discussions.

• Question 42:

  A Generative Al Engineer has created a RAG application to look up answers to questions about a series of fantasy novels that are being asked on the author's web forum. The fantasy novel texts are chunked and embedded into a vector store with metadata (page number, chapter number, book title), retrieved with the user' s query, and provided to an LLM for response generation. The Generative AI Engineer used their intuition to pick the chunking strategy and associated configurations but now wants to more methodically choose the best values.

  Which TWO strategies should the Generative AI Engineer take to optimize their chunking strategy and parameters? (Choose two.)

  A. Change embedding models and compare performance.
  B. Add a classifier for user queries that predicts which book will best contain the answer. Use this to filter retrieval.
  C. Choose an appropriate evaluation metric (such as recall or NDCG) and experiment with changes in the chunking strategy, such as splitting chunks by paragraphs or chapters. Choose the strategy that gives the best performance metric.
  D. Pass known questions and best answers to an LLM and instruct the LLM to provide the best token count. Use a summary statistic (mean, median, etc.) of the best token counts to choose chunk size.
  E. Create an LLM-as-a-judge metric to evaluate how well previous questions are answered by the most appropriate chunk. Optimize the chunking parameters based upon the values of the metric.
  C. Choose an appropriate evaluation metric (such as recall or NDCG) and experiment with changes in the chunking strategy, such as splitting chunks by paragraphs or chapters. Choose the strategy that gives the best performance metric.
  E. Create an LLM-as-a-judge metric to evaluate how well previous questions are answered by the most appropriate chunk. Optimize the chunking parameters based upon the values of the metric.

  ---

  Explanation

  To optimize a chunking strategy for a Retrieval-Augmented Generation (RAG) application, the Generative AI Engineer needs a structured approach to evaluating the chunking strategy, ensuring that the chosen configuration retrieves the most

  relevant information and leads to accurate and coherent LLM responses.

  Here's why and are the correct strategies: C E

  Strategy C: Evaluation Metrics (Recall, NDCG)

  Define an evaluation metric: Common evaluation metrics such as recall, precision, or NDCG (Normalized Discounted Cumulative Gain) measure how well the retrieved chunks match the user's query and the expected response.

  Recallmeasures the proportion of relevant information retrieved.

  NDCGis often used when you want to account for both the relevance of retrieved chunks and the ranking or order in which they are retrieved.

  Experiment with chunking strategies: Adjusting chunking strategies based on text structure (e.g., splitting by paragraph, chapter, or a fixed number of tokens) allows the engineer to experiment with various ways of slicing the text. Some

  chunks may better align with the user's query than others.

  Evaluate performance: By using recall or NDCG, the engineer can methodically test various chunking strategies to identify which one yields the highest performance. This ensures that the chunking method provides the most relevant

  information when embedding and retrieving data from the vector store.

  Strategy E: LLM-as-a-Judge Metric

  Use the LLM as an evaluator: After retrieving chunks, the LLM can be used to evaluate the quality of answers based on the chunks provided. This could be framed as a "judge" function, where the LLM compares how well a given chunk

  answers previous user queries.

  Optimize based on the LLM's judgment: By having the LLM assess previous answers and rate their relevance and accuracy, the engineer can collect feedback on how well different chunking configurations perform in real-world scenarios.

  This metric could be a qualitative judgment on how closely the retrieved information matches the user's intent.

  Tune chunking parameters: Based on the LLM's judgment, the engineer can adjust the chunk size or structure to better align with the LLM's responses, optimizing retrieval for future queries.

  By combining these two approaches, the engineer ensures that the chunking strategy is systematically evaluated using both quantitative (recall/NDCG) and qualitative (LLM judgment) methods. This balanced optimization process results in

  improved retrieval relevance and, consequently, better response generation by the LLM.

• Question 43:

  A Generative Al Engineer has built an LLM-based system that will automatically translate user text between two languages. They now want to benchmark multiple LLM's on this task and pick the best one. They have an evaluation set with known high quality translation examples. They want to evaluate each LLM using the evaluation set with a performant metric.

  Which metric should they choose for this evaluation?

  A. ROUGE metric
  B. BLEU metric
  C. NDCG metric
  D. RECALL metric
  B. BLEU metric

  ---

  Explanation

• Question 44:

  A Generative Al Engineer is ready to deploy an LLM application written using Foundation Model APIs. They want to follow security best practices for production scenarios.

  Which authentication method should they choose?

  A. Use an access token belonging to service principals
  B. Use a frequently rotated access token belonging to either a workspace user or a service principal
  C. Use OAuth machine-to-machine authentication
  D. Use an access token belonging to any workspace user
  A. Use an access token belonging to service principals

  ---

  Explanation

• Question 45:

  Which indicator should be considered to evaluate the safety of the LLM outputs when qualitatively assessing LLM responses for a translation use case?

  A. The ability to generate responses in code
  B. The similarity to the previous language
  C. The latency of the response and the length of text generated
  D. The accuracy and relevance of the responses
  D. The accuracy and relevance of the responses

  ---

  Explanation

  Problem Context: When assessing the safety and effectiveness of LLM outputs in a translation use case, it is essential to ensure that the translations accurately and relevantly convey the intended message. The evaluation should focus on how well the LLM understands and processes different languages and contexts.

  Explanation of Options:

  Option A: The ability to generate responses in code?This is not relevant to translation quality or safety.

  Option B: The similarity to the previous language?While ensuring that translations preserve the original's intent is important, this doesn't directly address the overall quality or safety of the translation.

  Option C: The latency of the response and the length of text generated?These operational metrics are less critical in assessing the qualitative aspects of translation safety.

  Option D: The accuracy and relevance of the responses?This is crucial in translation to ensure that the translated content is true to the original in meaning and appropriateness. Accuracy and relevance directly impact the effectiveness and

  safety of translations, especially in sensitive or nuanced contexts.

  Thus, Option D is the most important indicator when evaluating the safety of LLM outputs in translation, focusing on the core aspects that determine the utility and trustworthiness of translated content.

• Question 46:

  A Generative Al Engineer is tasked with improving the RAG quality by addressing its inflammatory outputs. Which action would be most effective in mitigating the problem of offensive text outputs?

  A. Increase the frequency of upstream data updates
  B. Inform the user of the expected RAG behavior
  C. Restrict access to the data sources to a limited number of users
  D. Curate upstream data properly that includes manual review before it is fed into the RAG system
  D. Curate upstream data properly that includes manual review before it is fed into the RAG system

  ---

  Explanation

  Addressing offensive or inflammatory outputs in a Retrieval-Augmented Generation (RAG) system is critical for improving user experience and ensuring ethical AI deployment. Here's why is the most effective D approach:

  Manual data curation: The root cause of offensive outputs often comes from the underlying data used to train the model or populate the retrieval system. By manually curating the upstream data and conducting thorough reviews before the

  data is fed into the RAG system, the engineer can filter out harmful, offensive, or inappropriate content.

  Improving data quality: Curating data ensures the system retrieves and generates responses from a high-quality, well-vetted dataset. This directly impacts the relevance and appropriateness of the outputs from the RAG system, preventing

  inflammatory content from being included in responses.

  Effectiveness: This strategy directly tackles the problem at its source (the data) rather than just mitigating the consequences (such as informing users or restricting access). It ensures that the system consistently provides non-offensive,

  relevant information.

  Other options, such as increasing the frequency of data updates or informing users about behavior expectations, may not directly mitigate the generation of inflammatory outputs.

• Question 47:

  A Generative Al Engineer is setting up a Databricks Vector Search that will lookup news articles by topic within 10 days of the date specified An example query might be "Tell me about monster truck news around January 5th 1992". They

  want to do this with the least amount of effort.

  How can they set up their Vector Search index to support this use case?

  A. Split articles by 10 day blocks and return the block closest to the query.
  B. Include metadata columns for article date and topic to support metadata filtering.
  C. pass the query directly to the vector search index and return the best articles.
  D. Create separate indexes by topic and add a classifier model to appropriately pick the best index.
  B. Include metadata columns for article date and topic to support metadata filtering.

  ---

  Explanation

• Question 48:

  When developing an LLM application, it's crucial to ensure that the data used for training the model complies with licensing requirements to avoid legal risks. Which action is NOT appropriate to avoid legal risks?

  A. Reach out to the data curators directly before you have started using the trained model to let them know.
  B. Use any available data you personally created which is completely original and you can decide what license to use.
  C. Only use data explicitly labeled with an open license and ensure the license terms are followed.
  D. Reach out to the data curators directly after you have started using the trained model to let them know.
  D. Reach out to the data curators directly after you have started using the trained model to let them know.

  ---

  Explanation

  Problem Context: When using data to train a model, it's essential to ensure compliance with licensing to avoid legal risks. Legal issues can arise from using data without permission, especially when it comes from third-party sources.

  Explanation of Options:

  Option A: Reaching out to data curators before using the data is an appropriate action. This allows you to ensure you have permission or understand the licensing terms before starting to use the data in your model. Option B: Using original data that you personally created is always a safe option. Since you have full ownership over the data, there are no legal risks, as you control the licensing.

  Option C: Using data that is explicitly labeled with an open license and adhering to the license terms is a correct and recommended approach. This ensures compliance with legal requirements.

  Option D: Reaching out to the data curators after you have already started using the trained model is not appropriate . If you've already used the data without understanding its licensing terms, you may have already violated the terms of use,

  which could lead to legal complications. It's essential to clarify the licensing terms before using the data, not after.

  Thus, Option D is not appropriate because it could expose you to legal risks by using the data without first obtaining the proper licensing permissions.

• Question 49:

  A Generative Al Engineer is creating an LLM system that will retrieve news articles from the year 1918 and related to a user's query and summarize them. The engineer has noticed that the summaries are generated well but often also include an explanation of how the summary was generated, which is undesirable.

  Which change could the Generative Al Engineer perform to mitigate this issue?

  A. Split the LLM output by newline characters to truncate away the summarization explanation.
  B. Tune the chunk size of news articles or experiment with different embedding models.
  C. Revisit their document ingestion logic, ensuring that the news articles are being ingested properly.
  D. Provide few shot examples of desired output format to the system and/or user prompt.
  D. Provide few shot examples of desired output format to the system and/or user prompt.

  ---

  Explanation

  To mitigate the issue of the LLM including explanations of how summaries are generated in its output, the best approach is to adjust the training or prompt structure. Here's why Option D is effective:

  Few-shot Learning: By providing specific examples of how the desired output should look (i.e., just the summary without explanation), the model learns the preferred format. This few-shot learning approach helps the model understand not

  only what content to generate but also how to format its responses.

  Prompt Engineering: Adjusting the user prompt to specify the desired output format clearly can guide the LLM to produce summaries without additional explanatory text. Effective prompt design is crucial in controlling the behavior of

  generative models.

  Why Other Options Are Less Suitable:

  A: While technically feasible, splitting the output by newline and truncating could lead to loss of important content or create awkward breaks in the summary.

  B: Tuning chunk sizes or changing embedding models does not directly address the issue of the model's tendency to generate explanations along with summaries.

  C: Revisiting document ingestion logic ensures accurate source data but does not influence how the model formats its output.

  By using few-shot examples and refining the prompt, the engineer directly influences the output format, making this approach the most targeted and effective solution.

• Question 50:

  A Generative Al Engineer at an automotive company would like to build a question- answering chatbot for customers to inquire about their vehicles. They have a database containing various documents of different vehicle makes, their hardware parts, and common maintenance information.

  Which of the following components will NOT be useful in building such a chatbot?

  A. Response-generating LLM
  B. Invite users to submit long, rather than concise, questions
  C. Vector database
  D. Embedding model
  B. Invite users to submit long, rather than concise, questions

  ---

  Explanation