Databricks DATABRICKS-CERTIFIED-GENERATIVE-AI-ENGINEER-ASSOCIATE Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-GENERATIVE-AI-ENGINEER-ASSOCIATE Online Questions & Answers

• Question 1:

  After changing the response generating LLM in a RAG pipeline from GPT-4 to a model with a shorter context length that the company self-hosts, the Generative AI Engineer is getting the following error:

  

  What TWO solutions should the Generative AI Engineer implement without changing the response generating model? (Choose two.)

  A. Use a smaller embedding model to generate
  B. Reduce the maximum output tokens of the new model
  C. Decrease the chunk size of embedded documents
  D. Reduce the number of records retrieved from the vector database
  E. Retrain the response generating model using ALiBi
  C. Decrease the chunk size of embedded documents
  D. Reduce the number of records retrieved from the vector database

  ---

  Explanation

  Problem Context: After switching to a model with a shorter context length, the error message indicating that the prompt token count has exceeded the limit suggests that the input to the model is too large.

  Explanation of Options:

  Option A: Use a smaller embedding model to generate?This wouldn't necessarily address the issue of prompt size exceeding the model's token limit.

  Option B: Reduce the maximum output tokens of the new model?This option affects the output length, not the size of the input being too large.

  Option C: Decrease the chunk size of embedded documents?This would help reduce the size of each document chunk fed into the model, ensuring that the input remains within the model's context length limitations.

  Option D: Reduce the number of records retrieved from the vector database?By retrieving fewer records, the total input size to the model can be managed more effectively, keeping it within the allowable token limits.

  Option E: Retrain the response generating model using ALiBi?Retraining the model is contrary to the stipulation not to change the response generating model.

  Options and are the most effective solutions to manage the model's shorter context length without changing C D the model itself, by adjusting the input size both in terms of individual document size and total documents retrieved.

• Question 2:

  A Generative AI Engineer has been asked to design an LLM-based application that accomplishes the following business objective: answer employee HR questions using HR PDF documentation. Which set of high level tasks should the Generative AI Engineer's system perform?

  A. Calculate averaged embeddings for each HR document, compare embeddings to user query to find the best document. Pass the best document with the user query into an LLM with a large context window to generate a response to the employee.
  B. Use an LLM to summarize HR documentation. Provide summaries of documentation and user query into an LLM with a large context window to generate a response to the user.
  C. Create an interaction matrix of historical employee questions and HR documentation. Use ALS to factorize the matrix and create embeddings. Calculate the embeddings of new queries and use them to find the best HR documentation. Use an LLM to generate a response to the employee question based upon the documentation retrieved.
  D. Split HR documentation into chunks and embed into a vector store. Use the employee question to retrieve best matched chunks of documentation, and use the LLM to generate a response to the employee based upon the documentation retrieved.
  D. Split HR documentation into chunks and embed into a vector store. Use the employee question to retrieve best matched chunks of documentation, and use the LLM to generate a response to the employee based upon the documentation retrieved.

  ---

  Explanation

  To design an LLM-based application that can answer employee HR questions using HR PDF documentation, the most effective approach is option

  D. Here's why:

  Chunking and Vector Store Embedding:HR documentation tends to be lengthy, so splitting it into smaller, manageable chunks helps optimize retrieval. These chunks are then embedded into avector store (a database that stores vector

  representations of text). Each chunk of text is transformed into an embedding using a transformer-based model, which allows for efficient similarity-based retrieval.

  Using Vector Search for Retrieval:When an employee asks a question, the system converts their query into an embedding as well. This embedding is then compared with the embeddings of the document chunks in the vector store. The most

  semantically similar chunks are retrieved, which ensures that the answer is based on the most relevant parts of the documentation.

  LLM to Generate a Response:Once the relevant chunks are retrieved, these chunks are passed into the LLM, which uses them as context to generate a coherent and accurate response to the employee's question.

  Why Other Options Are Less Suitable:

  A (Calculate Averaged Embeddings): Averaging embeddings might dilute important information. It doesn't provide enough granularity to focus on specific sections of documents.

  B (Summarize HR Documentation): Summarization loses the detail necessary for HR-related queries, which are often specific. It would likely miss the mark for more detailed inquiries.

  C (Interaction Matrix and ALS): This approach is better suited for recommendation systems and not for HR queries, as it's focused on collaborative filtering rather than text-based retrieval.

  Thus, option D is the most effective solution for providing precise and contextual answers based on HR documentation.

• Question 3:

  A Generative Al Engineer is developing a RAG application and would like to experiment with different embedding models to improve the application performance.

  Which strategy for picking an embedding model should they choose?

  A. Pick an embedding model trained on related domain knowledge
  B. Pick the most recent and most performant open LLM released at the time
  C. pick the embedding model ranked highest on the Massive Text Embedding Benchmark (MTEB) leaderboard hosted by HuggingFace
  D. Pick an embedding model with multilingual support to support potential multilingual user questions
  A. Pick an embedding model trained on related domain knowledge

  ---

  Explanation

• Question 4:

  A Generative Al Engineer is helping a cinema extend its website's chat bot to be able to respond to questions about specific showtimes for movies currently playing at their local theater. They already have the location of the user provided by location services to their agent, and a Delta table which is continually updated with the latest showtime information by location. They want to implement this new capability In their RAG application.

  Which option will do this with the least effort and in the most performant way?

  A. Create a Feature Serving Endpoint from a FeatureSpec that references an online store synced from the Delta table. Query the Feature Serving Endpoint as part of the agent logic / tool implementation.
  B. Query the Delta table directly via a SQL query constructed from the user's input using a text-to-SQL LLM in the agent logic / tool
  C. implementation. Write the Delta table contents to a text column.then embed those texts using an embedding model and store these in the vector index Look up the information based on the embedding as part of the agent logic / tool implementation.
  D. Set up a task in Databricks Workflows to write the information in the Delta table periodically to an external database such as MySQL and query the information from there as part of the agent logic / tool implementation.
  A. Create a Feature Serving Endpoint from a FeatureSpec that references an online store synced from the Delta table. Query the Feature Serving Endpoint as part of the agent logic / tool implementation.

  ---

  Explanation

• Question 5:

  A company has a typical RAG-enabled, customer-facing chatbot on its website.

  

  Select the correct sequence of components a user's questions will go through before the final output is returned. Use the diagram above for reference.

  A. 1.embedding model, 2.vector search, 3.context-augmented prompt, 4.response-generating LLM
  B. 1.context-augmented prompt, 2.vector search, 3.embedding model, 4.response-generating LLM
  C. 1.response-generating LLM, 2.vector search, 3.context-augmented prompt, 4.embedding model
  D. 1.response-generating LLM, 2.context-augmented prompt, 3.vector search, 4.embedding model
  A. 1.embedding model, 2.vector search, 3.context-augmented prompt, 4.response-generating LLM

  ---

  Explanation

  To understand how a typical RAG-enabled customer-facing chatbot processes a user's question, let's go through the correct sequence as depicted in the diagram and explained in option A:

  Embedding Model (1):The first step involves the user's question being processed through an embedding model. This model converts the text into a vector format that numerically represents the text. This step is essential for allowing the

  subsequent vector search to operate effectively.

  Vector Search (2):The vectors generated by the embedding model are then used in a vector search mechanism. This search identifies the most relevant documents or previously answered questions that are stored in a vector format in a

  database.

  Context-Augmented Prompt (3):The information retrieved from the vector search is used to create a context-augmented prompt. This step involves enhancing the basic user query with additional relevant information gathered to ensure the

  generated response is as accurate and informative as possible.

  Response-Generating LLM (4):Finally, the context-augmented prompt is fed into a response- generating large language model (LLM). This LLM uses the prompt to generate a coherent and contextually appropriate answer, which is then

  delivered as the final output to the user.

  Why Other Options Are Less Suitable:

  B, C, D: These options suggest incorrect sequences that do not align with how a RAG system typically processes queries. They misplace the role of embedding models, vector search, and response generation in an order that would not

  facilitate effective information retrieval and response generation. Thus, the correct sequence isembedding model, vector search, context-augmented prompt, response- generating LLM , which is option A.

• Question 6:

  What is the most suitable library for building a multi-step LLM-based workflow?

  A. Pandas
  B. TensorFlow
  C. PySpark
  D. LangChain
  D. LangChain

  ---

  Explanation

  Problem Context: The Generative AI Engineer needs a tool to build amulti-step LLM-based workflow . This type of workflow often involves chaining multiple steps together, such as query generation, retrieval of information, response generation, and post-processing, with LLMs integrated at several points.

  Explanation of Options:

  Option A: Pandas: Pandas is a powerful data manipulation library for structured data analysis, but it is not designed for managing or orchestrating multi-step workflows, especially those involving LLMs.

  Option B: TensorFlow: TensorFlow is primarily used for training and deploying machine learning models, especially deep learning models. It is not designed for orchestrating multi-step tasks in LLM-based workflows.

  Option C: PySpark: PySpark is a distributed computing framework used for large-scale data processing. While useful for handling big data, it is not specialized for chaining LLM-based operations.

  Option D: LangChain: LangChain is a purpose-built framework designed specifically for orchestrating multi-step workflows with large language models (LLMs). It enables developers to easily chain different tasks, such as retrieving

  documents, summarizing information, and generating responses, all in a structured flow. This makes it the best tool for building complex LLM-based workflows. Thus, LangChain is the most suitable library for creating multi-step LLM-based workflows.

• Question 7:

  A Generative AI Engineer wants to build an LLM-based solution to help a restaurant improve its online customer experience with bookings by automatically handling common customer inquiries. The goal of the solution is to minimize escalations to human intervention and phone calls while maintaining a personalized interaction. To design the solution, the Generative AI Engineer needs to define the input data to the LLM and the task it should perform.

  Which input/output pair will support their goal?

  A. Input: Online chat logs; Output: Group the chat logs by users, followed by summarizing each user's interactions
  B. Input: Online chat logs; Output: Buttons that represent choices for booking details
  C. Input: Customer reviews; Output: Classify review sentiment
  D. Input: Online chat logs; Output: Cancellation options
  B. Input: Online chat logs; Output: Buttons that represent choices for booking details

  ---

  Explanation

  Context : The goal is to improve the online customer experience in a restaurant by handling common inquiries about bookings, minimizing escalations, and maintaining personalized interactions.

  Explanation of Options :

  Option A: Grouping and summarizing chat logs by user could provide insights into customer interactions but does not directly address the task of handling booking inquiries or minimizing escalations.

  Option B: Using chat logs to generate interactive buttons for booking details directly supports the goal of facilitating online bookings, minimizing the need for human intervention by providing clear, interactive options for customers to self-serve.

  Option C: Classifying sentiment of customer reviews does not directly help with booking inquiries, although it might provide valuable feedback insights.

  Option D: Providing cancellation options is helpful but narrowly focuses on one aspect of the booking process and doesn't support the broader goal of handling common inquiries about bookings.

  Option Bbest supports the goal of improving online interactions by using chat logs to generate actionable items for customers, helping them complete booking tasks efficiently and reducing the need for human intervention.

• Question 8:

  A Generative AI Engineer has a provisioned throughput model serving endpoint as part of a RAG application and would like to monitor the serving endpoint's incoming requests and outgoing responses. The current approach is to include a micro-service in between the endpoint and the user interface to write logs to a remote server.

  Which Databricks feature should they use instead which will perform the same task?

  A. Vector Search
  B. Lakeview
  C. DBSQL
  D. Inference Tables
  D. Inference Tables

  ---

  Explanation

  Problem Context : The goal is to monitor the serving endpoint for incoming requests and outgoing responses in a provisioned throughput model serving endpoint within a Retrieval-Augmented Generation (RAG) application . The current

  approach involves using a microservice to log requests and responses to a remote server, but the Generative AI Engineer is looking for a more streamlined solution within Databricks.

  Explanation of Options :

  Option A: Vector Search: This feature is used to perform similarity searches within vector databases. It doesn't provide functionality for logging or monitoring requests and responses in a serving endpoint, so it's not applicable here.

  Option B: Lakeview: Lakeview is not a feature relevant to monitoring or logging request-response cycles for serving endpoints. It might be more related to viewing data in Databricks Lakehouse but doesn't fulfill the specific monitoring

  requirement.

  Option C: DBSQL: Databricks SQL (DBSQL) is used for running SQL queries on data stored in Databricks, primarily for analytics purposes. It doesn't provide the direct functionality needed to monitor requests and responses in real-time for an

  inference endpoint.

  Option D: Inference Tables: This is the correct answer. Inference Tables in Databricks are designed to store the results and metadata of inference runs. This allows the system to logincoming requests and outgoing responses directly within

  Databricks, making it an ideal choice for monitoring the behavior of a provisioned serving endpoint. Inference Tables can be queried and analyzed, enabling easier monitoring and debugging compared to a custom microservice.

  Thus, Inference Tables are the optimal feature for monitoring request and response logs within the Databricks infrastructure for a model serving endpoint.

• Question 9:

  A Generative AI Engineer is designing an LLM-powered live sports commentary platform. The platform provides real-time updates and LLM-generated analyses for any users who would like to have live summaries, rather than reading a series of potentially outdated news articles.

  Which tool below will give the platform access to real-time data for generating game analyses based on the latest game scores?

  A. DatabrickslQ
  B. Foundation Model APIs
  C. Feature Serving
  D. AutoML
  C. Feature Serving

  ---

  Explanation

  Problem Context: The engineer is developing an LLM-powered live sports commentary platform that needs to provide real-time updates and analyses based on the latest game scores. The critical requirement here is the capability to access

  and integrate real-time data efficiently with the platform for immediate analysis and reporting.

  Explanation of Options:

  Option A: DatabricksIQ: While DatabricksIQ offers integration and data processing capabilities, it is more aligned with data analytics rather than real-time feature serving, which is crucial for immediate updates necessary in a live sports

  commentary context.

  Option B: Foundation Model APIs: These APIs facilitate interactions with pre-trained models and could be part of the solution, but on their own, they do not provide mechanisms to access real- time game scores.

  Option C: Feature Serving: This is the correct answer as feature serving specifically refers to the real-time provision of data (features) to models for prediction. This would be essential for an LLM that generates analyses based on live game

  data, ensuring that the commentary is current and based on the latest events in the sport.

  Option D: AutoML: This tool automates the process of applying machine learning models to real-world problems, but it does not directly provide real-time data access, which is a critical requirement for the platform.

  Thus, Option C (Feature Serving) is the most suitable tool for the platform as it directly supports the real-time data needs of an LLM-powered sports commentary system, ensuring that the analyses and updates are based on the latest

  available information.

• Question 10:

  A Generative Al Engineer is creating an LLM-based application. The documents for its retriever have been chunked to a maximum of 512 tokens each. The Generative Al Engineer knows that cost and latency are more important than quality for this application. They have several context length levels to choose from.

  Which will fulfill their need?

  A. context length 514; smallest model is 0.44GB and embedding dimension 768
  B. context length 2048: smallest model is 11GB and embedding dimension 2560
  C. context length 32768: smallest model is 14GB and embedding dimension 4096
  D. context length 512: smallest model is 0.13GB and embedding dimension 384
  D. context length 512: smallest model is 0.13GB and embedding dimension 384

  ---

  Explanation

  When prioritizing cost and latency over quality in a Large Language Model (LLM)-based application, it is crucial to select a configuration that minimizes both computational resources and latency while still providing reasonable performance. Here's why is the best choice: D

  Context length: The context length of 512 tokens aligns with the chunk size used for the documents (maximum of 512 tokens per chunk). This is sufficient for capturing the needed information and generating responses without unnecessary overhead.

  Smallest model size: The model with a size of 0.13GB is significantly smaller than the other options. This small footprint ensures faster inference times and lower memory usage, which directly reduces both latency and cost.

  Embedding dimension: While the embedding dimension of 384 is smaller than the other options, it is still adequate for tasks where cost and speed are more important than precision and depth of understanding. This setup achieves the desired balance between cost-efficiency and reasonable performance in a latency- sensitive, cost-conscious application.