Databricks DATABRICKS-CERTIFIED-GENERATIVE-AI-ENGINEER-ASSOCIATE Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-GENERATIVE-AI-ENGINEER-ASSOCIATE Online Questions & Answers

• Question 51:

  A Generative AI Engineer I using the code below to test setting up a vector store:

  

  Assuming they intend to use Databricks managed embeddings with the default embedding model, what should be the next logical function call?

  A. vsc.get_index()
  B. vsc.create_delta_sync_index()
  C. vsc.create_direct_access_index()
  D. vsc.similarity_search()
  B. vsc.create_delta_sync_index()

  ---

  Explanation

  Context : The Generative AI Engineer is setting up a vector store using Databricks' VectorSearchClient. This is typically done to enable fast and efficient retrieval of vectorized data for tasks like similarity searches.

  Explanation of Options :

  Option A: vsc.get_index(): This function would be used to retrieve an existing index, not create one, so it would not be the logical next step immediately after creating an endpoint.

  Option B: vsc.create_delta_sync_index(): After setting up a vector store endpoint, creating an index is necessary to start populating and organizing the data. The create_delta_sync_index() function specifically creates an index that

  synchronizes with a Delta table, allowing automatic updates as the data changes. This is likely the most appropriate choice if the engineer plans to use dynamic data that is updated over time.

  Option C: vsc.create_direct_access_index(): This function would create an index that directly accesses the data without synchronization. While also a valid approach, it's less likely to be the next logical step if the default setup (typically

  accommodating changes) is intended.

  Option D: vsc.similarity_search(): This function would be used to perform searches on an existing index; however, an index needs to be created and populated with data before any search can be conducted.

  Given the typical workflow in setting up a vector store, the next step after creating an endpoint is to establish an index, particularly one that synchronizes with ongoing data updates, hence Option B .

• Question 52:

  A Generative AI Engineer developed an LLM application using the provisioned throughput Foundation Model API. Now that the application is ready to be deployed, they realize their volume of requests are not sufficiently high enough to create their own provisioned throughput endpoint. They want to choose a strategy that ensures the best cost-effectiveness for their application.

  What strategy should the Generative AI Engineer use?

  A. Switch to using External Models instead
  B. Deploy the model using pay-per-token throughput as it comes with cost guarantees
  C. Change to a model with a fewer number of parameters in order to reduce hardware constraint issues
  D. Throttle the incoming batch of requests manually to avoid rate limiting issues
  B. Deploy the model using pay-per-token throughput as it comes with cost guarantees

  ---

  Explanation

  Problem Context: The engineer needs a cost-effective deployment strategy for an LLM application with relatively low request volume.

  Explanation of Options:

  Option A: Switching to external models may not provide the required control or integration necessary for specific application needs.

  Option B: Using a pay-per-token model is cost-effective, especially for applications with variable or low request volumes, as it aligns costs directly with usage.

  Option C: Changing to a model with fewer parameters could reduce costs, but might also impact the performance and capabilities of the application.

  Option D: Manually throttling requests is a less efficient and potentially error-prone strategy for managing costs.

  Option is ideal, offering flexibility and cost control, aligning expenses directly with the application's usage B patterns.

• Question 53:

  A Generative AI Engineer is building a Generative AI system that suggests the best matched employee team member to newly scoped projects. The team member is selected from a very large team. Thematch should be based upon project date availability and how well their employee profile matches the project scope. Both the employee profile and project scope are unstructured text.

  How should the Generative Al Engineer architect their system?

  A. Create a tool for finding available team members given project dates. Embed all project scopes into a vector store, perform a retrieval using team member profiles to find the best team member.
  B. Create a tool for finding team member availability given project dates, and another tool that uses an LLM to extract keywords from project scopes. Iterate through available team members' profiles and perform keyword matching to find the best available team member.
  C. Create a tool to find available team members given project dates. Create a second tool that can calculate a similarity score for a combination of team member profile and the project scope. Iterate through the team members and rank by best score to select a team member.
  D. Create a tool for finding available team members given project dates. Embed team profiles into a vector store and use the project scope and filtering to perform retrieval to find the available best matched team members.
  D. Create a tool for finding available team members given project dates. Embed team profiles into a vector store and use the project scope and filtering to perform retrieval to find the available best matched team members.

  ---

  Explanation

  Problem Context: The problem involves matching team members to new projects based on two main factors:

  Availability: Ensure the team members are available during the project dates.

  Profile-Project Match: Use the employee profiles (unstructured text) to find the best match for a project's scope (also unstructured text).

  The two main inputs are the employee profiles and project scopes , both of which are unstructured. This means traditional rule-based systems (e.g., simple keyword matching) would be inefficient, especially when working with large datasets.

  Explanation of Options: Let's break down the provided options to understand why D is the most optimal answer.

  Option Asuggests embedding project scopes into a vector store and then performing retrieval using team member profiles. While embedding project scopes into a vector store is a valid technique, it skips an important detail: the focus should

  primarily be on embedding employee profiles because we're matching the profiles to a new project, not the other way around.

  Option Binvolves using a large language model (LLM) to extract keywords from the project scope and perform keyword matching on employee profiles. While LLMs can help with keyword extraction, this approach is too simplistic and doesn't

  leverage advanced retrieval techniques like vector embeddings, which can handle the nuanced and rich semantics of unstructured data. This approach may miss out on subtle but important similarities.

  Option Csuggests calculating a similarity score between each team member's profile and project scope. While this is a good idea, it doesn't specify how to handle the unstructured nature of data efficiently. Iterating through each member's

  profile individually could be computationally expensive in large teams. It also lacks the mention of using a vector store or an efficient retrieval mechanism.

  Option Dis the correct approach. Here's why:

  Embedding team profiles into a vector store: Using a vector store allows for efficient similarity searches on unstructured data. Embedding the team member profiles into vectors captures their semantics in a way that is far more flexible than

  keyword-based matching.

  Using project scope for retrieval: Instead of matching keywords, this approach suggests using vector embeddings and similarity search algorithms (e.g., cosine similarity) to find the team members whose profiles most closely align with the

  project scope.

  Filtering based on availability: Once the best-matched candidates are retrieved based on profile similarity, filtering them by availability ensures that the system provides a practically useful result.

  This method efficiently handles large-scale datasets by leveraging vector embeddings and similarity search techniques, both of which are fundamental tools in Generative AI engineering for handling unstructured text.

  Technical

  References:

  Vector embeddings: In this approach, the unstructured text (employee profiles and project scopes) is converted into high-dimensional vectors using pretrained models (e.g., BERT, Sentence-BERT, or custom embeddings). These embeddings

  capture the semantic meaning of the text, making it easier to perform similarity-based retrieval.

  Vector stores: Solutions like FAISS Milvus or allow storing and retrieving large numbers of vector embeddings quickly. This is critical when working with large teams where querying through individual profiles sequentially would be inefficient.

  LLM Integration: Large language models can assist in generating embeddings for both employee profiles and project scopes. They can also assist in fine-tuning similarity measures, ensuring that the retrieval system captures the nuances of

  the text data.

  Filtering: After retrieving the most similar profiles based on the project scope, filtering based on availability ensures that only team members who are free for the project are considered.

  This system is scalable, efficient, and makes use of the latest techniques in Generative AI , such as vector embeddings and semantic search.

• Question 54:

  A team wants to serve a code generation model as an assistant for their software developers. It should support multiple programming languages. Quality is the primary objective. Which of the Databricks Foundation Model APIs, or models available in the Marketplace, would be the best fit?

  A. Llama2-70b
  B. BGE-large
  C. MPT-7b
  D. CodeLlama-34B
  D. CodeLlama-34B

  ---

  Explanation

  For a code generation model that supports multiple programming languages and where quality is the primary objective, CodeLlama-34B is the most suitable choice. Here's the reasoning:

  Specialization in Code Generation:CodeLlama-34B is specifically designed for code generation tasks. This model has been trained with a focus on understanding and generating code, which makes it particularly adept at handling various

  programming languages and coding contexts.

  Capacity and Performance:The "34B" indicates a model size of 34 billion parameters, suggesting a high capacity for handling complex tasks and generating high-quality outputs. The large model size typically correlates with better

  understanding and generation capabilities in diverse scenarios.

  Suitability for Development Teams:Given that the model is optimized for code, it will be able to assist software developers more effectively than general-purpose models. It understands coding syntax, semantics, and the nuances of different

  programming languages.

  Why Other Options Are Less Suitable:

  A (Llama2-70b): While also a large model, it's more general-purpose and may not be as fine- tuned for code generation as CodeLlama.

  B (BGE-large): This model may not specifically focus on code generation.

  C (MPT-7b): Smaller than CodeLlama-34B and likely less capable in handling complex code generation tasks at high quality.

  Therefore, for a high-quality, multi-language code generation application, CodeLlama-34B (option D) is the best fit.

• Question 55:

  Which TWO chain components are required for building a basic LLM-enabled chat application that includes conversational capabilities, knowledge retrieval, and contextual memory?

  A. Vector Stores
  B. Conversation Buffer Memory
  C. External tools
  D. Chat loaders
  E. React Components
  A. Vector Stores
  B. Conversation Buffer Memory

  ---

  Explanation

• Question 56:

  A Generative AI Engineer has created a RAG application which can help employees retrieve answers from an internal knowledge base, such as Confluence pages or Google Drive. The prototype application is now working with some positive feedback from internal company testers. Now the Generative Al Engineer wants to formally evaluate the system's performance and understand where to focus their efforts to further improve the system.

  How should the Generative AI Engineer evaluate the system?

  A. Use cosine similarity score to comprehensively evaluate the quality of the final generated answers.
  B. Curate a dataset that can test the retrieval and generation components of the system separately. Use MLflow's built in evaluation metrics to perform the evaluation on the retrieval and generation components.
  C. Benchmark multiple LLMs with the same data and pick the best LLM for the job.
  D. Use an LLM-as-a-judge to evaluate the quality of the final answers generated.
  B. Curate a dataset that can test the retrieval and generation components of the system separately. Use MLflow's built in evaluation metrics to perform the evaluation on the retrieval and generation components.

  ---

  Explanation

  Problem Context: After receiving positive feedback for the RAG application prototype, the next step is to formally evaluate the system to pinpoint areas for improvement.

  Explanation of Options:

  Option A: While cosine similarity scores are useful, they primarily measure similarity rather than the overall performance of an RAG system.

  Option B: This option provides a systematic approach to evaluation by testing both retrieval and generation components separately. This allows for targeted improvements and a clear understanding of each component's performance, using

  MLflow's metrics for a structured and standardized assessment.

  Option C: Benchmarking multiple LLMs does not focus on evaluating the existing system's components but rather on comparing different models.

  Option D: Using an LLM as a judge is subjective and less reliable for systematic performance evaluation.

  Option is the most comprehensive and structured approach, facilitating precise evaluations and B improvements on specific components of the RAG system.

• Question 57:

  A Generative Al Engineer is building an LLM-based application that has an important transcription (speech-to-text) task. Speed is essential for the success of the application.

  Which open Generative Al models should be used?

  A. L!ama-2-70b-chat-hf
  B. MPT-30B-lnstruct
  C. DBRX
  D. whisper-large-v3 (1.6B)
  D. whisper-large-v3 (1.6B)

  ---

  Explanation

• Question 58:

  A Generative Al Engineer is developing a RAG system for their company to perform internal document QandA for structured HR policies, but the answers returned are frequently incomplete and unstructured It seems that the retriever is not returning all relevant context The Generative Al Engineer has experimented with different embedding and response generating LLMs but that did not improve results.

  Which TWO options could be used to improve the response quality? Choose 2 answers

  A. Add the section header as a prefix to chunks
  B. Increase the document chunk size
  C. Split the document by sentence
  D. Use a larger embedding model
  E. Fine tune the response generation model
  A. Add the section header as a prefix to chunks
  B. Increase the document chunk size

  ---

  Explanation

• Question 59:

  A Generative AI Engineer has been asked to build an LLM-based question-answering application. The application should take into account new documents that are frequently published. The engineer wants to build this application with the least cost and least development effort and have it operate at the lowest cost possible.

  Which combination of chaining components and configuration meets these requirements?

  A. For the application a prompt, a retriever, and an LLM are required. The retriever output is inserted into the prompt which is given to the LLM to generate answers.
  B. The LLM needs to be frequently with the new documents in order to provide most up-to-date answers.
  C. For the question-answering application, prompt engineering and an LLM are required to generate answers.
  D. For the application a prompt, an agent and a fine-tuned LLM are required. The agent is used by the LLM to retrieve relevant content that is inserted into the prompt which is given to the LLM to generate answers.
  A. For the application a prompt, a retriever, and an LLM are required. The retriever output is inserted into the prompt which is given to the LLM to generate answers.

  ---

  Explanation

  Problem Context : The task is to build an LLM-based question-answering application that integrates new documents frequently with minimal costs and development efforts.

  Explanation of Options :

  Option A: Utilizes a prompt and a retriever, with the retriever output being fed into the LLM. This setup is efficient because it dynamically updates the data pool via the retriever, allowing the LLM to provide up-to-date answers based on the

  latest documents without needing tofrequently retrain the model. This method offers a balance of cost-effectiveness and functionality.

  Option B: Requires frequent retraining of the LLM, which is costly and labor-intensive.

  Option C: Only involves prompt engineering and an LLM, which may not adequately handle the requirement for incorporating new documents unless it's part of an ongoing retraining or updating mechanism, which would increase costs.

  Option D: Involves an agent and a fine-tuned LLM, which could be overkill and lead to higher development and operational costs.

  Option Ais the most suitable as it provides a cost-effective, minimal development approach while ensuring the application remains up-to-date with new information.

• Question 60:

  A Generative AI Engineer is developing an LLM application that users can use to generate personalized birthday poems based on their names. Which technique would be most effective in safeguarding the application, given the potential for malicious user inputs?

  A. Implement a safety filter that detects any harmful inputs and ask the LLM to respond that it is unable to assist
  B. Reduce the time that the users can interact with the LLM
  C. Ask the LLM to remind the user that the input is malicious but continue the conversation with the user
  D. Increase the amount of compute that powers the LLM to process input faster
  A. Implement a safety filter that detects any harmful inputs and ask the LLM to respond that it is unable to assist

  ---

  Explanation

  In this case, the Generative AI Engineer is developing an application to generate personalized birthday poems, but there's a need to safeguard against malicious user inputs . The best solution is to implement a safety filter (option A) to detect

  harmful or inappropriate inputs.

  Safety Filter Implementation:Safety filters are essential for screening user input and preventing inappropriate content from being processed by the LLM. These filters can scan inputs for harmful language, offensive terms, or malicious content

  and intervene before the prompt is passed to the LLM.

  Graceful Handling of Harmful Inputs:Once the safety filter detects harmful content, the system can provide a message to the user, such as "I'm unable to assist with this request," instead of processing or responding to malicious input. This

  protects the system from generating harmful content and ensures a controlled interaction environment.

  Why Other Options Are Less Suitable:

  B (Reduce Interaction Time): Reducing the interaction time won't prevent malicious inputs from being entered.

  C (Continue the Conversation): While it's possible to acknowledge malicious input, it is not safe to continue the conversation with harmful content. This could lead to legal or reputational risks.

  D (Increase Compute Power): Adding more compute doesn't address the issue of harmful content and would only speed up processing without resolving safety concerns.

  Therefore, implementing a safety filter that blocks harmful inputs is the most effective technique for safeguarding the application.