Databricks DATABRICKS-CERTIFIED-GENERATIVE-AI-ENGINEER-ASSOCIATE Online Practice Questions and Exam Preparation

Databricks DATABRICKS-CERTIFIED-GENERATIVE-AI-ENGINEER-ASSOCIATE Online Questions & Answers

• Question 31:

  A Generative Al Engineer is tasked with developing an application that is based on an open source large language model (LLM). They need a foundation LLM with a large context window. Which model fits this need?

  A. DistilBERT
  B. MPT-30B
  C. Llama2-70B
  D. DBRX
  C. Llama2-70B

  ---

  Explanation

  Problem Context: The engineer needs an open-source LLM with a large context window to develop an application.

  Explanation of Options:

  Option A: DistilBERT: While an efficient and smaller version of BERT, DistilBERT does not provide a particularly large context window.

  Option B: MPT-30B: This model, while large, is not specified as being particularly notable for its context window capabilities.

  Option C: Llama2-70B: Known for its large model size and extensive capabilities, including a large context window. It is also available as an open-source model, making it ideal for applications requiring extensive contextual understanding.

  Option D: DBRX: This is not a recognized standard model in the context of large language models with extensive context windows.

  Thus, Option C (Llama2-70B) is the best fit as it meets the criteria of having a large context window and being available for open-source use, suitable for developing robust language understanding applications.

• Question 32:

  What is an effective method to preprocess prompts using custom code before sending them to an LLM?

  A. Directly modify the LLM's internal architecture to include preprocessing steps
  B. It is better not to introduce custom code to preprocess prompts as the LLM has not been trained with examples of the preprocessed prompts
  C. Rather than preprocessing prompts, it's more effective to postprocess the LLM outputs to align the outputs to desired outcomes
  D. Write a MLflow PyFunc model that has a separate function to process the prompts
  D. Write a MLflow PyFunc model that has a separate function to process the prompts

  ---

  Explanation

  The most effective way to preprocess prompts using custom code is to write a custom model, such as an MLflow PyFunc model . Here's a breakdown of why this is the correct approach:

  MLflow PyFunc Models:MLflow is a widely used platform for managing the machine learning lifecycle, including experimentation, reproducibility, and deployment. A PyFunc model is a generic Python function model that can implement custom logic, which includes preprocessing prompts.

  Preprocessing Prompts:Preprocessing could include various tasks like cleaning up the user input, formatting it according to specific rules, or augmenting it with additional context before passing it to the LLM. Writing this preprocessing as part

  of a PyFunc model allows the custom code to be managed, tested, and deployed easily.

  Modular and Reusable:By separating the preprocessing logic into a PyFunc model, the system becomes modular, making it easier to maintain and update without needing to modify the core LLM or retrain it.

  Why Other Options Are Less Suitable:

  A (Modify LLM's Internal Architecture): Directly modifying the LLM's architecture is highly impractical and can disrupt the model's performance. LLMs are typically treated as black-box models for tasks like prompt processing.

  B (Avoid Custom Code): While it's true that LLMs haven't been explicitly trained with preprocessed prompts, preprocessing can still improve clarity and alignment with desired input formats without confusing the model.

  C (Postprocessing Outputs): While postprocessing the output can be useful, it doesn't address the need for clean and well-formatted inputs, which directly affect the quality of the model's responses.

  Thus, using an MLflow PyFunc model allows for flexible and controlled preprocessing of prompts in a scalable way, making it the most effective method.

• Question 33:

  A small and cost-conscious startup in the cancer research field wants to build a RAG application using Foundation Model APIs.

  Which strategy would allow the startup to build a good-quality RAG application while being cost-conscious and able to cater to customer needs?

  A. Limit the number of relevant documents available for the RAG application to retrieve from
  B. Pick a smaller LLM that is domain-specific
  C. Limit the number of queries a customer can send per day
  D. Use the largest LLM possible because that gives the best performance for any general queries
  B. Pick a smaller LLM that is domain-specific

  ---

  Explanation

  For a small, cost-conscious startup in the cancer research field, choosing a domain-specific and smaller LLM is the most effective strategy. Here's why is the best choice: B

  Domain-specific performance: A smaller LLM that has been fine-tuned for the domain of cancer research will outperform a general-purpose LLM for specialized queries. This ensures high-quality responses without needing to rely on a large,

  expensive LLM.

  Cost-efficiency: Smaller models are cheaper to run, both in terms of compute resources and API usage costs. A domain-specific smaller LLM can deliver good quality responses without the need for the extensive computational power required

  by larger models.

  Focused knowledge: In a specialized field like cancer research, having an LLM tailored to the subject matter provides better relevance and accuracy for queries, while keeping costs low.Large, general- purpose LLMs may provide irrelevant

  information, leading to inefficiency and higher costs.

  This approach allows the startup to balance quality, cost, and customer satisfaction effectively, making it the most suitable strategy.

• Question 34:

  A Generative AI Engineer is testing a simple prompt template in LangChain using the code below, but is getting an error.

  

  Assuming the API key was properly defined, what change does the Generative AI Engineer need to make to fix their chain?

  

  A. Option A
  B. Option B
  C. Option C
  D. Option D
  D. Option D

  ---

  Explanation

  from langchain.chains import LLMChain from langchain_community.llms import OpenAI from langchain_core.prompts import PromptTemplate prompt_template = "Tell me a {adjective} joke" prompt = PromptTemplate( input_variables=["adjective"], template=prompt_template ) llm = LLMChain(llm=OpenAI(), prompt=prompt)

• Question 35:

  A Generative AI Engineer is designing a chatbot for a gaming company that aims to engage users on its platform while its users play online video games.

  Which metric would help them increase user engagement and retention for their platform?

  A. Randomness
  B. Diversity of responses
  C. Lack of relevance
  D. Repetition of responses
  B. Diversity of responses

  ---

  Explanation

  In the context of designing a chatbot to engage users on a gaming platform, diversity of responses (option B) is a key metric to increase user engagement and retention. Here's why:

  Diverse and Engaging Interactions:A chatbot that provides varied and interesting responses will keep users engaged, especially in an interactive environment like a gaming platform. Gamers typically enjoy dynamic and evolving

  conversations, and diversity of responses helps prevent monotony, encouraging users to interact more frequently with the bot.

  Increasing Retention:By offering different types of responses to similar queries, the chatbot can create a sense of novelty and excitement, which enhances the user's experience and makes them more likely to return to the platform.

  Why Other Options Are Less Effective:

  A (Randomness): Random responses can be confusing or irrelevant, leading to frustration and reducing engagement.

  C (Lack of Relevance): If responses are not relevant to the user's queries, this will degrade the user experience and lead to disengagement.

  D (Repetition of Responses): Repetitive responses can quickly bore users, making the chatbot feel uninteresting and reducing the likelihood of continued interaction.

  Thus, diversity of responses (option B) is the most effective way to keep users engaged and retain them on the platform.

• Question 36:

  A Generative AI Engineer is creating an agent-based LLM system for their favorite monster truck team. The system can answer text based questions about the monster truck team, lookup event dates via an API call, or query tables on the team's latest standings.

  How could the Generative AI Engineer best design these capabilities into their system?

  A. Ingest PDF documents about the monster truck team into a vector store and query it in a RAG architecture.
  B. Write a system prompt for the agent listing available tools and bundle it into an agent system that runs a number of calls to solve a query.
  C. Instruct the LLM to respond with "RAG", "API", or "TABLE" depending on the query, then use text parsing and conditional statements to resolve the query.
  D. Build a system prompt with all possible event dates and table information in the system prompt. Use a RAG architecture to lookup generic text questions and otherwise leverage the information in the system prompt.
  B. Write a system prompt for the agent listing available tools and bundle it into an agent system that runs a number of calls to solve a query.

  ---

  Explanation

  In this scenario, the Generative AI Engineer needs to design a system that can handle different types of queries about the monster truck team. The queries may involve text-based information, API lookups for event dates, or table queries for

  standings. The best solution is to implement a tool-based agent system .

  Here's how option B works, and why it's the most appropriate answer:

  System Design Using Agent-Based Model:In modern agent-based LLM systems, you can design a system where the LLM (Large Language Model) acts as a central orchestrator. The model can "decide" which tools to use based on the

  query. These tools can include API calls, table lookups, or natural language searches. The system should contain a system prompt that informs the LLM about the available tools.

  System Prompt Listing Tools:By creating a well-crafted system prompt , the LLM knows which tools are at its disposal. For instance, one tool may query an external API for event dates, another might look up standings in a database, and a

  third may involve searching a vector database for general text-based information. The agent will be responsible for calling the appropriate tool depending on the query.

  Agent Orchestration of Calls:The agent system is designed to execute a series of steps based on the incoming query. If a user asks for the next event date, the system will recognize this as a task that requires an API call. If the user asks

  about standings, the agent might query the appropriate table in the database. For text-based questions, it may call a search function over ingested data. The agent orchestrates this entire process, ensuring the LLM makes calls to the right

  resources dynamically.

  Generative AI Tools and Context:This is a standard architecture for integrating multiple functionalities into a system where each query requires different actions. The core design in option B is efficient because it keeps the system modular and

  dynamic by leveraging tools rather than overloading the LLM with static information in a system prompt (like option D).

  Why Other Options Are Less Suitable:

  A (RAG Architecture): While relevant, simply ingesting PDFs into a vector store only helps with text-based retrieval. It wouldn't help with API lookups or table queries.

  C (Conditional Logic with RAG/API/TABLE): Although this approach works, it relies heavily on manual text parsing and might introduce complexity when scaling the system.

  D (System Prompt with Event Dates and Standings): Hardcoding dates and table information into a system prompt isn't scalable. As the standings or events change, the system would need constant updating, making it inefficient.

  By bundling multiple tools into a single agent-based system (as in option B), the Generative AI Engineer can best handle the diverse requirements of this system.

• Question 37:

  A Generative AI Engineer just deployed an LLM application at a digital marketing company that assists with answering customer service inquiries. Which metric should they monitor for their customer service LLM application in production?

  A. Number of customer inquiries processed per unit of time
  B. Energy usage per query
  C. Final perplexity scores for the training of the model
  D. HuggingFace Leaderboard values for the base LLM
  A. Number of customer inquiries processed per unit of time

  ---

  Explanation

  When deploying an LLM application for customer service inquiries, the primary focus is on measuring the operational efficiency and quality of the responses. Here's why is the correct metric: A

  Number of customer inquiries processed per unit of time: This metric tracks the throughput of the customer service system, reflecting how many customer inquiries the LLM application can handle in a given time period (e.g., per minute or hour). High throughput is crucial in customer service applications where quick response times are essential to user satisfaction and business efficiency. Real-time performance monitoring: Monitoring the number of queries processed is an important part of ensuring that the model is performing well under load, especially during peak traffic times. It also helps ensure the system scales properly

  to meet demand.

  Why other options are not ideal:

  B. Energy usage per query: While energy efficiency is a consideration, it is not the primary concern for a customer-facing application where user experience (i.e., fast and accurate responses) is critical.

  C. Final perplexity scores for the training of the model: Perplexity is a metric for model training, but it doesn't reflect the real-time operational performance of an LLM in production.

  D. HuggingFace Leaderboard values for the base LLM: The HuggingFace Leaderboard is more relevant during model selection and benchmarking. However, it is not a direct measure of the model's performance in a specific customer service application in production.

  Focusing on throughput (inquiries processed per unit time) ensures that the LLM application is meeting business needs for fast and efficient customer service responses.

• Question 38:

  A Generative Al Engineer interfaces with an LLM with prompt/response behavior that has been trained on customer calls inquiring about product availability. The LLM is designed to output "In Stock" if the product is available or only the term "Out of Stock" if not.

  Which prompt will work to allow the engineer to respond to call classification labels correctly?

  A. Respond with "In Stock" if the customer asks for a product.
  B. You will be given a customer call transcript where the customer asks about product availability. The outputs are either "In Stock" or "Out of Stock". Format the output in JSON, for example: {"call_id": "123", "label": "In Stock"}.
  C. Respond with "Out of Stock" if the customer asks for a product.
  D. You will be given a customer call transcript where the customer inquires about product availability. Respond with "In Stock" if the product is available or "Out of Stock" if not.
  B. You will be given a customer call transcript where the customer asks about product availability. The outputs are either "In Stock" or "Out of Stock". Format the output in JSON, for example: {"call_id": "123", "label": "In Stock"}.

  ---

  Explanation

  Problem Context: The Generative AI Engineer needs a prompt that will enable an LLM trained on customer call transcripts to classify and respond correctly regarding product availability. The desired response should clearly indicate whether a

  product is "In Stock" or "Out of Stock," and it should be formatted in a way that is structured and easy to parse programmatically, such as JSON.

  Explanation of Options:

  Option A: Respond with "In Stock" if the customer asks for a product. This prompt is too generic and does not specify how to handle the case when a product is not available, nor does it provide a structured output format.

  Option B: This option is correctly formatted and explicit. It instructs the LLM to respond based on the availability mentioned in the customer call transcript and to format the response in JSON. This structure allows for easy integration into

  systems that may need to process this information automatically, such as customer service dashboards or databases.

  Option C: Respond with "Out of Stock" if the customer asks for a product. Like option A, this prompt is also insufficient as it only covers the scenario where a product is unavailable and does not provide a structured output.

  Option D: While this prompt correctly specifies how to respond based on product availability, it lacks the structured output format, making it less suitable for systems that require formatted data for further processing.

  Given the requirements for clear, programmatically usable outputs, Option B is the optimal choice because it provides precise instructions on how to respond and includes a JSON format example for structuring the output, which is ideal for

  automated systems or further data handling.

• Question 39:

  A Generative AI Engineer is building an LLM to generate article summaries in the form of a type of poem, such as a haiku, given the article content. However, the initial output from the LLM does not match the desired tone or style. Which approach will NOT improve the LLM's response to achieve the desired response?

  A. Provide the LLM with a prompt that explicitly instructs it to generate text in the desired tone and style
  B. Use a neutralizer to normalize the tone and style of the underlying documents
  C. Include few-shot examples in the prompt to the LLM
  D. Fine-tune the LLM on a dataset of desired tone and style
  B. Use a neutralizer to normalize the tone and style of the underlying documents

  ---

  Explanation

  The task at hand is to improve the LLM's ability to generate poem-like article summaries with the desired tone and style. Using a neutralizer to normalize the tone and style of the underlying documents (option B) will not help improve the

  LLM's ability to generate the desired poetic style. Here's why:

  Neutralizing Underlying Documents:A neutralizer aims to reduce or standardize the tone of input data. However, this contradicts the goal, which is to generate text with a specific tone and style (like haikus). Neutralizing the source documents

  will strip away the richness of the content, making it harder for the LLM to generate creative, stylistic outputs like poems.

  Why Other Options Improve Results:

  A (Explicit Instructions in the Prompt): Directly instructing the LLM to generate text in a specific tone and style helps align the output with the desired format (e.g., haikus). This is a common and effective technique in prompt engineering.

  C (Few-shot Examples): Providing examples of the desired output format helps the LLM understand the expected tone and structure, making it easier to generate similar outputs.

  D (Fine-tuning the LLM): Fine-tuning the model on a dataset that contains examples of the desired tone and style is a powerful way to improve the model's ability to generate outputs that match the target format.

  Therefore, using a neutralizer (option B) is not an effective method for achieving the goal of generating stylized poetic summaries.

• Question 40:

  A Generative Al Engineer has successfully ingested unstructured documents and chunked them by document sections. They would like to store the chunks in a Vector Search index. The current format of the dataframe has two columns: (i) original document file name (ii) an array of text chunks for each document.

  What is the most performant way to store this dataframe?

  A. Split the data into train and test set, create a unique identifier for each document, then save to a Delta table
  B. Flatten the dataframe to one chunk per row, create a unique identifier for each row, and save to a Delta table
  C. First create a unique identifier for each document, then save to a Delta table
  D. Store each chunk as an independent JSON file in Unity Catalog Volume. For each JSON file, the key is the document section name and the value is the array of text chunks for that section
  B. Flatten the dataframe to one chunk per row, create a unique identifier for each row, and save to a Delta table

  ---

  Explanation

  Problem Context: The engineer needs an efficient way to store chunks of unstructured documents to facilitate easy retrieval and search. The current dataframe consists of document filenames and associated text chunks.

  Explanation of Options:

  Option A: Splitting into train and test sets is more relevant for model training scenarios and not directly applicable to storage for retrieval in a Vector Search index.

  Option B: Flattening the dataframe such that each row contains a single chunk with a unique identifier is the most performant for storage and retrieval. This structure aligns well with how data is indexed and queried in vector search

  applications, making it easier to retrieve specific chunks efficiently.

  Option C: Creating a unique identifier for each document only does not address the need to access individual chunks efficiently, which is critical in a Vector Search application.

  Option D: Storing each chunk as an independent JSON file creates unnecessary overhead and complexity in managing and querying large volumes of files.

  Option is the most efficient and practical approach, allowing for streamlined indexing and retrieval processes B in a Delta table environment, fitting the requirements of a Vector Search index.