Free SAP-C02 Web TestEngine Demo

Q99 Single choice

A retailer is deploying a customer service AI agent that uses multiple tools to look up orders, process refunds, and make address changes. The retailer needs a solution for its CI/CD pipeline that blocks promotion if a new release regresses on specific metrics. The specific metrics include task completion, tool selection, and final response quality. The retailer wants to test the metric evaluation against a set of golden interaction traces. After deployment, the solution must also continuously monitor for agent degradation and perform a periodic human audit of production results.

Which solution will meet these requirements with the LEAST operational overhead?

A
Build custom checks by using Amazon CloudWatch Logs and AWS Lambda to parse tool calls and response text. Use weekly manual reviews to decide whether the deployed AI agent meets performance standards.
B
Use an Amazon Bedrock model evaluation job on the underlying foundation model (FM) by using prompt-response pairs. Promote the release if the model's helpfulness and correctness scores improve over the previous release.
C
Configure agent trace logging. Add Amazon Bedrock AgentCore Evaluations to the pipeline for on-demand evaluation by using built-in and custom evaluators. Enable online evaluation after deployment. Periodically review a sampled subset of sessions.
D
Use Amazon Bedrock AgentCore Evaluations in online mode after deployment by using built-in evaluators. Use post-deployment rollback if the evaluation detects regression.

SAP-C02 Web TestEngine demo