ISTQB Certified Tester - AI Testing (v 1.0)

The question asks which test is least likely to be performed during the ML model testing phase. Let's consider each option:

Testing the accuracy of the classification model (A): Accuracy testing is a fundamental part of the ML model testing phase. It ensures that the model correctly classifies the data as intended and meets the required performance metrics.

Testing the API of the service powered by the ML model (B): Testing the API is crucial, especially if the ML model is deployed as part of a service. This ensures that the service integrates well with other systems and that the API performs as expected.

Testing the speed of the training of the model (C): This is least likely to be part of the ML model testing phase. The speed of training is more relevant during the development phase when optimizing and tuning the model. During testing, the focus is more on the model's performance and behavior rather than how quickly it was trained.

Testing the speed of the prediction by the model (D): Testing the speed of prediction is important to ensure that the model meets performance requirements in a production environment, especially for real-time applications.

ISTQB CT-AI Syllabus Section 3.2 on ML Workflow and Section 5 on ML Functional Performance Metrics discuss the focus of testing during the model testing phase, which includes accuracy and prediction speed but not the training speed.

Bias detection at the data level is performed by reviewing data acquisition and preprocessing steps, as explained in Section 2.3 - Data Quality and Bias of the ISTQB CT-AI syllabus. Sample bias can arise when data is distorted or when preprocessing introduces unintended shifts, for example through filtering, normalization, or labeling steps that disproportionately affect certain subsets of the data. Option B correctly reflects this approach: during preprocessing review, testers can identify whether the data has been influenced in a way that could lead to sample distortions.

Option A is incorrect because algorithmic bias originates from the model and its learning process, not directly from data acquisition procedures.
Option C is incorrect because LIME is an explainability technique applied to trained models, not a method used during data source or preprocessing reviews.
Option D is incorrect because preprocessing affects sample bias rather than algorithmic bias.

Thus, Option B correctly matches the syllabus guidance on detecting bias during reviews of data sources, acquisition, and preprocessing.

The ISTQB CT-AI syllabus (Section 5.3 - AI Support for Defect Analysis) explains that AI can categorize defect reports using natural language processing or classification models. Categorization helps route defects efficiently and determine which areas of the system are affected. Thus, Option C is correct: AI can identify defect categories, supporting assignment and triage.

Option A is incorrect because machine learning models cannot reliably infer severity or real-world consequences without explicit domain-specific risk modeling.
Option B is incorrect because duplicate defect detection is a separate activity from assessing defect criticality; the syllabus clearly distinguishes
duplicate identification from severity analysis. Option D is incorrect because, while AI may support defect routing based on categories, the syllabus does not state that AI identifies specific developers responsible for fixing defects.

Thus, Option C is the syllabus-accurate statement.

The machine learning workflow typically proceeds iteratively, but once the objectives, business priorities, framework, and model have been agreed upon, the next step is data preparation. The ISTQB CT-AI syllabus explains that data preparation is required to provide suitable input data for training an ML algorithm and for evaluating the resulting model.

The syllabus notes that ML models cannot be trained or evaluated without properly prepared data and that data preparation is therefore a prerequisite for subsequent steps such as training, tuning, and evaluation.
This includes collecting, cleaning, transforming, and splitting the data for training and testing purposes.

This directly supports Option A as the correct next step in the machine learning workflow.

The scenario describes a model that shows good accuracy in isolation but performs poorly in real-world operation, particularly in detecting true cancer cases. This strongly indicates a generalization problem.

A lack of similarity between the training and testing data means that the training data was not sufficiently representative of real-world diagnostic data. This is a well-known ML risk that results in models performing well during development but failing in operational environments.

While input data quality (Option B) is important, the issue described is not primarily about corrupted or invalid data but about representativeness.

Choosing inappropriate functional performance metrics (Option C) could hide problems, but it would not by itself cause poor real-world detection if the data were representative.

Non-functional requirements testing (Option D) is important but does not directly explain the inability to
detect real cancer cases.

The ISTQB CT-AI syllabus (Section 4.2 - Training, Validation, and Test Datasets) emphasizes that a lack of representative data across training, validation, and operational environments is a major cause of poor
real-world ML performance.

Therefore, Option A is the most likely deficiency discovered by the test expert.

The ISTQB CT-AI syllabus defines flexibility and adaptability in Section 2.1 - Flexibility and Adaptability.
Flexibility is described as the ability of a system to operate in situations not explicitly covered by its original requirements, whereas adaptability refers to how easily the system can be modified to handle new environments or operating conditions. The syllabus emphasizes that both properties are especially important for self-learning AI-based systems, which may need to adjust their behavior autonomously as conditions change.

It further explains that such systems must be capable of determining when and how to adapt their behavior in evolving or previously unknown situations. This directly aligns with Option A, which highlights the importance of both adaptability and flexibility when a system needs to change its behavior and determine those changes on its own.

Option B is incorrect because it reverses the definitions: operating in unspecified situations relates to flexibility, not adaptability.
Option C is incorrect because self-learning AI-based systems are not classified as either adaptable-only or flexible-only; they typically require both characteristics. Option D is incorrect
because ease of reprogramming corresponds to adaptability, not flexibility.

Thus, Option A correctly reflects the syllabus definitions.

Section 1.7 - Neural Networks of the ISTQB CT-AI syllabus explains that neural networks consist of neurons connected by weighted links. During training, learning occurs by adjusting the weights on these connections. This is the essence of gradient descent and backpropagation. Option B correctly states this behavior: only the weights are modified, not the activation functions, the number of neurons, or the architectural structure.

Option A is incorrect because a neuron's bias is not determined by the activations of neurons in the previous layer; it is an independent trainable parameter added to the weighted sum of inputs. Option C is
incorrect because the syllabus states that a single-layer perceptron is a valid type of neural network, although it is limited to linearly separable problems. Option D is incorrect because there is no requirement that the number of input neurons be greater than or equal to the number of output neurons. Input neurons correspond to the number of features, while output neurons correspond to tasks or classes.

Therefore, Option B precisely reflects the syllabus definition of what changes during neural network training.

According to the ISTQB CT-AI syllabus, metamorphic testing works by applying metamorphic relations (MRs): predictable input transformations that should lead to predictable output changes. From the initial test data, clear relations emerge for defensive driving scoring. The score increases when: v_mean decreases, a_pos decreases, a_neg becomes less negative, and decreases when the opposite occurs.

A valid metamorphic follow-up test must modify inputs in a direction consistent with at least one metamorphic relation while keeping the expected output direction predictable.

Test A lowers v_mean compared to Test 1, with similar acceleration values. This directly satisfies the metamorphic relation that lower speed leads to a higher score.

Test C increases both acceleration and braking intensity compared to Test 2, making the reduced score range (30-70) consistent with more aggressive driving.

Test D modifies acceleration and braking magnitudes in ways consistent with Test 3's defensive-driving scoring boundaries.

Test B, however, changes multiple variables in contradictory directions:
v_mean increases (worse),
a_pos increases (worse), a_neg becomes less negative (better).

Because these changes conflict, the expected score trend becomes unpredictable, violating the premise of a metamorphic follow-up test.

Thus, Test B cannot be considered a metamorphic follow-up test, which makes Option C correct.

The ISTQB CT-AI syllabus describes A/B testing (also known as split testing) as a technique used to compare two variants of a system, feature, or model under the same conditions, in order to determine which performs better according to defined metrics.

Options A, B, and D all align with this definition. They involve comparing two alternatives (websites, offers, or ML implementations) while keeping the conditions consistent, such as the same users or the same input data.

Option C, however, compares the performance of a single ML system on two different input datasets. This changes the test conditions rather than the system variant, and therefore does not represent an A/B testing scenario. Instead, it reflects dataset evaluation or robustness testing.

Thus, Option C is the least representative of A/B testing.