Model quality drift in Watson OpenScale drift v2 metrics

Last updated: Jul 27, 2023

Watson OpenScale calculates model quality drift by comparing the estimated runtime accuracy to the training accuracy to measure the drop in accuracy.

How it works

Watson OpenScale builds its own drift detection model that processes your payload data when you configure drift v2 evaluations to predict whether your model generates accurate predictions without the ground truth. The drift detection model uses the input features and class probabilities from your model to create its own input features.

Do the math

Watson OpenScale uses the following formula to calculate model quality drift:

model quality score

Watson OpenScale calculates the accuracy of your model as the base_accuracy by measuring the fraction of correctly predicted transactions in your training data. During evaluations, your transactions are scored against the drift detection model to measure the amount of transactions that are likely predicted correctly by your model. These transactions are compared to the total number of transactions that Watson OpenScale processes to calculate the predicted_accuracy. If the predicted_accuracy is less than the base_accuracy, Watson OpenScale generates a model quality drift score.