0 / 0
Drop in accuracy in Watson OpenScale drift metrics

Drop in accuracy in Watson OpenScale drift metrics

Watson OpenScale drift evaluations estimate the drop in accuracy of your model at run time when compared to the training data. The model accuracy drops if there is an increase in transactions similar to those that the model did not evaluate correctly in the training data.

How it works

The drift monitor works differently in pre-production and production environments.

In pre-production environments, when you upload labeled test data, the data is added to the feedback and payload tables. The labeled data is added as an annotation in the payload table. Accuracy is calculated with the labeled data column and the prediction column from the payload table.

In production environments, Watson OpenScale creates a drift detection model by looking at the data that was used to train and test the model. For example, if the model has an accuracy of 90% on the test data, it means that it provides incorrect predictions on 10% of the test data. Watson OpenScale builds a binary classification model that accepts a data point and predicts whether that data point is similar to the data that the model either incorrectly (10%) or accurately (90%) predicted.

After Watson OpenScale creates the drift detection model, at run time, it scores this model by using all of the data that the client model receives. For example, if the client model received 1000 records in the past 3 hours, Watson OpenScale runs the drift detection model on those same 1000 data points. It calculates how many of the records are similar to the 10% of records on which the model made an error when training. If 200 of these records are similar to the 10%, then it implies that the model accuracy is likely to be 80%. Because the model accuracy at training time was 90%, it means that there is an accuracy drift of 10% in the model.

To mitigate drift after it is detected by Watson OpenScale, you must build a new version of the model that fixes the problem. A good place to start is with the data points that are highlighted as reasons for the drift. Introduce the new data to the predictive model after you manually label the drifted transactions and use them to retrain the model.

Do the math

The drop in accuracy metric is calculated for structured binary and multi-class classification models only. Watson OpenScale analyzes each transaction to estimate if the model prediction is accurate. If the model prediction is inaccurate, the transaction is marked as drifted. The estimated accuracy is then calculated as the fraction of nondrifted transactions to the total number of transactions analyzed. The base accuracy is the accuracy of the model on the test data. Watson OpenScale calculates the extent of the drift in accuracy as the difference between Base accuracy and Estimated accuracy. Further, Watson OpenScale analyzes all the drifted transactions; and then, groups transactions based on the similarity of each feature's contribution to the drift in accuracy. In each cluster, Watson OpenScale also estimates the important features that contributed to the drift in accuracy and classifies their feature impact as large, some, and small.

Learn more

Reviewing drift results

Parent topic: Drift detection overview

Generative AI search and answer
These answers are generated by a large language model in watsonx.ai based on content from the product documentation. Learn more