Drift detection

Watson OpenScale detects both drift in accuracy and drift in data.

  • Drop in accuracy

    Estimates the drop in accuracy of the model at runtime. Model accuracy drops if there is an increase in transactions that are similar to those that the model did not evaluate correctly in the training data. This type of drift is calculated for structured binary and multi-class classification models only.

  • Drop in data consistency

    Estimates the drop in consistency of the data at runtime as compared to the characteristics of the data at training time.

A drop in either model accuracy or data consistency lead to a negative impact on the business outcomes that are associated with the model and must be addressed by retraining the model.

Drift visualization

The drift visualization includes both graphical and numeric statistical data. By clicking the chart, you can display specific transactions that contribute to drift. The top reasons for detected drift display and includes a natural-language description of the observation as well as a list of unexpected values.

Specifically, from the Select a transaction set from the chart or list below section, you can choose the following views:

  • Transactions responsible for drop in accuracy
  • Transactions responsible for drop in accuracy and data consistency
  • Transactions responsible for drop in data consistency
  • Drift transactions are available in the transaction details screen, where you can click Explain to understand how a specific transaction has made it into the drift category.

Enabling notifications

To send email notifications, click the Share the recommendations button. To enable this feature, you must first connect to an SMTP server that is configured in IBM Cloud Pak for Data. For more information, see Enabling email notifications. (IBM Watson OpenScale for IBM Cloud Pak for Data only.)

Limitations

The following limitations apply to the drift monitor:

  • Drift is supported for structured data only.
  • Although classification models support both data and accuracy drift, regression models support only data drift.
  • Drift is not supported for Python functions.

Go further

Read about a scenario that uses drift: