Configuring the drift detection monitor

You must configure the IBM Watson OpenScale drift monitor before it can analyze your model. You can train your drift detection model online by using the IBM Watson OpenScale user interface or by using a notebook.

You can train your model online by using Watson OpenScale if you use IBM Watson Machine Learning and your data does not exceed 500 MB. Otherwise, you must use a notebook to train the model.

Requirements

Throughout this process, IBM Watson OpenScale analyzes your model and makes recommendations based on the most logical outcome. For the drift monitor to work properly, the data type of your prediction column in the training data must match the data type of the same column in the payload data. Assign matching string or numeric types to the prediction and label columns. To confirm data types, click Model details > Model output details > Edit. These selections ensure that you have accurate information for the following configuration steps. If for some reason you must change data types, you must redeploy the monitor to effect the changes.

On the successive pages of the Drift tab, you must provide the following information:

Alert threshold

(Required only for classification type models.) IBM Watson OpenScale tracks the degree of change in model accuracy when compared to accuracy at training time. The alert threshold, which must be at least 5%, indicates the degree of tolerance for change over time.

Sample size

By setting a minimum sample size, you prevent measuring drift until a minimum number of records are available in the evaluation data set. This setting ensures that the sample size is not too small to skew results. Every time drift checking runs, it uses the minimum sample size to decide the number of records on which it does the computation.

Steps to configure drift in Watson OpenScale

If you use IBM Watson Machine Learning, you have the option of using the Watson OpenScale user interface to configure drift detection.

To start the configuration process, from the Drift tab, in the Drift model box, click the Edit The edit icon icon. Use the Train in Watson OpenScale option.

Follow the prompts and enter required information. When you finish, a summary of your selections is presented for review. If you want to change anything, click the Edit icon for that section, otherwise, save your work.

https://www.ibm.com/support/producthub/icpdata/docs/content/SSQNUZ_current/wos/wos-monitor-drift.html#behavior-drift-config-steps-ntbk

Steps to configure drift without retraining

Reconfigure the drift monitor without retraining the drift model to update parameters without more processing. You update the minimum sample size and threshold to produce more data on the currently trained model without incurring more processing costs. It is one way to avoid intensive CPU usage when the underlying data is stable and you want to view drift magnitude with different thresholds. Your drift model requires retraining only when training data or schema changes.

To start the configuration process, from the Drift tab, in the Drift threshold box or Sample size box, click the Edit The edit icon icon. Update the current setting and save it.

Steps to configure drift by using a notebook

Use a notebook to configure drift in the following circumstances:

  • You do not want to share the training data with Watson OpenScale.
  • You do not have a means to share the training data on Db2 or IBM Cloud Object Storage, which are the only two training data locations that are supported by Watson OpenScale.

This option is useful if the training data is not stored in Db2 or IBM Cloud Object Storage. Using a notebook, you must read the training data into a dataframe. The specialized notebook that you can download from Watson OpenScale then creates a specialized output that you can upload to Watson OpenScale.

To generate the drift detection model, you must use scikit-learn version 0.20.2 in notebooks.

Create a notebook to generate the drift detection model. Use the sample notebook that is available from the Watson OpenScale UI. The drift detection model is converted into a .tar.gz file for you.

To start the configuration process, from the Drift tab, in the Drift model box, click the Edit The edit icon icon. Use the Train in a data science notebook option. You can drag your compressed drift detection model to the drop zone.

Follow the prompts and enter required information. When you finish, a summary of your selections is presented for review. If you want to change anything, click the Edit icon for that section, otherwise, save your work.

Next steps