0 / 0
Frequently asked questions for Watson OpenScale
Frequently asked questions for Watson OpenScale

Frequently asked questions for Watson OpenScale

Find answers to frequently asked questions about Watson OpenScale.

Watson OpenScale

What is Watson OpenScale

IBM Watson OpenScale tracks and measures outcomes from your AI models, and helps ensure they remain fair, explainable, and compliant wherever your models were built or are running. Watson OpenScale also detects and helps correct the drift in accuracy when an AI model is in production

How is Watson OpenScale priced?

There's a Standard pricing plan, for which you are charged per model per month. The up-to-date information is available in the IBM Cloud catalog.

Is there a free trial for Watson OpenScale?

Watson OpenScale offers a free trial plan. To sign up, see Watson OpenScale web page and click Get started now. You can use the free plan, subject to usage limits that reset every month.

Does Watson OpenScale work with Microsoft Azure ML engine?

Watson OpenScale supports both Microsoft Azure ML Studio and Microsoft Azure ML Service engines. For more information, see Microsoft Azure ML Studio frameworks and Microsoft Azure ML Service frameworks.

Does Watson OpenScale work with Amazon SageMaker ML engine?

Watson OpenScale supports Amazon SageMaker ML engine. For more information, see Amazon SageMaker frameworks.

Is Watson OpenScale available on IBM Cloud Pak for Data?

Watson OpenScale is one of the included services for IBM Cloud Pak for Data.

To run Watson OpenScale on my own servers, how much computer processing power is required?

There are specific guidelines for hardware configuration for three-node and six-node configurations. Your IBM Technical Sales team can also help you with sizing your specific configuration. Because Watson OpenScale run as an add-on to IBM Cloud Pak for Data, you need to consider the requirements for both software products.

How do I convert a prediction column from an integer data type to a categorical data type?

For fairness monitoring, the prediction column allows only an integer numerical value even though the prediction label is categorical. How do I configure a categorical feature that is not an integer? Is a manual conversion required?

The training data might have class labels such as “Loan Denied”, “Loan Granted”. The prediction value that is returned by IBM Watson Machine Learning scoring end point has values such as “0.0”, “1.0". The scoring end point also has an optional column that contains the text representation of prediction. For example, if prediction=1.0, the predictionLabel column might have a value “Loan Granted”. If such a column is available, when you configure the favorable and unfavorable outcome for the model, specify the string values “Loan Granted” and “Loan Denied”. If such a column is not available, then you need to specify the integer and double values of 1.0, 0.0 for the favorable, and unfavorable classes.

IBM Watson Machine Learning has a concept of output schema that defines the schema of the output of IBM Watson Machine Learning scoring end point and the role for the different columns. The roles are used to identify which column contains the prediction value, which column contains the prediction probability, and the class label value, etc. The output schema is automatically set for models that are created by using model builder. It can also be set by using the IBM Watson Machine Learning Python client. Users can use the output schema to define a column that contains the string representation of the prediction. Set the modeling_role for the column to ‘decoded-target’. The documentation for the IBM Watson Machine Learning Python client is available at: http://wml-api-pyclient-dev.mybluemix.net/#repository. Search for “OUTPUT_DATA_SCHEMA” to understand the output schema. The API call to use is the store_model call that accepts the OUTPUT_DATA_SCHEMA as a parameter.

Why does Watson OpenScale need access to training data?

You must either provide Watson OpenScale access to training data that is stored in Db2 or IBM Cloud Object Storage, or you must run a Notebook to access the training data.

Watson OpenScale needs access to your training data for the following reasons:

  • To generate contrastive explanations: To create explanations, access to statistics, such as median value, standard deviation, and distinct values from the training data is required.
  • To display training data statistics: To populate the bias details page, Watson OpenScale must have training data from which to generate statistics.
  • To build a drift detection model: The Drift monitor uses training data to create and calibrate drift detection.

In the Notebook-based approach, you are expected to upload the statistics and other information when you configure a deployment in Watson OpenScale. Watson OpenScale no longer has access to the training data outside of the Notebook, which is run in your environment. It has access only to the information uploaded during the configuration.

What internet browser does Watson OpenScale support?

The Watson OpenScale service requires the same level of browser software as is required by IBM Cloud. See the IBM Cloud Prerequisites topic for details.

Is there a command-line tool to use?

Yes! There is a ModelOps CLI tool, whose official name is the Watson OpenScale CLI model operations tool. Use it to run tasks related to the lifecycle management of machine learning models. This tool is complementary to the IBM Cloud CLI tool, augmented with the machine learning plug-in.

What version of Python can I use with Watson OpenScale?

Because Watson OpenScale is independent of your model-creation process, it supports whatever Python versions your machine learning provider supports. The Watson OpenScale Python client is a Python library that works directly with the Watson OpenScale service on IBM Cloud. For the most up-to-date version information, see the Requirements section. You can use the Python client, instead of the Watson OpenScale client UI, to directly configure a logging database, bind your machine learning engine, and select and monitor deployments. For examples of using the Python client in this way, see the Watson OpenScale sample Notebooks.

What does it mean if the fairness score is greater than 100 percent?

Depending on your fairness configuration, your fairness score can exceed 100 percent. It means that your monitored group is getting relatively more “fair” outcomes as compared to the reference group. Technically, it means that the model is unfair in the opposite direction.

Configuring a model requires information about the location of the training data and the options are Cloud Object Storage and Db2. If the data is in Netezza, can Watson OpenScale use Netezza?

Use this Watson OpenScale Notebook to read the data from Netezza and generate the training statistics and also the drift detection model.

Why doesn't Watson OpenScale see the updates that were made to the model?

Watson OpenScale works on a deployment of a model, not on the model itself. You must create a new deployment and then configure this new deployment as a new subscription in Watson OpenScale. With this arrangement, you are able to compare the two versions of the model.

How is fairness calculated in Watson OpenScale?

In Watson OpenScale, fairness is calculated by using disparate impact ratio and by perturbing monitored groups and reference groups. For more information, see Fairness metrics overview.

How is model bias mitigated by using Watson OpenScale?

The debiasing capability in Watson OpenScale is enterprise grade. It is robust, scalable and can handle a wide variety of models. Debiasing in Watson OpenScale consists of a two-step process: Learning Phase: Learning customer model behavior to understand when it acts in a biased manner.

Application Phase: Identifying whether the customer’s model acts in a biased manner on a specific data point and, if needed, fixing the bias. For more information, see Understanding how debiasing works and Debiasing options.

Is it possible to check for model bias on sensitive attributes, such as race and sex, even when the model is not trained on them?

Yes. Recently, Watson OpenScale delivered a ground-breaking feature called “Indirect Bias detection.” Use it to detect whether the model is exhibiting bias indirectly for sensitive attributes, even though the model is not trained on these attributes. For more information, see Understanding how debiasing works.

Is it possible to mitigate bias for regression-based models?

Yes. You can use Watson OpenScale to mitigate bias on regression-based models. No additional configuration is needed from you to use this feature. Bias mitigation for regression models is done out-of-box when the model exhibits bias.

What are the different methods of debiasing in Watson OpenScale?

You can use both Active Debiasing and Passive Debiasing for debiasing. For more information, see Debiasing options.

Can I configure model fairness through an API?

Yes, it is possible with the Watson OpenScale SDK. For more information, see IBM Watson OpenScale Python SDK documentation!.

What are various model frameworks supported in Watson OpenScale?

For the list of supported machine learning engines, frameworks, and models see the Watson OpenScale documentation Supported machine learning engines, frameworks, and models.

What are the supported machine learning providers for Watson OpenScale?

For the list of supported machine learning engines, frameworks, and models see the Watson OpenScale documentation Supported machine learning engines, frameworks, and models.

What are the various kinds of risks associated in using a machine learning model?

Multiple kinds of risks that are associated with machine learning models, such as any change in input data that is also known as Drift can cause the model to make inaccurate decisions, impacting business predictions. Training data can be cleaned to be free from bias but runtime data might induce biased behavior of model.

Traditional statistical models are simpler to interpret and explain, but unable to explain the outcome of the machine learning model can pose a serious threat to the usage of the model.

For more information, see Manage model risk .

What are the monitors that are available in Watson OpenScale?

In Watson OpenScale the machine learning models can be monitored for fairness, quality, drift (both model and data drift), and be able explain the transactions. And along with these Watson OpenScale provides the capability to plug-in custom monitors that customers can develop and hook it with Watson OpenScale.

Does Watson OpenScale detect drift in accuracy and drift in data?

Watson OpenScale detects both drift in accuracy and drift in data:

  • Drift in accuracy estimates the drop in accuracy of the model at run time. Model accuracy drops when there is an increase in transactions that are similar to those that the model did not evaluate correctly in the training data.
  • This type of drift is calculated for structured binary and multi-class classification models only. Whereas, drift in data estimates the drop in consistency of the data at runtime as compared to the characteristics of the data at training time.

What are the types of explanations shown in Watson OpenScale?

Watson OpenScale provides two types of explanations - Local explanation based on LIME, and Contrastive explanation. For more information, see Understanding the difference between contrastive explanations and LIME.

How do I infer from Local/LIME explanation from Watson OpenScale?

In in Watson OpenScale, LIME reveals which features played most important role in the model prediction for a specific data point. Along with the features their relative importance is also shown.

How do I infer contrastive explanation from Watson OpenScale?

Contrastive explanation in Watson OpenScale shows the minimum change to be made to the input data point that would give a different model prediction than the input data point.

What is what-if analysis in Watson OpenScale?

The explanations UI also provides ability to test what-if scenarios, where in the user can change the feature values of the input datapoint and check its impact on the model prediction and probability.

In Watson OpenScale, for which models is Local/LIME explanation supported?

Local explanation is supported for structured data models that use regression and classification problems and models that use unstructured text, unstructured image data, and only classification problems.

In Watson OpenScale, for which models is contrastive explanation and what-if analysis supported?

Contrastive explanations and what-if analyses are supported for models that use structured data and are of problem type classification only.

What are controllable features in Watson OpenScale explainability configuration?

Using controllable features some features of the input data point can be locked, so that they do not change when the contrastive explanation is generated and also they cannot be changed in what if analysis. The features that should not be changed should be set as non-controllable or NO in the explainability configuration.

While configuring the machine learning providers in Watson OpenScale, what is the difference between pre-production and production subscriptions?

Before putting a model into production, a model validator can configure and validate the model by using a pre-production service provider. Watson OpenScale provides the ability to designate a machine learning provider as part of pre-production processing, where you perform risk evaluations. After the model meets quality standards, then you can send the model to production. You can use different machine learning providers for pre-production and production, or you can use different instances of the same machine learning provider to keep pre-production and production environments segmented.

Must I keep monitoring the Watson OpenScale dashboard to make sure that my models behave as expected?

No, you can set up email alerts for your production model deployments in Watson OpenScale, so that you receive email alerts whenever a risk evaluation test fails, and then you can come and check the issues and address them.

How are IBM OpenPages and Watson OpenScale related in the overall model risk management arena?

IBM offers an end-to-end model risk management solution with IBM Watson OpenScale and IBM OpenPages with Watson. IBM OpenPages MRG offers model risk governance to store and manage a comprehensive model inventory. IBM Watson OpenScale monitors and measures outcomes from AI Models across its lifecycle and validates models.

For more information, see Configure model governance with IBM OpenPages MRG .

In a pre-production environment, that uses Watson OpenScale after the model is evaluated for risk and approved for usage, do I must reconfigure all the monitors again in production environment?

No, Watson OpenScale provides a way to copy the configuration of pre-production subscription to production subscription.

In Watson OpenScale, can I compare my model deployments in pre-production with a benchmark model to see how good or bad it is?

Yes, Watson OpenScale provides you with the option to compare two model deployments or subscriptions where you can see a side-by-side comparison of the behavior of the two models on each of the monitors configured. To compare go to the model summary page on Watson OpenScale dashboard and select Actions -> Compare.

Which Quality metrics are supported by Watson OpenScale?

Watson OpenScale supports 'Area under ROC', 'Area under Precision-Recall (PR)', 'Proportion explained variance', 'Mean absolute error', 'Mean squared error', 'R squared', 'Root of mean squared error', 'Accuracy', 'Weighted True Positive Rate', 'True positive rate', 'Weighted False Positive Rate', 'False positive rate', 'Weighted recall', 'Recall', 'Weighted precision', 'Precision', 'Weighted F1-Measure', 'F1-Measure', 'Logarithmic loss'.

Where can I find more information about the respective quality metrics that are monitored by Watson OpenScale?

You can find more about the metrics here: Supported quality metrics.

In Watson OpenScale, what data is used for Quality metrics computation?

Quality metrics are calculated that use manually labeled feedback data and monitored deployment responses for this data.

In Watson OpenScale, can the threshold be set for a metric other than 'Area under ROC' during configuration?

No, currently, the threshold can be set only for the 'Area under ROC' metric.

In Watson OpenScale, why are some of the configuration tabs disabled?

Some conditions enable particular tabs. You can see the reason why that tab is not enabled, by hovering your mouse over the circle icon on the tab.

Why an error “Training complete with errors” is shown on the UI when configuring drift?

It is because your drift model is partially configured. For more information, read the message that is shown on the UI by clicking the information icon in Drift Model tile.

What are the different kinds of drift that IBM Watson OpenScale detects?

Watson OpenScale detects both drift in model accuracy and drift in data.

What is model accuracy drift?

Watson OpenScale estimates the drop in accuracy of the model at run time. Model accuracy drops if there is an increase in transactions that are similar to those that the model did not evaluate correctly in the training data.

This type of drift is calculated for structured binary and multi-class classification models only.

What is data drift?

Watson OpenScale estimates the drop in consistency of the data at runtime as compared to the characteristics of the data at training time. This drop in consistency of data is also termed as data drift. This type of drift is calculated for all structured models.

Why should one be concerned about model accuracy drift or data drift?

A drop in either model accuracy or data consistency leads to a negative impact on the business outcomes that are associated with the model and must be addressed by retraining the model.

Are there any limitations for the drift monitor in IBM Watson OpenScale?

The following limitations apply to the drift monitor:

  • Drift is supported for structured data only.
  • Although classification models support both data and accuracy drift, regression models support only data drift.
  • Drift is not supported for Python functions.

How is drop in accuracy that is, model accuracy drift calculated in Watson OpenScale?

Watson OpenScale learns the behavior of the model by creating a proxy model, also known as a drift detection model. It looks at the training data and how the model is making predictions on the training data.

For more information, see Drift detection.

How is the drop in data consistency calculated in IBM Watson OpenScale?

IBM Watson OpenScale learns single and two-column constraints or boundaries on the training data at the time of configuration. It then analyzes all payload transactions to determine which transactions are causing drop in data consistency. For more information, see Drift in data.

Can Watson OpenScale detect drift in my classification model?

Yes, Watson OpenScale can detect both drop in model accuracy and drop in data consistency for structured classification models.

Can Watson OpenScale detect drift in my regression model?

Watson OpenScale can detect a drop in data consistency only for structured regression models.

Can Watson OpenScale detect drift in my model that is trained on text corpus?

Watson OpenScale cannot detect drift in text-based models as of now.

Can Watson OpenScale detect drift in my model that is trained on image data?

Watson OpenScale cannot detect drift in image-based models as of now.

Can Watson OpenScale detect drift in my Python function that is deployed on IBM Watson Machine Learning?

Watson OpenScale cannot detect drift in Python functions as of now.

Next steps