Configuring Watson OpenScale with advanced setup
You can use the advanced setup option to run a Python notebook that uses sample data to demonstrate how to configure model evaluations for Watson OpenScale.
Before you begin
This tutorial uses a Jupyter Notebook that must be run in a Watson Studio project. The notebook uses a "IBM Runtime 22.1 on Python 3.9 XS" runtime environment.
Ensure that you have provisioned Watson OpenScale service to your account. It also requires service credentials for the following IBM Cloud services:
- Cloud Object Storage (to store your Watson Studio project)
- IBM Watson Machine Learning
- Optional: Databases for PostgreSQL or Db2 Warehouse
The Jupyter Notebook does the following tasks:
- Trains, creates, and deploys a German Credit Risk model
- Configures Watson OpenScale to monitor that deployment
- Provides seven days' worth of historical records and measurements for viewing in the Watson OpenScale Insights dashboard.
You can also configure the model for continuous learning with Watson Studio and Spark.
Introduction
In this tutorial, you perform the following tasks:
- Provision IBM Cloud services
- Set up a Watson Studio project
- Create and deploy a machine learning model
- View results
Provision IBM Cloud Services
Log in to your IBM Cloud account with your IBMid. When provisioning services, particularly if you use Db2 Warehouse, verify that your selected organization and space are the same for all services.
Create a Watson Studio account
-
Create a Watson Studio instance if you do not already have one associated with your account:
-
Give your service a name, choose the Lite (free) plan, and click Create.
Provision an IBM Cloud Object Storage service
-
Provision an Cloud Object Storage service if you do not already one associated with your account:
-
Give your service a name, choose the Lite (free) plan, and click Create.
Provision an IBM Watson Machine Learning service
-
Provision a Machine Learning instance if you do not already have one associated with your account:
-
Give your service a name, choose the Lite (free) plan, and click Create.
Provision an IBM Watson OpenScale service
If you haven't already, ensure that you provision IBM Watson OpenScale.
-
Provision a Watson OpenScale instance if you do not already have one associated with your account:
-
Click Catalog > AI > Watson OpenScale.
- Give your service a name, choose a plan, and click Create.
- To start Watson OpenScale, click Get Started.
Optional: Provision a Databases for PostgreSQL or Db2 Warehouse service
If you have a paid IBM Cloud account, you can provision a Databases for PostgreSQL
or Db2 Warehouse
service to take full advantage of integration with Watson Studio and continuous learning services. If you choose
not to provision a paid service, you can use the free internal PostgreSQL storage with Watson OpenScale, but you are not able to configure continuous learning for your model.
-
Provision a Databases for PostgreSQL service or a Db2 Warehouse service if you do not already have one associated with your account:
-
Give your service a name, choose the Standard plan (Databases for PostgreSQL) or Entry plan (Db2 Warehouse), and click Create.
Set up a Watson Studio project
-
Log in to your Watson Studio account. Click the Avatar and verify that the account you are using is the same account you used to create your IBM Cloud services:
-
In Watson Studio, begin by creating a new project. Click the Create a project tile.
-
Click the Create an empty project tile.
-
Give your project a name and description, make sure that the IBM Cloud Object Storage service that you created is selected in Storage, and click Create.
Create and deploy a Machine Learning model
Add the Working with Watson Machine Learning
notebook to your Watson Studio project
-
Access the following file. If you have a GitHub account, you can sign in to clone the repository and download the file. Otherwise, you can view the raw version, by clicking Raw and copying the text of the file into a new file with the extension
.ipynb
. -
From the Assets tab in your Watson Studio project, click New asset and select Notebook:
-
Select From file:
-
Then, upload the file:
-
In the Select runtime section, choose the latest Python with Spark option.
Watson Studio does not offer a free environment with Python and Apache Spark. The only free runtime environment is for a Python-only environment.
-
Click Create.
Edit and run the Working with Watson Machine Learning
notebook
The Working with Watson Machine Learning
notebook contains detailed instructions for each step in the Python code that you run. As you work through the notebook, take some time to understand what each command is doing.
-
From the Assets tab in your Watson Studio project, click the Edit icon next to the
Working with Watson Machine Learning
notebook to edit it. -
In the "Provision services and configure credentials" section, make the following changes:
-
Follow the instructions in the notebook to create, copy, and paste an IBM Cloud API key.
-
Replace the IBM Watson Machine Learning service credentials with the ones that you created previously.
-
Replace the DB credentials with the ones that you created for Databases for PostgreSQL.
-
If you previously configured Watson OpenScale to use a free internal PostgreSQL database as your data mart, you can switch to a new data mart that uses your Databases for PostgreSQL service. To delete your old PostgreSQL configuration and create a new one, set the KEEP_MY_INTERNAL_POSTGRES variable to
False
.The notebook removes your existing internal PostgreSQL data mart and creates a new data mart with the supplied DB credentials. No data migration occurs.
-
-
After you provision your services and entered your credentials, your notebook is ready to run. Click the Kernel menu item, and select Restart & Clear Output from the menu:
-
Now, run each step of the notebook in sequence. Notice what is happening at each step, as described. Complete all the steps, up through and including the steps in the "Additional data to help debugging" section.
The net result is that you create, train, and deploy the Spark German Risk Deployment model to your Watson OpenScale service instance. Watson OpenScale is configured to check the model for bias against sex (in this case, Female) or age (in this case, 18-25 years old).
View results
View insights for your deployment
Using the Watson OpenScale dashboard, click the Insights tab:
The Insights Dashboard page provides an overview of metrics for your deployed models. You can easily see alerts for Fairness or Accuracy metrics that exceed the threshold set through the notebook. The data and settings that are used in this tutorial create Accuracy and Fairness metrics similar to the ones shown here.
View monitoring data for your deployment
- From the Insights Dashboard page, click the tile that corresponds to your deployment to view monitoring data.
The model deployment page displays data for the fairness, quality, and drift monitors. - Click one of the monitor tiles to view evaluation metrics for the monitored attributes.
A chart displays aggregated evaluations as data points across the timeline that you specify. - For any monitored attribute, slide the marker across the chart to select data for a specific timestamp.
-
Click a data point on the chart to view details about the monitoring data.
Now, you can review the data that you monitored. For the following example, you can see that, for the Sex attribute, the monitored group received favorable outcomes less often than the reference group.
View explainability for a model transaction
For each deployment, you can see explainability data for specific transactions. If you already know which transaction you want to view, you can find it with the transaction ID.
-
Click the Explain a transaction tab in the navigator.
-
Select a model in the Deployed model list.
The Recent transactions list displays all of the transactions that are processed by your model. The list contains columns that provide details about the outcome of each transaction. -
Enter a transaction ID.
If you use the internal Lite version of PostgreSQL, you might not be able to retrieve your database credentials. This might prevent you from seeing transactions. -
Click Explain in the Actions column.
The Transaction details page provides an analysis of the factors that influenced the outcome of the transaction. -
Optional: For further analysis, click the Inspect tab.
You can set new values to determine a different predicted outcome for the transaction. -
Optional: After you set new values, click Run analysis to show how different values can change the outcome of the transaction.
Next steps
Getting insights with Watson OpenScale
Parent topic: Watson OpenScale