0 / 0
Configuring Watson OpenScale with advanced setup
Configuring Watson OpenScale with advanced setup

Configuring Watson OpenScale with advanced setup

You can use the advanced setup option to run a Python notebook that uses sample data to demonstrate how to configure model evaluations for Watson OpenScale.

Before you begin

This tutorial uses a Jupyter Notebook that must be run in a Watson Studio project. The notebook uses a "IBM Runtime 22.1 on Python 3.9 XS" runtime environment.

Ensure that you have provisioned Watson OpenScale service to your account. It also requires service credentials for the following IBM Cloud services:

  • Cloud Object Storage (to store your Watson Studio project)
  • IBM Watson Machine Learning
  • Optional: Databases for PostgreSQL or Db2 Warehouse

The Jupyter Notebook does the following tasks:

  • Trains, creates, and deploys a German Credit Risk model
  • Configures Watson OpenScale to monitor that deployment
  • Provides seven days' worth of historical records and measurements for viewing in the Watson OpenScale Insights dashboard.

You can also configure the model for continuous learning with Watson Studio and Spark.

Introduction

In this tutorial, you perform the following tasks:

Provision IBM Cloud Services

Log in to your IBM Cloud account with your IBMid. When provisioning services, particularly if you use Db2 Warehouse, verify that your selected organization and space are the same for all services.

Create a Watson Studio account

  • Create a Watson Studio instance if you do not already have one associated with your account:

    Watson Studio tile is displayed

  • Give your service a name, choose the Lite (free) plan, and click Create.

Provision an IBM Cloud Object Storage service

Provision an IBM Watson Machine Learning service

Provision an IBM Watson OpenScale service

If you haven't already, ensure that you provision IBM Watson OpenScale.

  • Provision a Watson OpenScale instance if you do not already have one associated with your account:

    Watson OpenScale tile is displayed

  • Click Catalog > AI > Watson OpenScale.

  • Give your service a name, choose a plan, and click Create.
  • To start Watson OpenScale, click Get Started.

Optional: Provision a Databases for PostgreSQL or Db2 Warehouse service

If you have a paid IBM Cloud account, you can provision a Databases for PostgreSQL or Db2 Warehouse service to take full advantage of integration with Watson Studio and continuous learning services. If you choose not to provision a paid service, you can use the free internal PostgreSQL storage with Watson OpenScale, but you are not able to configure continuous learning for your model.

Set up a Watson Studio project

  1. Log in to your Watson Studio account. Click the Avatar and verify that the account you are using is the same account you used to create your IBM Cloud services:

    Same Account

  2. In Watson Studio, begin by creating a new project. Click the Create a project tile.

    Watson Studio create project

  3. Click the Create an empty project tile.

    Watson Studio Create an empty project tile is displayed

  4. Give your project a name and description, make sure that the IBM Cloud Object Storage service that you created is selected in Storage, and click Create.

Create and deploy a Machine Learning model

Add the Working with Watson Machine Learning notebook to your Watson Studio project

  • Access the following file. If you have a GitHub account, you can sign in to clone the repository and download the file. Otherwise, you can view the raw version, by clicking Raw and copying the text of the file into a new file with the extension .ipynb.

  • From the Assets tab in your Watson Studio project, click New asset and select Notebook:

    The choose asset type with a Notebook tile highlighted is displayed

  • Select From file:

    New Notebook Form

  • Then, upload the file:

    New Notebook Form

  • In the Select runtime section, choose the latest Python with Spark option.

    Watson Studio does not offer a free environment with Python and Apache Spark. The only free runtime environment is for a Python-only environment.

  • Click Create.

Edit and run the Working with Watson Machine Learning notebook

The Working with Watson Machine Learning notebook contains detailed instructions for each step in the Python code that you run. As you work through the notebook, take some time to understand what each command is doing.

  • From the Assets tab in your Watson Studio project, click the Edit icon next to the Working with Watson Machine Learning notebook to edit it.

  • In the "Provision services and configure credentials" section, make the following changes:

    • Follow the instructions in the notebook to create, copy, and paste an IBM Cloud API key.

    • Replace the IBM Watson Machine Learning service credentials with the ones that you created previously.

    • Replace the DB credentials with the ones that you created for Databases for PostgreSQL.

    • If you previously configured Watson OpenScale to use a free internal PostgreSQL database as your data mart, you can switch to a new data mart that uses your Databases for PostgreSQL service. To delete your old PostgreSQL configuration and create a new one, set the KEEP_MY_INTERNAL_POSTGRES variable to False.

      The notebook removes your existing internal PostgreSQL data mart and creates a new data mart with the supplied DB credentials. No data migration occurs.

  • After you provision your services and entered your credentials, your notebook is ready to run. Click the Kernel menu item, and select Restart & Clear Output from the menu:

    Restart and Clear

  • Now, run each step of the notebook in sequence. Notice what is happening at each step, as described. Complete all the steps, up through and including the steps in the "Additional data to help debugging" section.

The net result is that you create, train, and deploy the Spark German Risk Deployment model to your Watson OpenScale service instance. Watson OpenScale is configured to check the model for bias against sex (in this case, Female) or age (in this case, 18-25 years old).

View results

View insights for your deployment

Using the Watson OpenScale dashboard, click the Insights Insights icon is displayed tab:

The Insights Dashboard page provides an overview of metrics for your deployed models. You can easily see alerts for Fairness or Accuracy metrics that exceed the threshold set through the notebook. The data and settings that are used in this tutorial create Accuracy and Fairness metrics similar to the ones shown here.

Insight overview dashboard displays with a tile for the German Credit Risk model

View monitoring data for your deployment

  1. From the Insights Dashboard page, click the tile that corresponds to your deployment to view monitoring data.
    The model deployment page displays data for the fairness, quality, and drift monitors.
  2. Click one of the monitor tiles to view evaluation metrics for the monitored attributes.
    A chart displays aggregated evaluations as data points across the timeline that you specify.
  3. For any monitored attribute, slide the marker across the chart to select data for a specific timestamp.
  4. Click a data point on the chart to view details about the monitoring data.

    Monitor data

Now, you can review the data that you monitored. For the following example, you can see that, for the Sex attribute, the monitored group received favorable outcomes less often than the reference group.

Insight overview

View explainability for a model transaction

For each deployment, you can see explainability data for specific transactions. If you already know which transaction you want to view, you can find it with the transaction ID.

  1. Click the Explain a transaction Explain a transaction tab tab in the navigator.

  2. Select a model in the Deployed model list.
    The Recent transactions list displays all of the transactions that are processed by your model. The list contains columns that provide details about the outcome of each transaction.

    Transaction list

  3. Enter a transaction ID.
    If you use the internal Lite version of PostgreSQL, you might not be able to retrieve your database credentials. This might prevent you from seeing transactions.

  4. Click Explain in the Actions column.
    The Transaction details page provides an analysis of the factors that influenced the outcome of the transaction.

    Transaction details

  5. Optional: For further analysis, click the Inspect tab.
    You can set new values to determine a different predicted outcome for the transaction.

  6. Optional: After you set new values, click Run analysis to show how different values can change the outcome of the transaction.

    Transaction details on the inspect tab show values that might produce a different outcome

Next steps

Getting insights with Watson OpenScale

Parent topic: Watson OpenScale

Generative AI search and answer
These answers are generated by a large language model in watsonx.ai based on content from the product documentation. Learn more