Getting started tutorial

IBM Watson OpenScale tracks and measures outcomes from your AI models, and helps ensure they remain fair, explainable, and compliant wherever your models were built or are running. Watson OpenScale also detects and helps correct the drift in accuracy when an AI model is in production

Enterprises use IBM Watson OpenScale to automate and put into service AI lifecycle in business applications. This approach ensures that AI models are free from bias, can be easily explained and understood by business users, and are auditable in business transactions. Watson OpenScale supports AI models built and run in the tools and model serve frameworks of your choice.

 

Getting started with Watson OpenScale (automated setup)

Traditional lenders are under pressure to expand their digital portfolio of financial services to a larger and more diverse audience, which requires a new approach to credit risk modeling. Their data science teams currently rely on standard modeling techniques - like decision trees and logistic regression - which work well for moderate data sets, and make recommendations that can be easily explained. This approach satisfies regulatory requirements that credit lending decisions must be transparent and explainable.

To provide credit access to a wider and riskier population, applicant credit histories must expand beyond traditional lines of credit. In addition to mortgages and car loans, lenders need alternative credit sources, such as utility and mobile phone plan payment histories, plus education and job titles. These new data sources offer promise, but also introduce risk by increasing the likelihood of unexpected correlations that introduce bias based on an applicant’s age, gender, or other personal traits.

The data science techniques that are most suited to these diverse data sets, such as gradient boosted trees and neural networks, can generate highly accurate risk models, but at a cost. Such models, without explanation of the inner workings, generate opaque predictions that must somehow become apparent. You must ensure regulatory approval. Article 22 of the General Data Protection Regulation (GDPR) requires explainability. The United States Fair Credit Reporting Act (FCRA) that is managed by the Consumer Financial Protection Bureau also requires this level of accountability.

The credit risk model that is provided in this tutorial uses a training data set that contains 20 attributes about each loan applicant. Two of those attributes - age and sex - can be tested for bias. For this tutorial, the focus is on bias against sex and age. For more information, see Why does Watson OpenScale need access to my training data?

Watson OpenScale monitors the deployed model’s propensity for a favorable outcome (“No Risk”) for one group (the Reference Group) over another (the Monitored Group). In this tutorial, the Monitored Group for sex is female, while the Monitored Group for age is 19 to 25.

Setup options

Use one of the following setup options, depending on your preference and level of expertise:

  • The following automated setup guides you through the process by performing tasks for you in the background.

    Use of a tour means that you can watch and click through to the next part of the tour.

  • The interactive setup that provides you with an easy-to-follow script.

    Use the interface to complete common tasks with a sample model and injected data.

  • The advanced tutorial enables more technical users to install a Python module that automates the provisioning and configuration of prerequisite services. This advanced tutorial is for data scientists or users who are comfortable with coding, Python and Notebooks. It’s an example of how the Watson OpenScale client can be used to perform functionality programatically. The notebook that is used in this tutorial results in the same place as following the automated setup.

    This module requires that Python 3 is installed, which includes the pip package management system. For instructions, see, Installing a Python module to set up Watson OpenScale.

For other tutorials, see Additional resources.

Automated setup

To quickly see how Watson OpenScale monitors a model, run the demo scenario option that is provided when you first log into the Watson OpenScale UI. See Working with the UI demo.

Before you begin

Before you begin the tour, you must have the following resources:

The automated setup tour is designed to work with the least possible user interaction. It automatically makes the following decisions for you:

  • If you have multiple IBM Watson Machine Learning instances set up, the install process runs an API call to list the instances and chooses whichever Machine Learning instance appears first in the resulting list.
  • To create a new lite version IBM Watson Machine Learning, Watson OpenScale installer uses the default resource group for your IBM Cloud account.

Provision a Watson OpenScale service

If you haven’t already, ensure that you provision IBM Watson OpenScale. From the catalog, you can look in the AI category for Watson OpenScale. After you give your service a name, choose a plan, and click Create, you are able to start the Watson OpenScale service.

Auto setup

You can automatically set up your Watson OpenScale instance by using a sample model and sample data. Sign in to Watson OpenScale and choose the Auto setup option. As the Watson OpenScale services are being provisioned, you can review the demo scenario. When provisioning is complete, you can tour the Watson OpenScale dashboard, and proceed with Viewing results in the Watson OpenScale model monitor.

View insights in the Watson OpenScale model monitor

To view insights into the fairness and accuracy of the model, details of data that is monitored, and explainability for an individual transaction, open the Watson OpenScale dashboard. Each deployment is shown as a tile. The tour configures a deployment that is called GermanCreditRiskModel.

View insights

At a glance, the Insights page shows any issues with fairness and accuracy, as determined by the thresholds that are configured.

View monitoring data

From the Insights dashboard, click the GermanCreditRiskModelICP tile to view details about the monitored data. View summary information or click the tiles to see more details. For monitors that include a chart, click and drag the marker across the chart to view a day and time period that shows data and then click the View details link. You can click different time periods in the chart to change the data that you see.

For information about interpreting the time series chart, see Getting insights.

View explainability

To understand the factors that contribute when bias is present for a given time period, from the visualization screen, click the Biased transactions radio button. Transaction IDs for the past hour are listed for those transactions that have bias. For the model used in this module, bias exists for requests that are available.

For information, see Monitoring explainability.

Finishing the tour

After you finish the tour and the application setup, you can try one of the following tasks:

  • To add your own model to the dashboard, from the Model monitors tab, click the Add to dashboard button.
  • To continue exploring the tutorial model, from the Model monitors tab, click the German Credit Risk tile.

Next steps