0 / 0
Configure Watson OpenScale for model risk management
Configure Watson OpenScale for model risk management

Configure Watson OpenScale for model risk management

IBM offers a model risk management solution with IBM Cloud Pak for Data. Watson OpenScale monitors and measures outcomes from AI Models across its lifecycle and performs model validations. Management of model risk is critical to meet regulatory requirements and to protect institutions from operational and reputational risk.

Setup options

You can use one of the following options to set up your initial environment:

  • Automated setup

    Takes just a few minutes to create a workable system with sample data on a locally hosted system

  • Notebook setup

    Allows control over some of the setup parameters and can be used later for managing your own models

  • Manual setup

    Gives you the most control over resources and options

The auto setup option can be run when you launch the service for the first time.

Prerequisites

To perform model evaluation, you must already have an IBM Cloud instance and you must provision an IBM Watson OpenScale instance.

Steps

  1. Launch Watson OpenScale.

    1. From the IBM Cloud Dashboard, click Services.
    2. Click Watson OpenScale.
    3. Click Launch Application.
  2. When prompted about running automatic setup, click Auto setup.

Many of the functions of the auto setup option, can be replicated by running a Python notebook in Watson Studio. Although the results are the same, by choosing to run a notebook, you can gain experience that can more readily be applied to your own data, models, and pre-prod or prod systems.

Prerequisites

Before using the model risk management (MRM) features, set up the following services on IBM Cloud:

  • Watson OpenScale, which provides MRM features and metrics
  • IBM Watson Machine Learning (two separate instances, one for pre-prod and one for prod), which provides the engine for creating predictive models. This tutorial shows how to use IBM Watson Machine Learning as model-serving engine, but you can also use any other supported ML engine)
  • Watson Studio, which runs notebooks and secures assets. (This tutorial shows how to use Watson Studio to create the provided sample models, but you can also use any other IDE to build models)
  • [Optional] Cloud Object Storage provides a place to store model assets. This tutorial uses an internal database. However, you might want to set up Cloud Object Storage for your own work.
  • You must also have the OpenScale model risk management on IBM Cloud.ipynb notebook file, which you can download from the https://github.com/IBM/watson-openscale-samples GitHub repository.

Steps

Step 1: Create an IBMid and IBM Cloud account

If you do not have an IBM Cloud account yet, begin by creating one.

  1. Point your web browser to the following URL: https://cloud.ibm.com/registration
  2. Follow instructions to create an IBMid and IBM Cloud Account.

Step 2: Add services to your IBM Cloud account

As soon as you have an IBM Cloud account, you can use the dashboard to add the required services. For each service, you can choose the Lite or Free plan. You must have instances for the following services: Watson OpenScale, Watson Studio, and IBM Watson Machine Learning (two Instances).

  1. From the Navigation menu, click Resource list.
  2. Click Create resource.
  3. Search for each of the required services by entering keywords, such as openscale, studio, or machine learning.

If you cannot add two Lite plan instances of IBM Watson Machine Learning, create the second instance by using the IBM Watson Machine Learning standard plan or a IBM Cloud Account that is linked to a separate email address. (Link to register the second account: https://cloud.ibm.com/registration).

Step 3: Add a Cloud Object Storage instance

Use Cloud Object Storage to store training dat1. After you create an instance Cloud Object Storage, Watson Studio, IBM Watson Machine Learning, and Watson OpenScale will be able to access the buckets that are created as part of the model creation process.

  1. Use your primary IBMid to log in to your IBM Cloud account.
  2. From the IBM Cloud Dashboard, click Add resource, then click Storage.
  3. Click the Object Storage tile, select the Lite plan, then click Create.

Work in Watson Studio

In Watson Studio, create a project and runs a notebook to perform most setup tasks and the following steps:

  • create two machine models
  • connect Watson OpenScale to IBM OpenPages
  • create model deployments and configure monitors in Watson OpenScale

Step 1: Create the pre-prod project in Watson Studio

When you start Watson Studio (hint: use the IBM Cloud dashboard, find your instance of Watson Studio, and click Get Started) you have the option of taking a tour. Your first task is to create a project to which you associate the IBM Watson Machine Learning instance that you created for your pre-production work.

  1. Click the Create a project tile.
  2. Click the Create an empty project tile.
  3. Give the project a name and description: In the Name field, type MRM – Pre-prod. This project is used for all pre-production models.
  4. Create Cloud Object Storage that on IBM Cloud if you have not already.
  5. Click the Create.

Step 2: Associate your new project with the IBM Watson Machine Learning instance

Now you need to associate your pre-prod instance of Watson Machine Learning to your project. Do this by adding it as an associated service.

  1. From the MRM – Pre-prod project screen, click the Settings tab.
  2. In the Associated services panel, click Add service, and then click Watson.
  3. Find the IBM Watson Machine Learning option and click Add.
  4. From the machine learning configuration window, click the Existing tab.
  5. From the Existing Service Instance drop-down box, select the Machine Learning-Pre-Prod instance and click Select.

Step 3: Add the sample notebook to the project

As part of this tutorial, you are given access to a Watson Studio notebook. Use it to create and deploy pre-prod models, configure the model deployments in Watson OpenScale, and set up a connection between Watson OpenScale and IBM OpenPages,.

  1. From the project page, click New asset.
  2. Click the Notebook tile.
  3. Click the From file tab, click Choose file and then, select the OpenScale model risk management on IBM Cloud.ipynb notebook file that you can download from the https://github.com/IBM/watson-openscale-samples GitHub repository.
  4. Type a name and description and click the Create notebook.

Step 4: Run the sample notebook

The newly created notebook is opened in Watson Studio in the integrated notebook editor. You need to update some of the credentials and then run the notebook to create your pre-prod model.

  1. In the corresponding code box, paste your IBM Cloud API key:

    1. On the IBM Cloud toolbar, click your Account name.
    2. From the Manage menu, click Access (IAM).
    3. In the navigation bar, click IBM Cloud API keys.
    4. Click Create an IBM Cloud API key.
    5. Type a name and description and then click Save.
    6. Copy the newly created API key and paste it into your notebook in the CLOUD_API_KEY code box, which is the first code box.
  2. In the corresponding code boxes, paste your credentials from the pre-prod and prod instances of IBM Watson Machine Learning:

    1. Go to the IBM Cloud dashboard.
    2. In the Resource summary section, click Services.
    3. Click Machine Learning-Pre-Prod.
    4. In the navigation panel, click Service credentials.
    5. Click New credential.
    6. Copy your credentials by clicking the copy icon.
    7. Return to the notebook editor and update the credentials by replacing the sample credentials with your own in the second code box.
    8. Repeat the preceding steps for the prod instance in the third code box.
  3. To restart the notebook and clear the output, from the Kernel menu, click Restart & Clear Output.

  4. Run the notebook each cell at a time by using the Run option. Ensure that each cell completes before running the next cell. Be sure to read directions for steps that must be taken during the intervening cells. For example, at one point, you are directed to move your model into production before running the notebook.

You completed the tutorial and used a notebook to create a pre-prod model! Check inside Watson Studio to see the model that is listed as one of the assets. You have also deployed this model, which means that you can go to IBM Watson OpenScale to add the model there.

Work in IBM Watson OpenScale

Use IBM Watson OpenScale to validate and monitor your models and to process metrics. First, you need to do some setup.

Step 1: Activate model risk management features

As part of this tutorial, you can automatically set up the model risk management features on Watson OpenScale. The following sections detail how to activate the features on IBM Cloud:

To work with IBM Watson OpenScale, you must already have an IBM Cloud instance and you must provision an IBM Watson OpenScale instance.

  1. Launch Watson OpenScale.

    1. From the IBM Cloud Dashboard, click Services.
    2. Click Watson OpenScale
    3. Click Launch Application.
  2. When prompted about running automatic setup, select No thanks.

Manual setup

You can manually set up your entire Watson OpenScale model risk management service by completing the following steps. To successfully complete the steps, you must have detailed information about your machine learning provider, the database, and the data that is used for monitoring.

Steps

  1. Provision prerequisite IBM Cloud services. You must set up two instances of IBM Watson Machine Learning.
  2. Set up Watson Studio projects. You must set up projects for both pre-production and production models.
  3. Configure Watson OpenScale.
  4. Next steps: Continue setting up the monitors and data logging.

Next steps

Parent topic: Model risk management and model governance

Generative AI search and answer
These answers are generated by a large language model in watsonx.ai based on content from the product documentation. Learn more