0 / 0
Setting up model governance with IBM OpenPages

Setting up model governance with IBM OpenPages

By using IBM OpenPages with Watson with Watson OpenScale, you can create an end-to-end model governance solution. Start by creating a new model in IBM OpenPages Model Risk Governance. Take the model through the candidate and development workflows for model evaluation. Link the model to a sample model. Explore features and export model evaluation metrics for the model to IBM OpenPages as part of the pre-implementation validation process. Explore the ways to view and interpret these metrics.

Work in IBM OpenPages

When working inside IBM OpenPages, verify your access to the Model Risk Governance (MRG) objects. Also, when creating your model record, be sure to select the option Monitored with Watson OpenScale.

After you configure IBM OpenPages, if you change the instance or server, all the associated model deployments remain unchanged. To update the association, go to IBM OpenPages and click Model governance on the configuration screen. Then, select a model or unlink the obsolete association.

Perform the following steps in IBM OpenPages.

Step 1: Set up a New Model in IBM OpenPages

  1. Go to menu in the upper left of the screen and select Models under Inventory

  2. Click Add New

  3. Complete the following fields. Certain fields appear after Machine Learning Model is set to Yes.

    • Description
    • Model Status, set to Proposed
    • Model Owner, set to your account name
    • Model Non Model, set to Model
    • Machine Learning Model, set to Yes
    • Monitored with OpenScale, set to Yes
    • Parent Entity, where the Business Entity is set to your organization’s name
  4. Click Save.

Use model evaluation to monitor deployed models. The nonmodel designation is for regulatory compliance where you want to experiment with data analysis without deployment and reporting requirements.

Step 2: Move the Model through the Candidate Workflow

Different stages of the workflow require certain roles such as owner, developer, and head of model review.

  1. In the model created in Step 1, enter a Candidate Comment and click Save. This field allows the model owner describe why the proposed model is a model and not a nonmodel.
  2. From the Action drop-down menu, click Submit Candidate for Confirmation. In a live workflow, this step sends the candidate model to a reviewer for approval.
  3. From the Action drop-down menu, click Confirm Assessment. In a live workflow, a reviewer confirms that the model candidate is a model and the candidate workflow is complete.

Step 3: Move the Model through the Model Development Workflow to the Pre-Implementation Review Stage

  1. From the Action drop-down menu, click Start Model Development.
  2. Complete the required fields in the Model object that are related to the development – they are listed on the right side panel – and save.
  3. For purposes of this testing, you can enter your user account as the developer.
  4. From the Action drop-down menu, click Assign to Developer.
  5. From the Action drop-down menu, click Submit for Pre Implementation Review.

Work in Watson Studio

In IBM Watson Studio, you create a project and run a Notebook to set up the service, including the following steps:

• Create two machine models. • Connect Watson OpenScale to IBM OpenPages. • Create model deployments and configure monitors in Watson OpenScale.

Step 1. Create the pre-prod project in Watson Studio

After you launch Watson Studio (hint: use the IBM Cloud dashboard, find your instance of Watson Studio, and click Get Started) you have the option of taking a tour. Your first task is to create a project to which you associate the IBM Watson Machine Learning instance that you created for your pre-production work.

  1. Click Create a project.
  2. Click Create an empty project.
  3. Give the project a name and description: In the Name field, type MRM – Pre-prod. You use this project for all your pre-production models.
  4. You notice that an instance of Cloud Object Storage is indicated. Go ahead and create an instance on IBM Cloud if you haven’t already.
  5. Click Create.

Step 2. Associate your new project with the Watson Machine Learning instance

Now you need to associate your pre-prod instance of IBM Watson Machine Learning to your project by adding it as an associated service.

  1. From the MRM – Pre-prod project screen, click the Settings tab.
  2. In the Associated services panel, click Add service, and then click Watson.
  3. Find the IBM Watson Machine Learning option and click Add.
  4. From the machine learning configuration window, click the Existing tab.
  5. From the Existing Service Instance list, select the Machine Learning-Pre-Prod instance.

Step 3. Add the sample Notebook to the project

As part of this tutorial, you have access to a Watson Studio Notebook. You use it to set up your connection between Watson OpenScale and IBM OpenPages to create and deploy pre-prod models, and configure the model deployments in Watson OpenScale.

  1. From the project page, click the New asset.
  2. Click the Notebook.
  3. Click the From file tab, click Choose file and then, select the OpenScale and OpenPages model risk management on IBM Cloud.ipynb Notebook file that you can download from the https://github.com/IBM/watson-openscale-samples GitHub repository.
  4. Type a name and description and click Create Notebook.

Step 4. Run the sample Notebook

The newly created Notebook is opened in Watson Studio in the integrated Notebook editor. You need to update some of the credentials and then run the Notebook to create your pre-prod model.

  1. In the corresponding code box, paste your IBM Cloud API key:

    1. From the IBM Cloud toolbar, click your Account name, such as <Your user name>’s Account.
    2. From the Manage menu, click Access (IAM).
    3. In the navigation bar, click IBM Cloud API keys.
    4. Click the Create an IBM Cloud API ke.
    5. Type a name and description and then click Save.
    6. Copy the newly created API key and paste it into your Notebook in the CLOUD_API_KEY code box, which is the first code box.
  2. In the corresponding code boxes, paste your credentials from the pre-prod and prod instances of IBM Watson Machine Learning:

    1. Go to the IBM Cloud dashboard.
    2. In the Resource summary section, click Services.
    3. Click Machine Learning-Pre-Prod.
    4. In the navigation panel, click Service credentials.
    5. Click New credential.
    6. Copy your credentials by clicking the copy icon.
    7. Return to the Notebook editor and update the credentials by replacing the sample credentials with your own in the second code box. h. Repeat the preceding steps for the prod instance in the third code box.
  3. In the corresponding code box, paste your IBM OpenPages URL, username, and password.

  4. In the corresponding code box, paste the model_id number from the model that you created in IBM OpenPages.

  5. To restart the Notebook and clear the output, from the Kernel menu, click Restart & Clear Output.

  6. Run the Notebook one cell at a time by using the Run option. Ensure that a cell completes before you run the next cell. Be sure to read directions for steps that must be taken during the intervening cells. For example, at one point, you are directed to move your model into production before you continue to run the Notebook. You completed the tutorial! You used a Notebook to create a pre-prod model. You can check inside Watson Studio, where the model that is listed as one of the assets. You also deployed this model, which means that you can go to IBM Watson OpenScale to add the model there.

Work in IBM Watson OpenScale

Use IBM Watson OpenScale to validate and monitor your models and to process metrics. First, you need to do some setup.

Step 1: Connect Watson OpenScale to IBM OpenPages

So that the two services can be fully integrated, you must add your IBM OpenPages URL and authentication credentials to Watson OpenScale.

  1. Log in to Cloud Pak for Data and open your IBM OpenPages instance.

  2. Generate the API key.

  3. Copy the URL that is displayed in the Access information section of the window that provides details about your instance.
    the details about the openpages instance are displayed

  4. From the Watson OpenScale side bar, click the Configure configure icon is displayed icon.

  5. Click Integrations and then in the IBM OpenPages tile, click the Edit the edit icon is displayed icon.

  6. Enter the URL, username, and API key for your IBM OpenPages instance and click Save.
    Verify that the IBM OpenPages connection is saved.
    the OpenPages configuration screen with OpenPages location, URL, username, and API key fields

  7. Associate the model and publish the metrics.

For certain upgrade scenarios, where you are keeping an older version of IBM OpenPages, you need to select IBM Cloud in the OpenPages location box. Specifically, for a version 3.5.1 instance of IBM OpenPages you must select the OpenPages location information as IBM Cloud.

Step 2: Perform analysis in Watson OpenScale

After you run the setup notebook, you can both see and compare the sample evaluations in Watson OpenScale. There is a downloadable report, the Model Summary Report, that includes all the quality measures, fairness measures, and drift magnitude.

  1. From the Insights dashboard, click model deployment

  2. From the Actions drop-down box, click one of the following analysis options:

    1. Past evaluations: Lists all the previous versions of the evaluation.
    2. Compare: Compare any of the models, but especially versions of the same model, for best performance.
    3. Download report PDF: Generates the model summary report, which gives you all of the metrics and the explanation for why they were scored the way that they were.
    4. Send to OpenPages now: Sends all the metrics to the IBM OpenPages model record that you associated to the OpenScale model deployment. In IBM OpenPages, the metrics are listed in the graphical flow diagram and clickable so that you can drill down into each metric.

Work in IBM OpenPages

Review results in IBM OpenPages

After you send metrics to IBM OpenPages, you are able to review the metrics in IBM OpenPages as part of the Pre-Implementation Review. HINT: You can use the notebook to go directly to the model in IBM OpenPages. Go to the Create an OpenPages Model section (code cell 12) and find the Model Name = <model name> and OpenPages Model ID = <model ID> output. Use them to search for your model in IBM OpenPages.

  1. Find the model in IBM OpenPages by using the model name, such as MOD_0000206.
  2. Review the metrics from Watson OpenScale by expanding the metrics twisty:
  3. In addition to a list of metrics, you can view metrics by type in a graphical format by expanding the metrics summary twisty:
  4. The Associations view provides relationships in the form of a tree:
  5. The Supporting files and artifacts panel gives you access to all the Watson OpenScale model risk management reports that are run:

Change the model status in IBM OpenPages

As part of the overall model risk governance workflow, the models you create are typically worked on by several different personas or job roles. For example, there might be a data scientist who is the model owner, a model validator, and model reviewer. This tutorial will cover all roles to show how the model progresses from one status to the next. To do this, you must open the model and use the Actions drop-down box to set the model status. Use the ID, such as OpenPagesAdministrator that was given to you to be the model owner, validator, and final approver. For the next part of the tutorial, you’ll want the status to be Approved for Deployment.

  1. In IBM OpenPages, locate the model that you want to promote.
  2. From the Actions drop-down box, click Approved. If for some reason, you don’t see the Approved option, you might need to move the model through other steps, such as Submit for Pre Implementation Review or Submit for Approval.

Deploy a new model to production in Watson OpenScale

Push the best model to production. Create a production record by importing from a pre-production model. After the model is approved for deployment in IBM OpenPages, you can send the model to production in Watson OpenScale.

  1. Review the status of the model deployment:
  2. Return to the sample notebook and run the cells to send the model to production.
  3. You can now view the production model deployment til5. In a regular production environment, it initially appears empty until enough data is gathered and time has passed for metric calculation to be triggered. The notebook adds data and runs the monitors so that you can see the results right away.

Next steps

APIs, SDKs, and tutorials

Parent topic: Model risk management and model governance

Generative AI search and answer
These answers are generated by a large language model in watsonx.ai based on content from the product documentation. Learn more