By using IBM OpenPages with Watson OpenScale, you can create an end-to-end model governance solution. Start by creating a new model in IBM OpenPages MRG. Take the model through the candidate and development workflows in Watson OpenScale. Link the model to a sample model in Watson OpenScale. Explore the Watson OpenScale features, export Watson OpenScale metrics for the model to OpenPages as part of the pre-implementation validation process. Explore the ways to view and interpret these metrics.

Work in IBM OpenPages

Perform the following steps in IBM OpenPages MRG.

Step 1. Set up a New Model in OpenPages

1. Go to menu in the upper left of the screen and select Models under Inventory
2. Click Add New in the upper right of the screen

3. Complete the following fields:

   1. Description
   2. Model Status to “Proposed”
   3. Model Owner to your account name
   4. Model Non-Model to “Model”
   5. Machine Learning Model to “Yes”
   6. Monitored with OpenScale to “Yes”
   7. These fields appear after the Machine Learning Model field is set to “Yes”
   8. For Parent Entity, select the Business Entity with your organization’s name.
4. Click Save

Watson OpenScale monitors deployed models. The non-model designation is for regulatory compliance where you want to experiment with data analysis without deployment and reporting requirements.

Step 2. Move the Model through the Candidate Workflow

For the tutorial, the typical workflow rules are relaxed to enable the same user to move the model through the workflows. Typically, different stages of the workflow would require users who are assigned to certain roles such as owner, developer, and head of model review.

1. In the model created in Step 1, enter a Candidate Comment and click Save. This field lets the model owner describe why the proposed model is a model and not a non-model.
2. Select the Action “Submit Candidate for Confirmation.” In a live workflow, this step sends the candidate model to a reviewer for approval.
3. Select the Action “Confirm Assessment”. In a live workflow, a reviewer confirms that the model candidate is a model and the candidate workflow is complete.

Step 3. Move the Model through the Model Development Workflow to the Pre-Implementation Review Stage

1. From the Action menu, click Start Model Development.
2. Complete the fields in the Model object that are related to the development – they are listed on the right side pane – and save.
3. For purposes of this testing, you can enter your user account as the developer.
4. Select Action “Assign to Developer”
5. Select Action “Submit for Pre Implementation Review”

Work in Watson Studio

In IBM Watson Studio, you create a project and run a Notebook to set up the service, including the following steps:

• Create two machine models. • Connect Watson OpenScale to IBM OpenPages. • Create model deployments and configure monitors in Watson OpenScale.

Step 1. Create the pre-prod project in Watson Studio

After you start Watson Studio (hint: use the IBM Cloud dashboard, find your instance of Watson Studio, and click Get Started) you have the option of taking a tour. Your first task is to create a project to which you associate the IBM Watson Machine Learning instance that you created for your pre-production work.

1. Click the Create a project tile.
2. Click the Create an empty project tile.
3. Give the project a name and description: In the Name field, type MRM – Pre-prod. You use this project for all your pre-production models.
4. You notice that an instance of Cloud Object Storage is indicated. Go ahead and create an instance on IBM Cloud if you haven’t already.
5. Click Create.

Step 2. Associate your new project with the Watson Machine Learning instance

Now you need to associate your pre-prod instance of IBM Watson Machine Learning to your project by adding it as an associated service.

1. From the MRM – Pre-prod project screen, click the Settings tab.
2. In the Associated services pane, click Add service, and then click Watson.
3. Find the IBM Watson Machine Learning option and click Add.
4. From the Machine Learning configuration window, click the Existing tab.
5. From the Existing Service Instance list, select the Machine Learning-Pre-Prod instance and click Select.

Step 3. Add the sample Notebook to the project

As part of this tutorial, you have access to a Watson Studio Notebook. You use it to set up your connection between Watson OpenScale and IBM OpenPages to create and deploy pre-prod models, and configure the model deployments in Watson OpenScale.

1. From the project page, click the Add to project button.
2. Click the Notebook tile.
3. Click the From file tab, click Choose file and then, select the OpenScale and OpenPages model risk management on IBM Cloud.ipynb Notebook file that you can download from the https://github.com/IBM/watson-openscale-samples GitHub repository.
4. Add a name and description and click Create Notebook.

Step 4. Run the sample Notebook

The newly created Notebook is opened in Watson Studio in the integrated Notebook editor. You need to update some of the credentials and then run the Notebook to create your pre-prod model.

1. In the corresponding code box, paste your IBM Cloud API key:

   1. From the IBM Cloud toolbar, click your Account name, such as <Your user name>’s Account.
   2. From the Manage menu, click Access (IAM).
   3. In the navigation bar, click IBM Cloud API keys.
   4. Click the Create an IBM Cloud API key button.
   5. Type a name and description and then click Save.
   6. Copy the newly created API key and paste it into your Notebook in the CLOUD_API_KEY code box, which is the first code box.

2. In the corresponding code boxes, paste your credentials from the pre-prod and prod instances of IBM Watson Machine Learning:

   1. Go to the IBM Cloud dashboard.
   2. In the Resource summary section, click Services.
   3. Click Machine Learning-Pre-Prod.
   4. In the navigation pane, click Service credentials.
   5. Click New credential.
   6. Copy your credentials by clicking the copy icon.
   7. Return to the Notebook editor and update the credentials by replacing the sample credentials with your own in the second code box. h. Repeat the preceding steps for the prod instance in the third code box.
3. In the corresponding code box, paste your IBM OpenPages URL, username, and password.
4. In the corresponding code box, paste the model_id number from the model that you created in IBM OpenPages.
5. To restart the Notebook and clear the output, from the Kernel menu, click Restart & Clear Output.
6. Run the Notebook one cell at a time by using the Run option. Ensure that a cell completes before you run the next cell. Be sure to read directions for steps that must be taken during the intervening cells. For example, at one point, you are directed to move your model into production before you continue to run the Notebook. You completed the tutorial! You used a Notebook to create a pre-prod model. You can check inside Watson Studio, where you will now see the model that is listed as one of the assets. You also deployed this model, which means that you can go to IBM Watson OpenScale to add the model there.

Work in Watson OpenScale

You use IBM Watson OpenScale to validate and monitor your models and to process metrics. First, you need to do some set-up.

Step 1. Activate model risk management features

As part of this tutorial, you can use the model risk management features on Watson OpenScale. To work with IBM Watson OpenScale, you must already have an IBM Cloud instance and you must provision an IBM Watson OpenScale instance.

1. Launch Watson OpenScale.

   1. From the IBM Cloud Dashboard, click Services.
   2. Click Watson OpenScale
   3. Click the Launch Application button.

2. When prompted about running automatic setup, click No thanks.

Ensure that whenever you work inside IBM OpenPages that your profile indicates that you are integrating your work with OpenScale. Also, be sure to select the option Monitored with Watson OpenScale when you create your model record.

Step 2. Connect Watson OpenScale to IBM OpenPages

So that the two services can be fully integrated, you must add your IBM OpenPages URL and authentication credentials to Watson OpenScale.

1. From the Watson OpenScale side bar, click the Configure icon.
2. Click Integrations and then in the IBM OpenPages tile, click the Edit icon.
3. Select the OpenPages location, which is a specific instance of IBM OpenPages either on cloud or on premises.
4. Type the URL, username, and password for the IBM OpenPages instance that you use to manage your models and click Save.

.

Step 3. Perform analysis in Watson OpenScale

After you run the setup Notebook, you can both see and compare the sample evaluations in Watson OpenScale by using a downloadable report. The Model Summary Report includes all the quality measures, fairness measures, and drift magnitude.

1. From the Insights dashboard, click the model deployment tile.

2. From the Actions list, click one of the following analysis options:

   1. Past evaluations: Lists all the previous versions of the evaluation.
   2. Compare: Compare any of the models, but especially versions of the same model, for best performance.
   3. Download report PDF: Generates the model summary report, which gives you all of the metrics and the explanation for why they were scored the way they were.
   4. Send to OpenPages now: Sends all the metrics to the OpenPages model record that you associated to the OpenScale model deployment. In OpenPages, the metrics are listed in the graphical flow diagram and are clickable so that you can drill down into each metric.

Work in OpenPages

Perform the following steps in IBM OpenPages MRG.

Review results in IBM OpenPages

After you send metrics to IBM OpenPages, you are able to review the metrics in OpenPages as part of the Pre-Implementation Review. HINT: You can use the Notebook to go directly to the model in OpenPages. Go to the Create an OpenPages Model section (code cell 12) and find the Model Name = <model name> and OpenPages Model ID = <model ID> output. Use them to search for your model in OpenPages.

1. Find the model in IBM OpenPages by using the model name, such as MOD_0000206.
2. Review the metrics from Watson OpenScale by expanding the metrics twistie:
3. In addition to a list of metrics, you can view metrics by type in a graphical format by expanding the metrics summary twistie:
4. The Associations view provides relationships in the form of a tree:
5. The Supporting files and artifacts pane gives you access to all the Watson OpenScale model risk management reports that are run:

Change the model status in IBM OpenPages

As part of the overall model risk governance workflow, the models you create are typically worked on by several different personas or job roles. For example, there might be a data scientist who is the model owner, a model validator, and model reviewer. For the purposes of the tutorial, you act in all of these roles to see how the model can progress from one status to the next.

You must open the model and use the Actions list to set the model status. Use the ID, such as OpenPagesAdministrator that was given to you to be the model owner, validator, and final approver. For the next part of the tutorial, you’ll want the status to be Approved for Deployment.

1. In IBM OpenPages, locate the model that you want to promote.
2. From the Actions drop-down box, click Approved. If for some reason, you don’t see the Approved option, you might need to move the model through other steps, such as Submit for Pre Implementation Review or Submit for Approval.

Deploy a new model to production in Watson OpenScale

Push the best model to production. Create a production record by importing from a pre-production model. After the model is approved for deployment in IBM OpenPages, you can send the model to production in Watson OpenScale.

1. Review the status of the model deployment:
2. Return to the sample Notebook and run the cells to send the model to production.
3. You can now view the production model deployment tile. In a regular production environment, it initially appears empty until enough data is gathered and time passes for metric calculation to be triggered. For the tutorial, the Notebook adds data and runs the monitors so that you can see the results right away.

Next steps

Use the analysis of fairness to redefine the model, possibly by using a different algorithm. Use Watson OpenScale to compare models by looking at the key metrics in a side-by-side comparison. Use this feature to determine which version of a model is the best one to send to production or which model might need work:

Because of the connection to Watson OpenScale, you can get alerts in IBM OpenPages for items that require attention or missing pieces of information:

You can set up a regular schedule for sending metrics to IBM OpenPages.