This notebook should be run in a Watson Studio project, using Default Python 3 runtime environment. It requires service credentials for the following Cloud services:
If you have a paid Cloud account, you may also provision a Databases for PostgreSQL or Db2 Warehouse service to take full advantage of integration with Watson Studio and continuous learning services. If you choose not to provision this paid service, you can use the free internal PostgreSQL storage with OpenScale, but will not be able to configure continuous learning for your model.
The notebook will configure OpenScale to monitor that deployment, and inject seven days' worth of historical records and measurements for viewing in the OpenScale Insights dashboard.
In order to execute this notebook you will need a deployed model. You can perform it with following notebook. Then you will need to copy deployment_uid
to this notebook.
Before you use the sample code in this notebook, you must perform the following setup tasks:
Authenticate the Watson Machine Learning service on IBM Cloud. You need to provide platform api_key
and instance location
.
You can use IBM Cloud CLI to retrieve platform API Key and instance location.
API Key can be generated in the following way:
ibmcloud login
ibmcloud iam api-key-create API_KEY_NAME
In result, get the value of api_key
from the output.
Location of your WML instance can be retrieved in the following way:
ibmcloud login --apikey API_KEY -a https://cloud.ibm.com
ibmcloud resource service-instance WML_INSTANCE_NAME
In result, get the value of location
from the output.
Tip: Your Cloud API key
can be generated by going to the Users section of the Cloud console. From that page, click your name, scroll down to the API Keys section, and click Create an IBM Cloud API key. Give your key a name and click Create, then copy the created key and paste it below. You can also get a service specific url by going to the Endpoint URLs section of the Watson Machine Learning docs. You can check your instance location in your Watson Machine Learning (WML) Service instance details.
You can also get service specific apikey by going to the Service IDs section of the Cloud Console. From that page, click Create, then copy the created key and paste it below.
Action: Enter your api_key
and location
in the following cell.
apikey = 'PASTE YOUR PLATFORM API KEY HERE'
location = 'PASTE YOUR INSTANCE LOCATION HERE'
wml_credentials = {
"apikey": apikey,
"url": 'https://' + location + '.ml.cloud.ibm.com'
}
!pip install -U ibm-watson-machine-learning
from ibm_watson_machine_learning import APIClient
client = APIClient(wml_credentials)
First of all, you need to create a space that will be used for your work. If you do not have space already created, you can use Deployment Spaces Dashboard to create one.
space_id
and paste it belowTip: You can also use SDK to prepare the space for your work. More information can be found here.
Action: Assign space ID below
space_id = 'PASTE YOUR SPACE ID HERE'
You can use list
method to print all existing spaces.
client.spaces.list(limit=10)
To be able to interact with all resources available in Watson Machine Learning, you need to set space which you will be using.
client.set.default_space(space_id)
List deployments.
client.deployments.list()
Put deployment_id
of model deployed in prerequisite notebook in the next cell.
deployment_uid = "PUT_YOUR_DEPLOYMENT_ID_HERE"
Get deployment_url
and model_uid
from deployment details.
deployment_url = client.deployments.get_details(deployment_uid)['entity']['status']['online_url']['url']
model_uid = client.deployments.get_details(deployment_uid)['entity']['asset']['id']
!pip install -U ibm-watson-openscale
!pip install -U ibm-cloud-sdk-core
This tutorial can use Databases for PostgreSQL, Db2 Warehouse, or a free internal verison of PostgreSQL to create a datamart for OpenScale.
If you have previously configured OpenScale, it will use your existing datamart, and not interfere with any models you are currently monitoring. Do not update the cell below.
If you do not have a paid Cloud account or would prefer not to provision this paid service, you may use the free internal PostgreSQL service with OpenScale. Do not update the cell below.
To provision a new instance of Db2 Warehouse, locate Db2 Warehouse in the Cloud catalog, give your service a name, and click Create. Once your instance is created, click the Service Credentials link on the left side of the screen. Click the New credential button, give your credentials a name, and click Add. Your new credentials can be accessed by clicking the View credentials button. Copy and paste your Db2 Warehouse credentials into the cell below.
To provision a new instance of Databases for PostgreSQL, locate Databases for PostgreSQL in the Cloud catalog, give your service a name, and click Create. Once your instance is created, click the Service Credentials link on the left side of the screen. Click the New credential button, give your credentials a name, and click Add. Your new credentials can be accessed by clicking the View credentials button. Copy and paste your Databases for PostgreSQL credentials into the cell below.
DB_CREDENTIALS = None
SCHEMA_NAME = None
If you previously configured OpenScale to use the free internal version of PostgreSQL, you can switch to a new datamart using a paid database service. If you would like to delete the internal PostgreSQL configuration and create a new one using service credentials supplied in the cell above, set the KEEP_MY_INTERNAL_POSTGRES variable below to False below. In this case, the notebook will remove your existing internal PostgreSQL datamart and create a new one with the supplied credentials. NO DATA MIGRATION WILL OCCUR.
KEEP_MY_INTERNAL_POSTGRES = True
In next cells, you will need to paste some credentials to Cloud Object Storage. If you haven't worked with COS yet please visit getting started with COS tutorial.
You can find COS_API_KEY_ID
and COS_RESOURCE_CRN
variables in Service Credentials in menu of your COS instance. Used COS Service Credentials must be created with Role parameter set as Writer. Later training data file will be loaded to the bucket of your instance and used as training refecence in subsription.
COS_ENDPOINT
variable can be found in Endpoint field of the menu.
COS_API_KEY_ID = "***"
COS_RESOURCE_CRN = "***"
COS_ENDPOINT = "***"
IAM_AUTH_ENDPOINT = "***"
BUCKET_NAME = "***"
At this point, the notebook is ready to run. You can either run the cells one at a time, or click the Kernel option above and select Restart and Run All to run all the cells.
!rm german_credit_data_biased_training.csv
!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/german_credit_data_biased_training.csv
Retrieve data filename, model name and deployment name.
training_data_file_name = "german_credit_data_biased_training.csv"
MODEL_NAME = "Scikit German Risk Model WML V4"
DEPLOYMENT_NAME = "Scikit German Risk Deployment WML V4"
The notebook will now import the necessary libraries and set up a Python OpenScale client.
from ibm_cloud_sdk_core.authenticators import IAMAuthenticator
from ibm_watson_openscale import APIClient
from ibm_watson_openscale.supporting_classes.enums import *
from ibm_watson_openscale.supporting_classes import *
authenticator = IAMAuthenticator(apikey=wml_credentials["apikey"])
wos_client = APIClient(authenticator=authenticator)
wos_client.version
Watson OpenScale uses a database to store payload logs and calculated metrics. If database credentials were not supplied above, the notebook will use the free, internal lite database. If database credentials were supplied, the datamart will be created there unless there is an existing datamart and the KEEP_MY_INTERNAL_POSTGRES variable is set to True. If an OpenScale datamart exists in Db2 or PostgreSQL, the existing datamart will be used and no data will be overwritten.
Prior instances of the German Credit model will be removed from OpenScale monitoring.
wos_client.data_marts.show()
data_marts = wos_client.data_marts.list().result.data_marts
if len(data_marts) == 0:
if DB_CREDENTIALS is not None:
if SCHEMA_NAME is None:
print("Please specify the SCHEMA_NAME and rerun the cell")
print("Setting up external datamart")
added_data_mart_result = wos_client.data_marts.add(
background_mode=False,
name="WOS Data Mart",
description="Data Mart created by WOS tutorial notebook",
database_configuration=DatabaseConfigurationRequest(
database_type=DatabaseType.POSTGRESQL,
credentials=PrimaryStorageCredentialsLong(
hostname=DB_CREDENTIALS["connection"]["postgres"]["hosts"][0]["hostname"],
username=DB_CREDENTIALS["connection"]["postgres"]["authentication"]["username"],
password=DB_CREDENTIALS["connection"]["postgres"]["authentication"]["password"],
db=DB_CREDENTIALS["connection"]["postgres"]["database"],
port=DB_CREDENTIALS["connection"]["postgres"]["hosts"][0]["port"],
ssl=True,
sslmode=DB_CREDENTIALS["connection"]["postgres"]["query_options"]["sslmode"],
certificate_base64=DB_CREDENTIALS["connection"]["postgres"]["certificate"]["certificate_base64"]
),
location=LocationSchemaName(
schema_name= SCHEMA_NAME
)
)
).result
else:
print("Setting up internal datamart")
added_data_mart_result = wos_client.data_marts.add(
background_mode=False,
name="WOS Data Mart",
description="Data Mart created by WOS tutorial notebook",
internal_database = True).result
data_mart_id = added_data_mart_result.metadata.id
else:
data_mart_id=data_marts[0].metadata.id
print("Using existing datamart {}".format(data_mart_id))
Multiple service providers for the same engine instance are avaiable in Watson OpenScale. To avoid multiple service providers of used WML instance in the tutorial notebook the following code deletes existing service provder(s) and then adds new one.
SERVICE_PROVIDER_NAME = "Watson Machine Learning V2"
SERVICE_PROVIDER_DESCRIPTION = "Added by tutorial WOS notebook"
service_providers = wos_client.service_providers.list().result.service_providers
for service_provider in service_providers:
service_instance_name = service_provider.entity.name
if service_instance_name == SERVICE_PROVIDER_NAME:
service_provider_id = service_provider.metadata.id
wos_client.service_providers.delete(service_provider_id)
print("Deleted existing service_provider for WML instance: {}".format(service_provider_id))
Watson OpenScale needs to be bound to the Watson Machine Learning instance to capture payload data into and out of the model.
Note: You can bind more than one engine instance if needed by calling wos_client.service_providers.add
method. Next, you can refer to particular service provider using service_provider_id
.
added_service_provider_result = wos_client.service_providers.add(
name=SERVICE_PROVIDER_NAME,
description=SERVICE_PROVIDER_DESCRIPTION,
service_type=ServiceTypes.WATSON_MACHINE_LEARNING,
deployment_space_id = space_id,
operational_space_id = "production",
credentials=WMLCredentialsCloud(
apikey=wml_credentials["apikey"],
url=wml_credentials["url"],
instance_id=None
),
background_mode=False
).result
service_provider_id = added_service_provider_result.metadata.id
wos_client.service_providers.show()
wos_client.subscriptions.show()
This code removes previous subscriptions to the German Credit model to refresh the monitors with the new model and new data.
subscriptions = wos_client.subscriptions.list().result.subscriptions
for subscription in subscriptions:
sub_model_id = subscription.entity.asset.asset_id
if sub_model_id == model_uid:
wos_client.subscriptions.delete(subscription.metadata.id)
print("Deleted existing subscription for model", model_uid)
Following cells create the model subscription in OpenScale using the Python client API. Note that we need to provide the model unique identifier, and some information about the model itself.
asset = Asset(
asset_id=model_uid,
url=deployment_url,
asset_type=AssetTypes.MODEL,
input_data_type=InputDataType.STRUCTURED,
problem_type=ProblemType.BINARY_CLASSIFICATION
)
asset_deployment = AssetDeploymentRequest(
deployment_id=deployment_uid,
name=DEPLOYMENT_NAME,
deployment_type=DeploymentTypes.ONLINE,
url=deployment_url
)
training_data_reference = TrainingDataReference(
type="cos",
location=COSTrainingDataReferenceLocation(
bucket=BUCKET_NAME,
file_name=training_data_file_name
),
connection=COSTrainingDataReferenceConnection.from_dict(
{
"resource_instance_id": COS_RESOURCE_CRN,
"url": COS_ENDPOINT,
"api_key": COS_API_KEY_ID,
"iam_url": "https://iam.bluemix.net/oidc/token"
}
)
)
asset_properties_request = AssetPropertiesRequest(
label_column="Risk",
probability_fields=["probability"],
prediction_field="prediction",
feature_fields=["CheckingStatus","LoanDuration","CreditHistory","LoanPurpose","LoanAmount","ExistingSavings","EmploymentDuration","InstallmentPercent","Sex","OthersOnLoan","CurrentResidenceDuration","OwnsProperty","Age","InstallmentPlans","Housing","ExistingCreditsCount","Job","Dependents","Telephone","ForeignWorker"],
categorical_fields=["CheckingStatus","CreditHistory","LoanPurpose","ExistingSavings","EmploymentDuration","Sex","OthersOnLoan","OwnsProperty","InstallmentPlans","Housing","Job","Telephone","ForeignWorker"],
training_data_reference=training_data_reference
)
subscription_details = wos_client.subscriptions.add(
data_mart_id=data_mart_id,
service_provider_id=service_provider_id,
asset=asset,
deployment=asset_deployment,
asset_properties=asset_properties_request).result
subscription_id = subscription_details.metadata.id
print(subscription_details)
import time
time.sleep(5)
payload_data_set_id = None
payload_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING,
target_target_id=subscription_id,
target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id
if payload_data_set_id is None:
print("Payload data set not found. Please check subscription status.")
else:
print("Payload data set id:", payload_data_set_id)
wos_client.data_sets.show()
Get subscription list
wos_client.subscriptions.show()
Now that the WML service has been bound and the subscription has been created, we need to send a request to the model before we configure OpenScale. This allows OpenScale to create a payload log in the datamart with the correct schema, so it can capture data coming into and out of the model. The sends a few records for predictions.
fields = ["CheckingStatus","LoanDuration","CreditHistory","LoanPurpose","LoanAmount","ExistingSavings","EmploymentDuration","InstallmentPercent","Sex","OthersOnLoan","CurrentResidenceDuration","OwnsProperty","Age","InstallmentPlans","Housing","ExistingCreditsCount","Job","Dependents","Telephone","ForeignWorker"]
values = [
["no_checking",10,"credits_paid_to_date","car_new",1343,"100_to_500","1_to_4",2,"female","none",3,"savings_insurance",46,"none","own",2,"skilled",1,"none","yes"],
["no_checking",20,"prior_payments_delayed","furniture",4567,"500_to_1000","1_to_4",4,"male","none",4,"savings_insurance",36,"none","free",2,"management_self-employed",1,"none","yes"],
["0_to_200",24,"all_credits_paid_back","car_new",863,"less_100","less_1",2,"female","co-applicant",2,"real_estate",38,"none","own",1,"skilled",1,"none","yes"],
["0_to_200",17,"no_credits","car_new",2368,"less_100","1_to_4",3,"female","none",3,"real_estate",29,"none","own",1,"skilled",1,"none","yes"],
["0_to_200",66,"no_credits","car_new",250,"less_100","unemployed",2,"female","none",3,"real_estate",23,"none","rent",1,"management_self-employed",1,"none","yes"],
["no_checking",14,"credits_paid_to_date","car_new",832,"100_to_500","1_to_4",2,"male","none",2,"real_estate",42,"none","own",1,"skilled",1,"none","yes"],
["no_checking",35,"outstanding_credit","appliances",5696,"unknown","greater_7",4,"male","co-applicant",4,"unknown",54,"none","free",2,"skilled",1,"yes","yes"],
["0_to_200",23,"prior_payments_delayed","retraining",1375,"100_to_500","4_to_7",3,"male","none",3,"real_estate",37,"none","own",2,"management_self-employed",1,"none","yes"]
]
payload_scoring = {"input_data": [{"fields": fields, "values": values}]}
predictions = client.deployments.score(deployment_uid, payload_scoring)
print("Single record scoring result:", "\n fields:", predictions["predictions"][0]["fields"], "\n values: ", predictions["predictions"][0]["values"][0])
import uuid
from ibm_watson_openscale.supporting_classes.payload_record import PayloadRecord
time.sleep(5)
pl_records_count = wos_client.data_sets.get_records_count(payload_data_set_id)
print("Number of records in the payload logging table: {}".format(pl_records_count))
if pl_records_count == 0:
print("Payload logging did not happen, performing explicit payload logging.")
wos_client.data_sets.store_records(data_set_id=payload_data_set_id, request_body=[PayloadRecord(
scoring_id=str(uuid.uuid4()),
request=payload_scoring,
response=predictions,
response_time=460
)])
time.sleep(5)
pl_records_count = wos_client.data_sets.get_records_count(payload_data_set_id)
print("Number of records in the payload logging table: {}".format(pl_records_count))
The code below waits ten seconds to allow the payload logging table to be set up before it begins enabling monitors. First, it turns on the quality (accuracy) monitor and sets an alert threshold of 70%. OpenScale will show an alert on the dashboard if the model accuracy measurement (area under the curve, in the case of a binary classifier) falls below this threshold.
The second paramater supplied, min_records, specifies the minimum number of feedback records OpenScale needs before it calculates a new measurement. The quality monitor runs hourly, but the accuracy reading in the dashboard will not change until an additional 50 feedback records have been added, via the user interface, the Python client, or the supplied feedback endpoint.
import time
time.sleep(10)
target = Target(
target_type=TargetTypes.SUBSCRIPTION,
target_id=subscription_id
)
parameters = {
"min_feedback_data_size": 50
}
quality_monitor_details = wos_client.monitor_instances.create(
data_mart_id=data_mart_id,
background_mode=False,
monitor_definition_id=wos_client.monitor_definitions.MONITORS.QUALITY.ID,
target=target,
parameters=parameters
).result
quality_monitor_instance_id = quality_monitor_details.metadata.id
The code below downloads and stores enough feedback data to meet the minimum threshold so that OpenScale can calculate a new accuracy measurement. It then kicks off the accuracy monitor. The monitors run hourly, or can be initiated via the Python API, the REST API, or the graphical user interface.
!rm additional_feedback_data_v2.json
!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/additional_feedback_data_v2.json
feedback_dataset_id = None
feedback_dataset = wos_client.data_sets.list(type=DataSetTypes.FEEDBACK,
target_target_id=subscription_id,
target_target_type=TargetTypes.SUBSCRIPTION).result
print(feedback_dataset)
feedback_dataset_id = feedback_dataset.data_sets[0].metadata.id
if feedback_dataset_id is None:
print("Feedback data set not found. Please check quality monitor status.")
import json
with open('additional_feedback_data_v2.json') as feedback_file:
additional_feedback_data = json.load(feedback_file)
wos_client.data_sets.store_records(feedback_dataset_id, request_body=additional_feedback_data, background_mode=False)
wos_client.data_sets.get_records_count(data_set_id=feedback_dataset_id)
run_details = wos_client.monitor_instances.run(monitor_instance_id=quality_monitor_instance_id, background_mode=False).result
wos_client.monitor_instances.show_metrics(monitor_instance_id=quality_monitor_instance_id)
The code below configures fairness monitoring for our model. It turns on monitoring for two features, Sex and Age. In each case, we must specify:
Additionally, we must specify which outcomes from the model are favourable outcomes, and which are unfavourable. We must also provide the number of records OpenScale will use to calculate the fairness score. In this case, OpenScale's fairness monitor will run hourly, but will not calculate a new fairness rating until at least 200 records have been added. Finally, to calculate fairness, OpenScale must perform some calculations on the training data, so we provide the dataframe containing the data.
wos_client.monitor_instances.show()
target = Target(
target_type=TargetTypes.SUBSCRIPTION,
target_id=subscription_id
)
parameters = {
"features": [
{"feature": "Sex",
"majority": ['male'],
"minority": ['female'],
"threshold": 0.95
},
{"feature": "Age",
"majority": [[26, 75]],
"minority": [[18, 25]],
"threshold": 0.95
}
],
"favourable_class": ["No Risk"],
"unfavourable_class": ["Risk"],
"min_records": 4
}
fairness_monitor_details = wos_client.monitor_instances.create(
data_mart_id=data_mart_id,
background_mode=False,
monitor_definition_id=wos_client.monitor_definitions.MONITORS.FAIRNESS.ID,
target=target,
parameters=parameters).result
fairness_monitor_instance_id =fairness_monitor_details.metadata.id
monitor_instances = wos_client.monitor_instances.list().result.monitor_instances
for monitor_instance in monitor_instances:
monitor_def_id=monitor_instance.entity.monitor_definition_id
if monitor_def_id == "drift" and monitor_instance.entity.target.target_id == subscription_id:
wos_client.monitor_instances.delete(monitor_instance.metadata.id)
print('Deleted existing drift monitor instance with id: ', monitor_instance.metadata.id)
target = Target(
target_type=TargetTypes.SUBSCRIPTION,
target_id=subscription_id
)
parameters = {
"min_samples": 20,
"train_drift_model": True,
"enable_model_drift": False,
"enable_data_drift": True
}
drift_monitor_details = wos_client.monitor_instances.create(
data_mart_id=data_mart_id,
background_mode=False,
monitor_definition_id=wos_client.monitor_definitions.MONITORS.DRIFT.ID,
target=target,
parameters=parameters
).result
drift_monitor_instance_id = drift_monitor_details.metadata.id
This next section randomly selects 200 records from the data feed and sends those records to the model for predictions. This is enough to exceed the minimum threshold for records set in the previous section, which allows OpenScale to begin calculating fairness.
from IPython.utils import io
!rm german_credit_feed.json
with io.capture_output() as captured:
!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/german_credit_feed.json -O german_credit_feed.json
!ls -lh german_credit_feed.json
Score 200 randomly chosen records
import random
with open('german_credit_feed.json', 'r') as scoring_file:
scoring_data = json.load(scoring_file)
fields = scoring_data['fields']
values = []
for _ in range(200):
values.append(random.choice(scoring_data['values']))
payload_scoring = {"input_data": [{"fields": fields, "values": values}]}
scoring_response = client.deployments.score(deployment_uid, payload_scoring)
time.sleep(5)
if pl_records_count == 8:
print("Payload logging did not happen, performing explicit payload logging.")
wos_client.data_sets.store_records(data_set_id=payload_data_set_id, request_body=[PayloadRecord(
scoring_id=str(uuid.uuid4()),
request=payload_scoring,
response=scoring_response,
response_time=460
)])
time.sleep(5)
pl_records_count = wos_client.data_sets.get_records_count(payload_data_set_id)
print("Number of records in the payload logging table: {}".format(pl_records_count))
print("Number of records in payload table:", wos_client.data_sets.get_records_count(data_set_id=payload_data_set_id))
Kick off a fairness monitor run on current data. The monitor runs hourly, but can be manually initiated using the Python client, the REST API, or the graphical user interface.
run_details = wos_client.monitor_instances.run(monitor_instance_id=fairness_monitor_instance_id, background_mode=False)
wos_client.monitor_instances.show_metrics(monitor_instance_id=fairness_monitor_instance_id)
Kick off a drift monitor run on current data. The monitor runs every hour, but can be manually initiated using the Python client, the REST API.
drift_run_details = wos_client.monitor_instances.run(monitor_instance_id=drift_monitor_instance_id, background_mode=False)
wos_client.monitor_instances.show_metrics(monitor_instance_id=drift_monitor_instance_id)
Finally, we provide OpenScale with the training data to enable and configure the explainability features.
target = Target(
target_type=TargetTypes.SUBSCRIPTION,
target_id=subscription_id
)
parameters = {
"enabled": True
}
explainability_details = wos_client.monitor_instances.create(
data_mart_id=data_mart_id,
background_mode=False,
monitor_definition_id=wos_client.monitor_definitions.MONITORS.EXPLAINABILITY.ID,
target=target,
parameters=parameters
).result
explainability_monitor_id = explainability_details.metadata.id
pl_records_resp = wos_client.data_sets.get_list_of_records(data_set_id=payload_data_set_id, limit=1, offset=0).result
scoring_ids = [pl_records_resp["records"][0]["entity"]["values"]["scoring_id"]]
print("Running explanations on scoring IDs: {}".format(scoring_ids))
explanation_types = ["lime", "contrastive"]
result = wos_client.monitor_instances.explanation_tasks(scoring_ids=scoring_ids, explanation_types=explanation_types).result
print(result)
def get_definition(monitor_name):
monitor_definitions = wos_client.monitor_definitions.list().result.monitor_definitions
for definition in monitor_definitions:
if monitor_name == definition.entity.name:
return definition
return None
monitor_name = 'my model performance'
metrics = [MonitorMetricRequest(name='sensitivity',
thresholds=[MetricThreshold(type=MetricThresholdTypes.LOWER_LIMIT, default=0.8)]),
MonitorMetricRequest(name='specificity',
thresholds=[MetricThreshold(type=MetricThresholdTypes.LOWER_LIMIT, default=0.75)])]
tags = [MonitorTagRequest(name='region', description='customer geographical region')]
existing_definition = get_definition(monitor_name)
if existing_definition is None:
custom_monitor_details = wos_client.monitor_definitions.add(name=monitor_name, metrics=metrics, tags=tags, background_mode=False).result
else:
custom_monitor_details = existing_definition
wos_client.monitor_definitions.show()
custom_monitor_id = custom_monitor_details.metadata.id
print(custom_monitor_id)
custom_monitor_details = wos_client.monitor_definitions.get(monitor_definition_id=custom_monitor_id).result
print('Monitor definition details:', custom_monitor_details)
target = Target(
target_type=TargetTypes.SUBSCRIPTION,
target_id=subscription_id
)
thresholds = [MetricThresholdOverride(metric_id='sensitivity', type = MetricThresholdTypes.LOWER_LIMIT, value=0.9)]
custom_monitor_instance_details = wos_client.monitor_instances.create(
data_mart_id=data_mart_id,
background_mode=False,
monitor_definition_id=custom_monitor_id,
target=target
).result
custom_monitor_instance_id = custom_monitor_instance_details.metadata.id
custom_monitor_instance_details = wos_client.monitor_instances.get(custom_monitor_instance_id).result
print(custom_monitor_instance_details)
from datetime import datetime, timezone, timedelta
from ibm_watson_openscale.base_classes.watson_open_scale_v2 import MonitorMeasurementRequest
custom_monitoring_run_id = "11122223333111abc"
measurement_request = [MonitorMeasurementRequest(timestamp=datetime.now(timezone.utc),
metrics=[{"specificity": 0.78, "sensitivity": 0.67, "region": "us-south"}], run_id=custom_monitoring_run_id)]
print(measurement_request[0])
published_measurement_response = wos_client.monitor_instances.measurements.add(
monitor_instance_id=custom_monitor_instance_id,
monitor_measurement_request=measurement_request).result
published_measurement_id = published_measurement_response[0]["measurement_id"]
print(published_measurement_response)
time.sleep(5)
published_measurement = wos_client.monitor_instances.measurements.get(monitor_instance_id=custom_monitor_instance_id, measurement_id=published_measurement_id).result
print(published_measurement)
print('Datamart:', data_mart_id)
print('Model:', model_uid)
print('Deployment:', deployment_uid)
Transaction IDs identified by the cells below can be copied and pasted into the Explainability tab of the OpenScale dashboard.
wos_client.data_sets.show_records(data_set_id=payload_data_set_id, limit=5)
You successfully completed this notebook!
You have finished the hands-on lab for IBM Watson OpenScale. You can now view the OpenScale Dashboard. Click on the tile for the German Credit model to see fairness, accuracy, and performance monitors. Click on the timeseries graph to get detailed information on transactions during a specific time window.
OpenScale shows model performance over time. You have two options to keep data flowing to your OpenScale graphs:
Check out our Online Documentation for more samples, tutorials, documentation, how-tos, and blog posts.
Lukasz Cmielowski, PhD, is an Automation Architect and Data Scientist at IBM with a track record of developing enterprise-level applications that substantially increases clients' ability to turn data into actionable knowledge.
Szymon Kucharczyk, Software Engineer at IBM Watson Machine Learning.
Copyright © 2020 IBM. This notebook and its source code are released under the terms of the MIT License.