Creating custom monitors and metrics

Custom monitors consolidate a set of custom metrics that enable you to track, in a quantitative way, any aspect of your model deployment and business application. You can define custom metrics, and use them alongside the standard metrics, such as model quality, performance, or fairness metrics that are monitored in IBM Watson OpenScale.

To manage custom monitors and metrics you must use the programmatic interface that is part of the Python SDK. In similar way, you can store custom metrics in the Watson OpenScale datamart to access them when needed. Custom metrics are also visualized on the Watson OpenScale Dashboard.

Managing custom metrics

To work with custom metrics you must perform the following tasks:

  1. Register custom monitor with metrics definition.
  2. Enable custom monitor.
  3. Store metric values.

The following advanced tutorial shows how to do this:

You can disable and enable again custom monitoring at any time. You can remove custom monitor if you do not need it anymore.

For more information, see the Python SDK documentation.

Step 1: Register custom monitor with metrics definition.

Before you can start using custom metrics, you must register the custom monitor, which is the processor that tracks the metrics. You also must define the metrics themselves.

  1. Use the get_definition(monitor_name) method to import the Metric and Tag objects.
  2. Use the metrics method to define the metrics, which require name, thresholds, and type values.
  3. Use the tags method to define metadata.

The following code is from the working sample notebook that was previously mentioned:

def get_definition(monitor_name):
    monitor_definitions = wos_client.monitor_definitions.list().result.monitor_definitions
   
    for definition in monitor_definitions:
        if monitor_name == definition.entity.name:
            return definition
   
    return None
 
 
monitor_name = 'my model performance'
metrics = [MonitorMetricRequest(name='sensitivity',
                                thresholds=[MetricThreshold(type=MetricThresholdTypes.LOWER_LIMIT, default=0.8)]),
          MonitorMetricRequest(name='specificity',
                                thresholds=[MetricThreshold(type=MetricThresholdTypes.LOWER_LIMIT, default=0.75)])]
tags = [MonitorTagRequest(name='region', description='customer geographical region')]
 
existing_definition = get_definition(monitor_name)
 
if existing_definition is None:
    custom_monitor_details = wos_client.monitor_definitions.add(name=monitor_name, metrics=metrics, tags=tags, background_mode=False).result
else:
    custom_monitor_details = existing_definition

To check how you’re doing, run the client.data_mart.monitors.list() command to see if your newly-created monitor and metrics are configured properly.

You can also get the monitor ID by running the following command:

custom_monitor_id = custom_monitor_details.metadata.id
 
print(custom_monitor_id)

For a more detailed look, run the following command:

custom_monitor_details = wos_client.monitor_definitions.get(monitor_definition_id=custom_monitor_id).result
print('Monitor definition details:', custom_monitor_details)

Step 2: Enable custom monitor.

Next, you must enable the custom monitor for subscription. This activates the monitor and sets the thresholds.

  1. Use the target method to import the Threshold object.
  2. Use the thresholds method to set the metric lower_limit value. You’ll need to supply the metric_id value as one of the parameters. If you don’t remember, you can always use the custom_monitor_details command to get the details as shown in the previous example.

The following code is from the working sample notebook that was previously mentioned:

target = Target(
        target_type=TargetTypes.SUBSCRIPTION,
        target_id=subscription_id
    )
 
thresholds = [MetricThresholdOverride(metric_id='sensitivity', type = MetricThresholdTypes.LOWER_LIMIT, value=0.9)]
 
custom_monitor_instance_details = wos_client.monitor_instances.create(
            data_mart_id=data_mart_id,
            background_mode=False,
            monitor_definition_id=custom_monitor_id,
            target=target
).result

To check on your configuration details, use the subscription.monitoring.get_details(monitor_uid=monitor_uid) command.

Step 3: Store metric values.

You must store, or save, your custom metrics to the region where your Watson OpenScale instance exists.

  1. Use the metrics method to set which metrics you are storing.
  2. Use the subscription.monitoring.store_metrics method to commit the metrics.

The following code is from the working sample notebook that was previously mentioned:

from datetime import datetime, timezone, timedelta
from ibm_watson_openscale.base_classes.watson_open_scale_v2 import MonitorMeasurementRequest
custom_monitoring_run_id = "11122223333111abc"
measurement_request = [MonitorMeasurementRequest(timestamp=datetime.now(timezone.utc),
                                                 metrics=[{"specificity": 0.78, "sensitivity": 0.67, "region": "us-south"}], run_id=custom_monitoring_run_id)]
print(measurement_request[0])
 
published_measurement_response = wos_client.monitor_instances.measurements.add(
    monitor_instance_id=custom_monitor_instance_id,
    monitor_measurement_request=measurement_request).result
published_measurement_id = published_measurement_response[0]["measurement_id"]
print(published_measurement_response)

To list all custom monitors, run the following command:

published_measurement = wos_client.monitor_instances.measurements.get(monitor_instance_id=custom_monitor_instance_id, measurement_id=published_measurement_id).result
print(published_measurement)

Accessing and visualizing custom metrics

To access and visualize custom metrics you can use programmatic interface. The following advanced tutorial shows how to do this:

Visualization of your custom metrics appear on the Watson OpenScale Dashboard.