To create custom evaluations, select a set of custom metrics to quantitatively track your model deployment and business application. You can define these custom metrics and use them alongside metrics that are generated by other types of evaluations.
You can use one of the following methods to manage custom evaluations and metrics:
Step 1: Register custom monitor with metrics definition.
Copy link to section
Before you can start by using custom metrics, you must register the custom monitor, which is the processor that tracks the metrics. You also must define the metrics themselves.
Use the get_definition(monitor_name) method to import the Metric and Tag objects.
Use the metrics method to define the metrics, which require name, thresholds, and type values.
Use the tags method to define metadata.
The following code is from the working sample notebook that was previously mentioned:
defget_definition(monitor_name):
monitor_definitions = wos_client.monitor_definitions.list().result.monitor_definitions
for definition in monitor_definitions:
if monitor_name == definition.entity.name:
return definition
returnNone
monitor_name = 'my model performance'
metrics = [MonitorMetricRequest(name='sensitivity',
thresholds=[MetricThreshold(type=MetricThresholdTypes.LOWER_LIMIT, default=0.8)]),
MonitorMetricRequest(name='specificity',
thresholds=[MetricThreshold(type=MetricThresholdTypes.LOWER_LIMIT, default=0.75)])]
tags = [MonitorTagRequest(name='region', description='customer geographical region')]
existing_definition = get_definition(monitor_name)
if existing_definition isNone:
custom_monitor_details = wos_client.monitor_definitions.add(name=monitor_name, metrics=metrics, tags=tags, background_mode=False).result
else:
custom_monitor_details = existing_definition
Copy to clipboardCopied to clipboardShow more
To check how you're doing, run the client.data_mart.monitors.list() command to see whether your newly created monitor and metrics are configured properly.
You can also get the monitor ID by running the following command:
Next, you must enable the custom monitor for subscription. This activates the monitor and sets the thresholds.
Use the target method to import the Threshold object.
Use the thresholds method to set the metric lower_limit value. Supply the metric_id value as one of the parameters. If you don't remember, you can always use the custom_monitor_details command to get the details as shown in the previous example.
The following code is from the working sample notebook that was previously mentioned:
If you want to configure a metric group manually, click Configure new group. a. Specify a name and a description for the metric group. The length of the name that you specify must be less than or equal to 48 characters. b. Click the Edit icon on the Input parameters tile and the specify the details for your input parameters. The parameter name that you specify must match the parameter name that is specified in the
metric API. c. If the parameter is required to configure your custom monitor, select the Required parameter checkbox. d. Click Add. After you add the input parameters, click Next. e. Select the model types that your evaluation supports and click Next. f. If you don't want to specify an evaluation schedule, click Save. g. If you want to specify an evaluation schedule, click
the toggle. You must specify the interval for the evaluation schedule and click Save. h. Click Add metric and specify the metric details. Click Save.
If you want to configure a metric group by using a JSON file, click Import from file. Upload a JSON file and click Import.
Step 2: Add metric endpoints
Copy link to section
In the Metric endpoints section, click Add metric endpoint.
Specify a name and a description for the metric endpoint.
Click the Edit icon on the Connection tile and specify the connection details. Click Next.
Select the metric groups that you want associate with the metric endpoint and click Save.
Step 3: Configure custom monitors
Copy link to section
On the Insights Dashboard page, select Configure monitors on a model deployment tile.
In the Evaluations section, select the name of the metric group that you added.
Select the Edit icon on the Metric endpoint tile.
Select a metric endpoint and click Next. If you don't want to use a metric endpoint, select None.
Use the toggles to specify the metrics that you want to use to evaluate the model and provide threshold values. Click Next.
Specify values for the input parameters. If you selected JSON as the data type for the metric group, add the JSON data. Click Next.
You can now evaluate models with a custom monitor.
Accessing and visualizing custom metrics
Copy link to section
To access and visualize custom metrics, you can use programmatic interface. The following advanced tutorial shows how to do this:
About cookies on this siteOur websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising.For more information, please review your cookie preferences options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.