0 / 0
Managing data for model evaluations
Managing data for model evaluations

Managing data for model evaluations

To configure monitors, you must send a scoring payload to log the data to be monitored.

For models deployed in IBM Watson Machine Learning, you must score your model by using IBM Watson Machine Learning API. The scoring payload is automatically sent to Watson OpenScale.

For other machine learning engines (for example: Microsoft Azure, Amazon SageMaker) the scoring payload must be sent by using the Payload Logging API. For more information, see Payload logging for non-IBM Watson Machine Learning service instances.

Steps for payload logging

  1. From the Watson OpenScale dashboard, click a deployment tile.
  2. Click Configure monitors.
  3. In the navigation panel, click Payload logging.
  4. Choose whether to use the cURL or Python code by clicking the cURL or Python tab.
  5. Click Copy to clipboard and paste it into to log model deployment request and response data.

For more information, see Payload logging for non-IBM Watson Machine Learning service instances.

The fields and values in the code snippets need to be substituted with your real values, as the ones provided are only examples.

Select database

After you run your payload logging, a checkmark appears in the Ready to Monitor column for the selected deployment. Click Configure Monitors to continue.

Payload logging fields

The following fields make up a typical payload logging request:

  • scoring_id: ID of the scoring transaction.
  • response_time: The time (ms) taken to make prediction (for performance monitoring).
  • request_data: The request section specifies fields and values for both the training data and the metadata, which are features that you can use to measure indirect bias.
    • fields: An array of field names.
    • values: An array of field values.
  • meta: The meta section specifies fields and values for the metadata.
    • fields: Lists the fields for the metadata.
    • values: Lists the values for the metadata.
  • response_data: The response section specifies the fields and values of the result.
    • fields: An array of field names.
    • values: An array of field values.
  • subscription_id: The unique ID of the subscription.
  • deployment_id: The unique ID of the deployment in Watson OpenScale.

Meta fields and values are not processed or known by the model but can be passed to Watson OpenScale to log additional details or sensitive attributes that are related to a model transaction. When configuring model settings, meta fields should be included in the training data set. When a meta field is selected for fairness evaluation, Watson OpenScale finds correlated training features and evaluates the model for indirect bias.

Payload logging cURL sample

The following sample shows the cURL payload logging command:

curl -k -X POST https://$ICP_HOSTNAME:$PORT/openscale/00000000-0000-0000-0000-000000000000/v2/data_sets/<dataset_id>/records -d "$SCORING_PAYLOAD" \
--header 'Content-Type: application/json' --header 'Accept: application/json' --header "Authorization: Bearer $ICP_TOKEN"

Payload logging Python sample

The following file shows a sample Python payload logging command with the required authentication:

from ibm_cloud_sdk_core.authenticators import IAMAuthenticator
from ibm_watson_openscale import APIClient

service_credentials = {
                  "apikey": "*****",
                  "url": "https://api.aiopenscale.cloud.ibm.com"
               }

authenticator = IAMAuthenticator(
        apikey=service_credentials['apikey']
    )
client = APIClient(
    service_url=service_credentials['url'],
    service_instance_id="230a8e9f-2453-4c2d-a560-2a75399210bf"
    authenticator=authenticator
)

from ibm_watson_openscale.data_sets import DataSetTypes, TargetTypes

# Put your subscription ID here
SUBSCRIPTION_ID = "<subscription_id>"
payload_logging_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, target_target_id=SUBSCRIPTION_ID, target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id

from ibm_watson_openscale.supporting_classes.payload_record import PayloadRecord

# Put your scoring ID here
SCORING_ID = "<scoring_id>"
REQUEST_DATA = <put_your_data_here>
RESPONSE_DATA = <put_your_data_here>
wos_client.data_sets.store_records(data_set_id=payload_logging_data_set_id, request_body=[PayloadRecord(scoring_id=SCORING_ID, request=REQUEST_DATA, response=RESPONSE_DATA, response_time=460)])

Understanding the number of scoring requests

Scoring requests are a part of Watson OpenScale processing. Each transaction record that you send to the model to be scored generates extra processing so that the Watson OpenScale service perturbs data and creates explanations.

  • For tabular, text, or image models the following baseline request generates data points:

    • One scheduling service-scoring request generates 5000 data points.
  • For tabular classification models only, there are more scoring requests that are made for contrastive explanation. The following requests are added to the preceding baseline request:

    • 100 scoring requests generate 51 extra data points per request.
    • 101 scoring requests generate one extra data point per request.

Next steps

Configure models for evaluation.

Parent topic: Configure Watson OpenScale

Generative AI search and answer
These answers are generated by a large language model in watsonx.ai based on content from the product documentation. Learn more