Sending a scoring request

To configure monitors, you must send a scoring payload to log the data to be monitored.

For models deployed in IBM Watson Machine Learning, you must score your model by using IBM Watson Machine Learning API. The scoring payload is automatically sent to Watson OpenScale when you score the model.

For other machine learning engines, such as Microsoft Azure, Amazon SageMaker, or a custom machine learning engine the scoring payload must be sent by using the Payload Logging API. For more information, see Payload logging for non-IBM Watson Machine Learning service instances.

Steps for payload logging

  1. From the Watson OpenScale dashboard, click a deployment tile.
  2. Click Configure monitors.
  3. In the navigation pane, click Payload logging.
  4. Choose whether to use the cURL or Python code by clicking the cURL or Python tab.
  5. Click Copy to clipboard and paste it into to log model deployment request and response data.

For more information, see Payload logging for non-IBM Watson Machine Learning service instances.

The fields and values in the code snippets need to be substituted with your real values, as the ones provided are only examples.

Select database

After you run your payload logging, a check mark appears in the Ready to Monitor column for the selected deployment. Click Configure Monitors to continue.

Payload logging fields

The following fields make up a typical payload logging request:

  • scoring_id: ID of the scoring transaction.
  • response_time: The time (ms) taken to make prediction (for performance monitoring).
  • request_data: The request section specifies fields and values for both the training data and the meta data, which are features that you can use to measure indirect bias.
    • fields: An array of field names.
    • values: An array of field values.
  • meta: The meta section specifies fields and values for the meta data.
    • fields: Lists the fields for the meta data.
    • values: Lists the values for the meta data.
  • response_data: The response section specifies the fields and values of the result.
    • fields: An array of field names.
    • values: An array of field values.
  • subscription_id: The unique ID of the subscription.
  • deployment_id: The unique ID of the deployment in Watson OpenScale.

Meta fields and values are not processed or known by the model but can be passed to Watson OpenScale to log additional details or sensitive attributes related to a model transaction. Meta fields should be included in the training data set identified when configuring model settings. When a meta field is selected for fairness evaluation, Watson OpenScale finds correlated training features and evaluates the model for indirect bias.

Payload logging cURL sample

The following sample shows the cURL payload logging command:

curl -k -X POST https://$ICP_HOSTNAME:$PORT/openscale/00000000-0000-0000-0000-000000000000/v2/data_sets/<dataset_id>/records -d "$SCORING_PAYLOAD" \
--header 'Content-Type: application/json' --header 'Accept: application/json' --header "Authorization: Bearer $ICP_TOKEN"

Payload logging Python sample

The following file shows a sample Python payload logging command with the required authentication:

from ibm_cloud_sdk_core.authenticators import IAMAuthenticator
from ibm_watson_openscale import APIClient

service_credentials = {
                  "apikey": "*****",
                  "url": "https://api.aiopenscale.cloud.ibm.com"
               }

authenticator = IAMAuthenticator(
        apikey=service_credentials['apikey']
    )
client = APIClient(
    service_url=service_credentials['url'],
    service_instance_id="230a8e9f-2453-4c2d-a560-2a75399210bf"
    authenticator=authenticator
)

from ibm_watson_openscale.data_sets import DataSetTypes, TargetTypes

# Put your subscription ID here
SUBSCRIPTION_ID = "<subscription_id>"
payload_logging_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, target_target_id=SUBSCRIPTION_ID, target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id

from ibm_watson_openscale.supporting_classes.payload_record import PayloadRecord

# Put your scoring ID here
SCORING_ID = "<scoring_id>"
REQUEST_DATA = <put_your_data_here>
RESPONSE_DATA = <put_your_data_here>
wos_client.data_sets.store_records(data_set_id=payload_logging_data_set_id, request_body=[PayloadRecord(scoring_id=SCORING_ID, request=REQUEST_DATA, response=RESPONSE_DATA, response_time=460)])

Understanding the number of scoring requests

Scoring requests are a part of Watson OpenScale processing. Each transaction record that you send to the model to be scored generates extra processing so that the Watson OpenScale service perturbs data and creates explanations.

  • For tabular, text, or image models the following baseline request generates data points:

    • 1 scheduling service scoring request generates 5000 data points.
  • For tabular classification models only, there are more scoring requests that are made for contrastive explanation. The following requests are added to the preceding baseline request:

    • One hundred (100) scoring requests generate 51 additional data points per request.
    • One hundred one (101) scoring requests generate 1 additional data point per request.

Next steps

Configure monitors.