Amazon SageMaker frameworks

You can use Amazon SageMaker to log payload, feedback, and to measure performance accuracy, bias detection, explainability, and auto-debias function in IBM Watson OpenScale.

IBM Watson OpenScale fully supports the following Amazon SageMaker frameworks:

Table 1. Framework support details

Framework Problem type Data type
Native Classification Structured
Native Regression1 Structured

1Support for regression models does not include drift magnitude.

Adding Amazon SageMaker to Watson OpenScale

You can configure Watson OpenScale to work with Amazon SageMaker by using one of the following methods:

Sample Notebooks

The following Notebooks show how to work with Amazon SageMaker:

Specifying an Amazon SageMaker ML service instance

Your first step in the Watson OpenScale tool is to specify an Amazon SageMaker service instance. Your Amazon SageMaker service instance is where you store your AI models and deployments.

Connect your Amazon SageMaker service instance

Watson OpenScale connects to AI models and deployments in an Amazon SageMaker service instance. To connect your service to Watson OpenScale, go to the Configure The configuration tab icon tab, add a machine learning provider, and click the Edit The configuration tab icon displays. icon. In addition to a name and description and whether the environment Pre-production or Production, you must provide the following information that is specific to this type of service instance:

  • Access Key ID: Your AWS access key ID, aws_access_key_id, which verifies who you are and authenticates and authorizes calls that you make to AWS.
  • Secret Access Key: Your AWS secret access key, aws_secret_access_key, which is required to verify who you are and to authenticate and authorize calls that you make to AWS.
  • Region: Enter the region where your Access Key ID was created. Keys are stored and used in the region in which they were created and cannot be transferred to another region. 

You are now ready to select deployed models and configure your monitors. Watson OpenScale lists your deployed models on the Insights dashboard where you can click Add to dashboard. Select the deployments that you want to monitor and click Configure.

For more information, see Configure monitors.

Payload logging with the Amazon SageMaker machine learning engine

Add your Amazon SageMaker machine learning engine

A non-IBM Watson Machine Learning engine is bound as Custom by using metadata. No direct integration with the non-IBM Watson Machine Learning service is possible.

SAGEMAKER_ENGINE_CREDENTIALS = {
                   'access_key_id':””,
                   'secret_access_key':””,
                   'region': '}
 
wos_client.service_providers.add(
        name="AWS",
        description="AWS Service Provider",
        service_type=ServiceTypes.AMAZON_SAGEMAKER,
        credentials=SageMakerCredentials(
            access_key_id=SAGEMAKER_ENGINE_CREDENTIALS['access_key_id'],
            secret_access_key=SAGEMAKER_ENGINE_CREDENTIALS['secret_access_key'],
            region=SAGEMAKER_ENGINE_CREDENTIALS['region']
        ),
        background_mode=False
    ).result

To see your service subscription, run the following code:

client.service_providers.list()

SageMaker ML binding

Add Amazon SageMaker ML subscription

To add the subscription, run the following code:

asset_deployment_details = wos_client.service_providers.list_assets(data_mart_id=data_mart_id, service_provider_id=service_provider_id).result
asset_deployment_details
 
deployment_id='684e35eee8a479470cee05983e1f9d64'
for model_asset_details in asset_deployment_details['resources']:
    if model_asset_details['metadata']['guid']==deployment_id:
        break
 
  
aws_asset = Asset(
        asset_id=model_asset_details['entity']['asset']['asset_id'],
        name=model_asset_details['entity']['asset']['name'],
        url=model_asset_details['entity']['asset']['url'],
        asset_type=model_asset_details['entity']['asset']['asset_type'] if 'asset_type' in model_asset_details['entity']['asset'] else 'model',
        problem_type=ProblemType.BINARY_CLASSIFICATION,
        input_data_type=InputDataType.STRUCTURED,
    )
 
from ibm_watson_openscale.base_classes.watson_open_scale_v2 import ScoringEndpointRequest
deployment_scoring_endpoint = model_asset_details['entity']['scoring_endpoint']
scoring_endpoint = ScoringEndpointRequest(url = model_asset_details['entity']['scoring_endpoint']['url'] )
 
deployment = AssetDeploymentRequest(
        deployment_id=model_asset_details['metadata']['guid'],
        url=model_asset_details['metadata']['url'],
        name=model_asset_details['entity']['name'],
        #description="asset['entity']['description']",
        deployment_type=model_asset_details['entity']['type'],
        scoring_endpoint =  scoring_endpoint
    )
 
asset_properties = AssetPropertiesRequest(
        label_column="Risk",
        prediction_field='predicted_label',
        probability_fields=['score'],
        training_data_reference=training_data_reference,
        training_data_schema=None,
        input_data_schema=None,
        output_data_schema=None,
        feature_fields=feature_columns,
        categorical_fields=categorical_columns
    )
 
subscription_details = wos_client.subscriptions.add(
        data_mart_id=data_mart_id,
        service_provider_id=service_provider_id,
        asset=aws_asset,
        deployment=deployment,
        asset_properties=asset_properties,
        background_mode=False
).result

To get the subscription list, run the following command:

subscription_id = subscription_details.metadata.id
subscription_id
 
details: wos_client.subscriptions.get(subscription_id).result.to_dict()

Enable payload logging

To enable payload logging, run the following command:

request_data = {'fields': feature_columns, 
                'values': [[payload_values]]}

To get logging details, run the following command:

response_data = {'fields': list(result['predictions'][0]),
                 'values': [list(x.values()) for x in result['predictions']]}

Scoring and payload logging

To store the request and response in the payload logging table, run the following code:

wos_client.data_sets.store_records(data_set_id=payload_data_set_id, request_body=[PayloadRecord(
           scoring_id=str(uuid.uuid4()),
           request=request_data,
           response=response_data,
           response_time=460
)])
 

For languages other than Python, you can also log payload by using a REST API.

Next steps