Custom ML frameworks

You can use your custom machine learning framework to perform payload logging, feedback logging, and to measure performance accuracy, runtime bias detection, explainability, drift detection, and auto-debias function in IBM Watson OpenScale. The custom machine learning framework must have equivalency to IBM Watson Machine Learning.

IBM Watson OpenScale fully supports the following custom machine learning frameworks:

Table 1. Framework support details

Framework Problem type Data type
Equivalent to IBM Watson Machine Learning Classification Structured
Equivalent to IBM Watson Machine Learning Regression Structured

For a model that is not equivalent to IBM Watson Machine Learning, you must create a wrapper for the custom model that exposes the required REST API endpoints and bridge the input/output between Watson OpenScale and the actual custom machine learning engine.

When is a custom machine learning engine the best choice for me?

A custom machine learning engine is the best choice when the following situations are true:

  • You are not using any available out-of-the-box products to serve your machine learning models. You have just developed your own system to do that. There is no, and will be no, direct support in Watson OpenScale for that.
  • The serving engine you are using from a 3rd-party supplier is not supported by Watson OpenScale yet. In this case, consider developing a custom machine learning engine as a wrapper to your original or native deployments.

How it works

The following image shows the custom environment support:

How Custom works chart is displayed. It shows boxes for the custom environment with the client API and the Watson OpenScale API

You can also reference the following links:

Watson OpenScale payload logging API

Custom deployment API

Python client SDK

Python SDK for IBM Watson OpenScale

  • Input criteria for model to support monitors

    Your model should take as input a feature vector, which is essentially a collection of named fields and their values (the fields being monitored for bias being one of those fields):

    {
      "fields": [
          "name",
          "age",
          "position"
      ],
      "values": [
          [
              "john",
              33,
              "engineer"
          ],
          [
              "mike",
              23,
              "student"
          ]
      ]
      
    

    In this example, “age” could be a field someone is evaluating for fairness.

    If the input is a tensor/matrix, which is transformed from the input feature space (which is often the case in deep learning from text or images), that model cannot be handled by the Watson OpenScale platform in the current release. By extension, deep learning models with text or image inputs cannot be handled for bias detection and mitigation.

    Additionally, training data should be loaded to support Explainability.

    For explainability on text, the full text should be one of the features. Explainability on images for a Custom model is not supported in the current release.

  • Output criteria for model to support monitors

    Your model should output the input feature vector alongside the prediction probabilities of various classes in that model.

    {
      "fields": [
          "name",
          "age",
          "position",
          "prediction",
          "probability"
      ],
      "labels": [
          "personal",
          "camping"
      ],
      "values": [
          [
              "john",
              33,
              "engineer",
              "personal",
              [
                  0.6744664422398081,
                  0.3255335577601919
              ]
          ],
          [
              "mike",
              23,
              "student"
              "camping",
              [
                  0.2794765664946941,
                  0.7205234335053059
              ]
          ]
      ]
    }
    

    In this example, "personal” and “camping” are the possible classes, and the scores in each scoring output are assigned to both classes. If the prediction probabilities are missing, bias detection will work, but auto-debias will not.

    The preceding scoring output should be accessible from a live scoring endpoint which Watson OpenScale could call over REST. For CUSTOMML, SageMaker, and IBM Watson Machine Learning, Watson OpenScale directly connects to the native scoring endpoints, (so you don’t have to worry about implementing the scoring spec).

Custom machine learning engine

A custom machine learning engine provides the infrastructure and hosting capabilities for machine learning models and web applications. Custom machine learning engines that are supported by Watson OpenScale must conform to the following requirements:

  • Expose two types of REST API endpoints:

    • discovery endpoint (GET list of deployments and details)
    • scoring endpoints (online and real-time scoring)
  • All endpoints need to be compatible with the swagger specification to be supported.

  • Input payload and output to or from the deployment must be compliant with the JSON file format that is described in the specification.

At this stage only the BasicAuth or none formats are supported.

To see the REST API endpoints specification, see Watson OpenScale REST API.

Adding a custom machine learning engine to Watson OpenScale

You can configure Watson OpenScale to work with a custom machine learning provider by using one of the following methods:

  • If this is the first time that you are adding a custom machine learning provider to Watson OpenScale, you can use the configuration interface. For more information, see Specifying a custom machine learning instance.
  • You can also add your machine learning provider by using the Python SDK. You must use this method if you want to have more than one provider. For more information on performing this programmatically, see Add your custom machine learning engine.

Explore further

Use a custom machine learning monitor to create a way to interact with other services, such as IBM Watson Assistant.

Specifying a Custom ML service instance

Your first step in the Watson OpenScale tool is to specify a service instance. Your service instance is where you store your AI models and deployments.

Connect your Custom service instance

Watson OpenScale connects to AI models and deployments in a service instance. You can connect a custom service. To connect your service to Watson OpenScale, go to the Configure The configuration tab icon tab, add a machine learning provider, and click the Edit The configuration tab icon icon. In addition to a name and description and whether this is a Pre-production or Production environment type, you must provide the following information that is specific to this type of service instance:

  • Username
  • Password
  • API endpoint, that uses the format https://host:port, such as https://custom-serve-engine.mybluemix.net:8443

Choose whether to connect to your deployments by requesting a list or by entering individual scoring endpoints.

Requesting the list of deployments

If you selected the Request the list of deployments tile, enter your credentials and API Endpoint, then save your configuration.

Providing individual scoring endpoints

If you selected the Enter individual scoring endpoints tile, enter your credentials for the API Endpoint, then save your configuration.

You are now ready to select deployed models and configure your monitors. Watson OpenScale lists your deployed models on the Insights dashboard where you can click the Add to dashboard button. Select the deployments you want to monitor and click Configure.

For more information, see Configure monitors.

Custom machine learning engine examples

Use the following ideas to set up your own custom machine learning engine.

Python and flask

You can use Python and flask to serve scikit-learn model.

To generate the drift detection model, you must use scikit-learn version 0.20.2 in the notebook.

The app can be deployed locally for testing purposes as well as an application on IBM Cloud.

Node.js

You can also find example of custom machine learning engine written in Node.js here.

End2end code pattern

Code pattern showing end2end example of custom engine deployment and integration with Watson OpenScale.

Payload logging with the Custom machine learning engine

To configure payload logging for a non-IBM Watson Machine Learning or custom machine learning engine, you must bind the ML engine as custom.

Add your Custom machine learning engine

A non- engine is added as custom by using metadata; there is no direct integration with the non-IBM Watson Machine Learning service. You can add more than one machine learning engine to Watson OpenScale by using the wos_client.service_providers.add method.

CUSTOM_ENGINE_CREDENTIALS = {
    "url": "***",
    "username": "***",
    "password": "***",
}
 
wos_client.service_providers.add(
        name=SERVICE_PROVIDER_NAME,
        description=SERVICE_PROVIDER_DESCRIPTION,
        service_type=ServiceTypes.CUSTOM_MACHINE_LEARNING,
        credentials=CustomCredentials(
            url= CUSTOM_ENGINE_CREDENTIALS['url'],
            username= CUSTOM_ENGINE_CREDENTIALS['username'],
            password= CUSTOM_ENGINE_CREDENTIALS['password'],
        ),
        background_mode=False
    ).result

You can see your service provider with the following command:

client.service_providers.get(service_provider_id).result.to_dict()

Generic ML binding

Add Custom subscription

To add a custom subscription, run the following command:

custom_asset = Asset(
        asset_id=asset['entity']['asset']['asset_id'],
        name=asset['entity']['asset']['name'],
        url = "dummy_url",
        asset_type=asset['entity']['asset']['asset_type'] if 'asset_type' in asset['entity']['asset'] else 'model',
        problem_type=ProblemType.MULTICLASS_CLASSIFICATION,
        input_data_type=InputDataType.STRUCTURED,
    )
deployment = AssetDeploymentRequest(
        deployment_id=asset['metadata']['guid'],
        url=asset['metadata']['url'],
        name=asset['entity']['name'],
        deployment_type=asset['entity']['type'],
        scoring_endpoint =  scoring_endpoint
    )
asset_properties = AssetPropertiesRequest(
        prediction_field='predicted_label',
        probability_fields = ["probability"],
        training_data_reference=None,
        training_data_schema=None,
        input_data_schema=None,
        output_data_schema=output_schema,
    )
result = ai_client.subscriptions.add(
        data_mart_id=cls.datamart_id,
        service_provider_id=cls.service_provider_id,
        asset=custom_asset,
        deployment=deployment,
        asset_properties=asset_properties,
        background_mode=False
    ).result

To get the subscription list, run the following command:

subscription_id = subscription_details.metadata.id
subscription_id
 
details: wos_client.subscriptions.get(subscription_id).result.to_dict()

Enable payload logging

To enable payload logging in subscription, run the following command:

request_data = {'fields': feature_columns, 
                'values': [[payload_values]]}

To get logging details, run the following command:

response_data = {'fields': list(result['predictions'][0]),
                 'values': [list(x.values()) for x in result['predictions']]}

For more information, see Payload logging.

Scoring and payload logging

  • Score your model.
  • Store the request and response in the payload logging table

      records_list = [PayloadRecord(request=request_data, response=response_data, response_time=response_time), PayloadRecord(request=request_data, response=response_data, response_time=response_time)]
    
      subscription.payload_logging.store(records=records_list)
    

For languages other than Python, you can also log payload by using a REST API.

Next steps

Watson OpenScale is now ready for you to configure monitors.