External machine learning service engines, such as Microsoft Azure ML Studio, Microsoft Azure ML Service, and Amazon SageMaker can be integrated for model evaluations.
You can use the following methods to integrate integrate 3rd-party engines:
-
Engine binding level
Ability to list deployments and get deployment details.
-
Deployment subscription level
You must score subscribed deployment in the proper format, such as the IBM watsonx.ai Runtime format and receive the output in the same compatible format. You must configure your model evaluations to process both input and output formats.
-
Payload logging
Each input and output to the deployed model triggered by a user’s application must be stored in a payload store. The format of the payload records follows the same specification as mentioned in the preceding section on deployment subscription levels.
Those records are used during model evaluations to calculate bias, explanations, and others. It is not possible to automatically store transactions that run on the user site. This method is one of the ways that proprietary information is safeguarded during model evaluations. Use the Rest API or Python SDK to work with secure data.
Steps to implement this solution
- Learn about custom machine learning engines.
- Set up payload logging.
- Set up a custom machine learning engine by using one of these Custom machine learning engine examples.
Parent topic: Supported machine learning engines, frameworks, and models