External machine learning service engines, such as Microsoft Azure ML Studio, Microsoft Azure ML Service, and Amazon SageMaker can be integrated for model evaluations.

You can use the following methods to integrate integrate 3rd-party engines:

• Engine binding level

  Ability to list deployments and get deployment details.

• Deployment subscription level

  You must score subscribed deployment in the proper format, such as the IBM watsonx.ai Runtime format and receive the output in the same compatible format. You must configure your model evaluations to process both input and output formats.

• Payload logging

  Each input and output to the deployed model triggered by a user’s application must be stored in a payload store. The format of the payload records follows the same specification as mentioned in the preceding section on deployment subscription levels.

  Those records are used during model evaluations to calculate bias, explanations, and others. It is not possible to automatically store transactions that run on the user site. This method is one of the ways that proprietary information is safeguarded during model evaluations. Use the Rest API or Python SDK to work with secure data.