Benefits of converting models to ONNX runtime

Converting a model to ONNX runtime offers several benefits, especially in the context of machine learning and deep learning applications. Some of the advantages of converting models to ONNX runtime are as follows:

• Cross-platform compatibility: ONNX provides a standard format for representing machine learning models, which makes it easier to deploy models across different frameworks such as PyTorch or Tensorflow. You can train models in one frameworks and deploy them in another framework that supports ONNX runtime.

• Improved performance: ONNX runtime optimizes models for inferencing by applying various hardware and software-specific optimizaitons, such as graph optimizations. Also, it supports execution on diverse hardware, such as CPUs and GPUs, ensuring efficient utilization of resources.

• Interoperability: ONNX provides a way to train models, such as PyTorch, TensorFlow, and scikit-learn in one framework and then export them to run in another environment, which streamlines the workflows. It breaks down the barriers between different deep learning frameworks, allowing developers to leverage the strengths of different libraries without getting locked into a single ecosystem.

Supported frameworks for conversion

You can convert machine learning models that use the following frameworks to ONNX format:

1. PyTorch
2. TensorFlow
3. CatBoost
4. LightGBM
5. XGBoost
6. Scikit-learn

Converting PyTorch models to ONNX format

Follow this process to convert your trained model in PyTorch to the ONNX format:

1. Import libraries: Start by importing the essential libraries, such as onnxruntime for running the model, torch for PyTorch functionalities, and other libraries required for your application.

2. Create or download PyTorch model: You can create a PyTorch model by using your own data set or use models provided by external open source model repositories like Hugging Face.

3. Convert PyTorch model to ONNX format: To convert the PyTorch model to ONNX format:

   a. Prepare the model: Ensure that your PyTorch model is in evaluation mode by using model.eval() function. You may need a dummy input tensor to match the shape of the model.

   b. Export the model: Use the torch.onnx.export function to convert the model to ONNX format.

4. Verify the conversion: After converting the model, verify that the model is functioning as expected by using the onnx library.

Converting TensorFlow models to ONNX format

Follow this process to convert your model TensorFlow to the ONNX format:

1. Import libraries: Start by importing the essential libraries, such as tf2onnx to facilitate conversion of TensorFlow models to ONNX, and other libraries required for your application.

2. Download TensorFlow model: You must download the externally created TensorFlow model and the data that is used for training the model.

3. Convert TensorFlow model to ONNX format: Use the tf2onnx.convert command to convert your TensorFlow model that is created in the SavedModel format to ONNX format. If you want to convert a TensorFlow Lite model, use the --tflite flag instead of the --saved-model flag.

4. Verify the conversion: After converting the model, verify that the model is functioning as expected by using the onnx library.

Converting CatBoost models to ONNX format

Follow this process to convert your trained model in CatBoost to the ONNX format:

1. Import libraries: Start by importing the essential libraries, such as onnxruntime for running the model, catboost for CatBoost functionalities, and other libraries required for your application.

2. Create or download CatBoost model: You can create a CatBoost model by using your own data set or use models provided by external open source model repositories such as Hugging Face.

3. Convert CatBoost model to ONNX format: To convert the CatBoost model to ONNX format:

   a. Load the CatBoost model: You can load the CatBoost model by using libraries such as pickle:

   catboost_model = pickle.load(file)
   

   b. Export the model: Use the catboost_model.save_model function with format parameter set to onnx to convert the model to ONNX format.

   catboost_model.save_model(
       onnx_model_name,
       format="onnx",
       export_parameters={
           'onnx_domain': 'ai.catboost',
           'onnx_model_version': 1,
           'onnx_doc_string': 'test model for Regressor',
           'onnx_graph_name': 'CatBoostModel_for_Regression'
       }
   )
   

4. Verify the conversion: After converting the model, verify that the model is functioning as expected by using the onnx library.

Converting LightGBM models to ONNX format

Follow this process to convert your trained model in LightGBM to the ONNX format:

1. Import libraries: Start by importing the essential libraries, such as onnxruntime for running the model, lightgbm for LightGBM functionalities, and onnxmltools for conversion and other libraries required for your application.

2. Create or download LightGBM model: You can create a LightGBM model by using your own data set or use models provided by external open source model repositories like Hugging Face.

3. Convert LightGBM model to ONNX format: To convert the LightGBM model to ONNX format:

   a. Load the LightGBM model: You can load the LightGBM model by using libraries such as pickle:

   lgbm_model = pickle.load(file)
   

   b. Export the model: Use the convert_lightgbm function to convert the model to ONNX format.

   from onnxmltools import convert_lightgbm
   from skl2onnx.common.data_types import FloatTensorType
   from onnxmltools.utils import save_model      
   
   initial_types = [("float_input", FloatTensorType([None, X.shape[1]]))]
   onnx_model = convert_lightgbm(model=lgbm_model, initial_types=initial_types)
   onnx_model_filename = "lgbm_model.onnx"
   save_model(onnx_model, onnx_model_filename)
   

   Tip:

   For troubleshooting problems with converting LightGBM models to ONNX format, see Troubleshooting watsonx.ai Runtime.

4. Verify the conversion: After converting the model, verify that the model is functioning as expected by using the onnx library.

Converting XGBoost models to ONNX format

Follow this process to convert your trained model in XGBoost to the ONNX format:

1. Import libraries: Start by importing the essential libraries, such as onnxruntime for running the model, xgboost for XGBoost functionalities, and other libraries required for your application.

2. Create or download XGBoost model: You can create a XGBoost model by using your own data set or use models provided by external open source model repositories like Hugging Face.

3. Convert XGBoost model to ONNX format: To convert the XGBoost model to ONNX format:

   a. Load the XGboost model: You can load the XGBoost model by using libraries such as pickle:

   xgboost_model = pickle.load(file)
   

   b. Export the model: Use the convert_xgboost function to convert the model to ONNX format.

   from onnxmltools import convert_xgboost
   from onnxconverter_common.data_types import FloatTensorType
   from onnxmltools.utils import save_model
   
   initial_types = [("float_input", FloatTensorType([None, X.shape[1]]))]
   onnx_model = convert_xgboost(xgboost_model, initial_types=initial_types)
   onnx_model_filename = "xgboost_onnx_model.onnx"
   save_model(onnx_model, onnx_model_filename)
   

4. Verify the conversion: After converting the model, verify that the model is functioning as expected by using the onnx library.

Converting scikit-learn models to ONNX format

Follow this process to convert your trained model in scikit-learn to the ONNX format:

1. Import libraries: Start by importing the essential libraries, such as onnxruntime for running the model, sklearn for scikit-learn functionalities, skl2onnx for conversion and other libraries required for your application.

2. Create or download scikit-learn model: You can create a scikit-learn model by using your own data set or use models provided by external open source model repositories like Hugging Face.

3. Convert scikit-learn model to ONNX format: To convert the scikit-learn model to ONNX format:

   a. Load the scikit-learn model: You can load the scikit-learn model by using libraries such as pickle:

   sklearn_model = pickle.load(file)
   

   b. Export the model: Use the to_onnx function to convert the model to ONNX format.

   from skl2onnx import to_onnx
   onnx_model = to_onnx(sklearn_model, X, target_opset=19)
   
   with open("sklearn_model.onnx", "wb") as f:
      f.write(onnx_model.SerializeToString())
   

4. Verify the conversion: After converting the model, verify that the model is functioning as expected by using the onnx library.

Deploying models converted to ONNX format

Use the onnxruntime_opset_19 software specification to deploy your machine learning model converted to ONNX format. You must specify the software specification and model type when you store the model to the watsonx.ai Runtime repository. For more information, see Supported software specifications.

To deploy models converted to ONNX format from the user interface, follow these steps:

1. In your deployment space, go to the Assets tab.

2. Find your model in the asset list, click the Menu icon Menu icon, and select Deploy.

3. Select the deployment type for your model. Choose between online and batch deployment options.

4. Enter a name for your deployment and optionally enter a serving name, description, and tags.

   Note:

   • Use the Serving name field to specify a name for your deployment instead of deployment ID.
   • The serving name must be unique within the namespace.
   • The serving name must contain only these characters: [a-z,0-9,_] and must be a maximum 36 characters long.
   • In workflows where your custom foundation model is used periodically, consider assigning your model the same serving name each time you deploy it. This way, after you delete and then re-deploy the model, you can keep using the same endpoint in your code.

5. Select a hardware specification for your model.

6. Select a configuration and a software specification for your model.

7. Click Create.

Testing the model

Follow these steps to test your deployed models converted to ONNX format:

1. In your deployment space, open the Deployments tab and click the deployment name.
2. Click the Test tab to input prompt text and get a response from the deployed asset.
3. Enter test data in one of the following formats, depending on the type of asset that you deployed:
   • Text: Enter text input data to generate a block of text as output.
   • JSON: Enter JSON input data to generate output in JSON format.
4. Click Generate to get results that are based on your prompt.

Sample notebooks

The following sample notebooks demonstrate how to deploy machine learning models converted from PyTorch or TensorFlow to the ONNX format by using the Python client library:

Parent topic: Deploying machine learning assets