Adding an external model to the model inventory
With AI Factsheets you can add a model entry for a model that you trained outside of Watson Studio to the model inventory so that you can track the lifecycle details for the model.
Preparing to track external models
External models can be any type that is supported for evaluation by the Watson OpenScale service. For more information, see Supported machine learning engines, frameworks, and models.
These points are an overview of the process for preserving facts for an external model.
- A user with Admin access to the Watson Knowledge Catalog must first set up a Platform assets catalog for registering external models, then enable tracking for external models in the model inventory.
- You can use the API in a model notebook to define an external model asset to the Platform assets catalog.
- You can evaluate the third-party provider model deployments in OpenScale to define an external model asset to the Platform assets catalog.
- Associate the external model asset with a model entry in the model inventory to start preserving the facts.
For details, see:
- Enabling tracking for external models
- Associating an external model asset with a model entry
- Populating the model entry
- Viewing facts for an external model
Enabling tracking for external models
When external models are tracked, facts are preserved for external models that are evaluated with Watson OpenScale or when a notebook model is saved. To enable tracking of external models, you must have Admin access to the Watson Knowledge Catalog for this task.
- Click the Manage tab in the Model inventory. This tab is only available to users with Admin access to the catalog.
- You are prompted to create a Platform Asset catalog if one does not exist. You create the Platform Asset catalog and return to the model inventory to complete the task.
- Click the tile for External model tracking.
Creating the Platform assets catalog
- Complete the details for the catalog.
- Click Create to create the catalog.
- Close the tab to return to the model inventory so you can enable tracking.
Enabling external model tracking
- Click the Manage tab in the model inventory.
- Enable the option for External model tracking on the Manage tab.
Note: You can toggle the external tracking off if you no longer want to allow tracking of external models from OpenScale. Existing facts are preserved but facts for newly added models are not tracked.
Associating an external model asset with a model entry
Automatic external model tracking adds any external models that are evaluated in Watson OpenScale to the Platform assets catalog. When the model is in the Platform assets catalog, you can register it to the model inventory. You can associate an external model asset with a model entry in a model inventory in the following ways:
- Use the API to register the external model asset programmatically from a notebook. The external model asset can then be associated with a model entry in the Platform assets catalog.
- Register an external model from Watson OpenScale and associate it with a model entry in the Platform assets catalog.
Creating an external model asset with the API
- Create a model in a notebook.
- Save the model. For example, you can save to an S3 bucket.
- Use the API to create an external model asset (a representation of the external model) in the Platform assets catalog. For more information on API commands that interact with the model inventory, see the IBM_AIGOV_FACTS_CLIENT documentation.
Registering an external model asset with the Model inventory
- Open the Assets tab in the Platform assets catalog where you want to track the model.
- Select the External model asset that you want to track.
- Return to the Assets tab in the Platform assets catalog and click Track this model.
- Select an existing model entry or create a new one.
- Complete the entry with the model entry details and save the details to the inventory.
Populating the model entry
When facts are saved for an external model asset, they are associated with the pillar that represents their phase in the lifecycle, as follows:
- If the external model asset is created from a notebook without deployment, it displays in the Develop pillar.
- If the external model asset is created from a notebook with deployment, it displays in the Deploy pillar
- When the external model deployment is evaluated in OpenScale, it displays in the Validate or Operate stage, depending on how you classified the machine learning provider for the model.
Example: tracking a Sagemaker model
This sample model, created in Sagemaker, is registered for tracking and moves through the Test, Validate, and Operate phases.
Viewing facts for an external model
Viewing facts for an external model is slightly different from viewing facts for a Watson Machine Learning model. These rules apply:
- The external model facts are always stored in the Platform assets catalog. Click the Assets tab to view external model assets.
- Unlike Watson Machine Learning model entries, which have different fact sheets for models and deployments, fact sheets for external models combine information for the model and deployments on the same page.
- Multiple assets can be created with the same name in the Platform assets catalog. To differentiate them the tags development, pre-production and production are assigned automatically to reflect their state.
Parent topic: Managing an inventory of model assets.