With AI Factsheets you can add a model use case for a model that you trained outside of Watson Studio to the model inventory so that you can track the lifecycle details for the model.
Use the governance support provided by AI Factsheets to govern the lifecycle management of external models in these ways:
- Add deployments to the existing external models.
- Move the model from environment to another environment in the life cycle tracking view of the model use case.
- Move the deployment from one environment to another environment using the Python client API and REST API.
- Provide additional model metadata and training data information when an external model is saved to the catalog through Python API and REST API.
Preparing to track external models
External models can be any type that is supported for evaluation by the Watson OpenScale service. For more information, see Supported machine learning engines, frameworks, and models. Additionally, models developed in notebooks are considered external models, so you can use AI Factsheets to govern models that you develop, deploy, and monitor on platforms other than Cloud Pak for Data.
These points are an overview of the process for preserving facts for an external model.
- A user with Admin access to the IBM Knowledge Catalog must first set up a Platform assets catalog for registering external models, then enable tracking for external models in the model inventory.
- The administrator for the Platform access catalog can give users owner permission for external model assets added from Watson OpenScale.
- You can use the API in a model notebook to save an external model asset to a catalog.
- Associate the external model asset with a model use case in the model inventory to start preserving the facts. Along with model metadata, new fields
External model identifierandExternal deployment identifierdescribe how the models and deployments are identified in external systems, for example: AWS or Azure. - You can evaluate third-party provider model deployments in OpenScale to define an external model asset. The destination catalog is established following these rules:
- The external model is created in the Platform assets catalog if its corresponding development-time model already exists in the Platform assets catalog or if there is no development-time model created in any catalog.
- If the corresponding development-time model is created in a catalog (other than the Platform assets catalog) using Python client , then the model will be created in that catalog.
For details, see:
Associating an external model asset with a model use case
Automatic external model tracking adds any external models that are evaluated in Watson OpenScale to the catalog where the development-time model exists. After the model is in the catalog, you can register it to the model inventory. You can associate an external model asset with a model use case in a model inventory in the following ways:
- Use the API to save the external model asset to any catalog programmatically from a notebook. The external model asset can then be associated with a model use case.
- Associate the external model that is created with Watson OpenScale evaluation with a model use case.
Creating an external model asset with the API
- Create a model in a notebook.
- Save the model. For example, you can save to an S3 bucket.
- Use the API to create an external model asset (a representation of the external model) in a catalog. For more information on API commands that interact with the model inventory, see the IBM_AIGOV_FACTS_CLIENT documentation.
Registering an external model asset with the Model inventory
- Open the Assets tab in the catalog where you want to track the model.
- Select the External model asset that you want to track.
- Return to the Assets tab in the catalog and click Track this model.
- Select an existing model use case or create a new one.
- Complete the use case with the model use case details and save the details to the inventory.
Registering an external model from Watson OpenScale
If you are validating an external model in Watson OpenScale, you can associate an external model with an use case to track the lifecycle facts.
- Add an external model to the OpenScale dashboard.
- If you already defined a model use case with the API, the system recognizes the use case association.
- As you create and monitor a deployment, the facts are registered with the associated use case. These facts display in the Validate or Operate stage, depending on how you classified the machine learning provider for the model.
Populating the model use case
When facts are saved for an external model asset, they are associated with the pillar that represents their phase in the lifecycle, as follows:
- If the external model asset is created from a notebook without deployment, it displays in the Develop pillar.
- If the external model asset is created from a notebook with deployment, it displays in the Test pillar.
- When the external model deployment is evaluated in OpenScale, it displays in the Validate or Operate stage, depending on how you classified the machine learning provider for the model.
Example: tracking an Amazon SageMaker model
This sample model, created in Amazon SageMaker, is registered for tracking and moves through the Test, Validate, and Operate phases.
Viewing facts for an external model
Viewing facts for an external model is slightly different from viewing facts for a Watson Machine Learning model. These rules apply:
- Click the Assets tab of the catalog containing the external model assets to view facts.
- Unlike Watson Machine Learning model use cases, which have different fact sheets for models and deployments, fact sheets for external models combine information for the model and deployments on the same page.
- Multiple assets with the same name can be created in a catalog. To differentiate them the tags development, pre-production and production are assigned automatically to reflect their state.
Learn more
Parent topic: Managing an inventory of model assets.