Planning the implementation workflow for your machine learning solution

Defining an AI use case

An AI use case consists of a set of factsheets that contain lineage, history, and other relevant information about the lifecycle of an AI asset such as a model.

Your organization might require that you track and document your AI solution for transparency or regulatory compliance. However, AI use cases are useful even when they are not required because they provide an integrated way to track progress, decisions, and metrics about your solution.

To create an AI use case, first create an inventory, and then create the use case. Add data scientists, data engineers, and other users who are involved in the creation, testing, or governing of your solution as use case collaborators.

Developing governance workflows

A governance workflow enforces a review and approval process for AI use cases and model use.

Your organization might require one or more of the following types of governance workflows:

• Model risk governance workflows to approve AI use cases, approve foundation model lifecycle events, run risk assessments, or automate performance monitoring of models.
• Regulatory compliance management workflows to process alerts published by regulatory agencies.
• Operational risk management workflows to track model risk along with other operational risks across the enterprise.

To set up a governance workflow, you configure it in the Governance console.

Setting up a project

A project is a collaborative workspace where people work with models and data to fulfill a shared goal.

You need a project to prepare data, run experiments, and build machine learning models.

A project for a machine learning solution typically contains the following types of items that are either explicitly added by data scientists or AI engineers or are created as the result of a process:

• Connection assets to data sources, such as where you store training data.
• Data assets that represent data sets for training models.
• Notebooks that you create or that are generated by processes such as running an AutoAI experiment.
• Experiment and flow assets that you create from running tools, such as AutoAI, SPSS Modeler, or Decision Optimization.
• Jobs that are created by running assets in tools.

You have an automatically created sandbox project. However, you might want to create a project with a name that reflects your goal. You can create a project from the home page or from the navigation menu. Add everyone who you want to work on the solution. You assign a role to each collaborator to control their permissions in the project.

Preparing your data

Data preparation involves providing access to the required data in the format and at the quality level that your solution needs.

You need to provide the training data for your machine learning model. Your training data must include labeled data for the column to predict or classify with a machine learning algorithm. You can also provide unlabeled holdout data set or backtest data sets. You might need to cleanse or shape your data to prepare it for analysis and model training.

You can provide training data in the following ways:

• Upload a file from your local system
• Connect to the data source that contains the data set

You can prepare your data with data preparation tools and then use that data in other tools such as AutoAI or notebooks:

• Upload your data files or connect to tables in your data sources.
• Cleanse, shape, and visualize your data with Data Refinery.
• Generate synthetic tabular data based on production data or a custom data schema with Synthetic Data Generator.

You can prepare your data within the same tool that you train the machine learning model:

• Cleanse, transform, reduce, integrate, and validate your data within SPSS Modeler.
• Call preinstalled open source and IBM libraries or install and call custom libraries in notebooks.

Train your machine learning model

A machine learning model is trained on a set of data to develop algorithms that it can use to analyze and learn from new data.

You can experiment with training your machine learning model by altering conditions in the following ways:

• Testing different algorithms to best fit your data
• Selecting and scaling features to best represent the problem
• Adjusting hyperparameter settings to optimize performance and accuracy

To build a machine learning model, you can experiment with Python code in notebooks. To automate finding the algorithms, transformations, and parameter settings to create the best predictive model, run an Auto AI for machine learning experiment. To explore your data, model outcomes, try different models, and investigate relationships to find useful information, build an SPSS Modeler flow. To solve an optimization business problem, build a Decision Optimization model.

Evaluating your machine learning assets

An evaluation of a machine learning model tests the quality of the model output for the set of metrics that you choose. Some metrics are based on comparing model output against the appropriate output that you provide in the testing data set. How efficiently your model generates responses is also evaluated.

Your organization might require evaluations for regulatory compliance or internal policies. However, evaluations are useful even when they are not required because the metric scores can indicate the quality of your solution and might predict decreased user satisfaction when scores drop.

When you evaluate an AI asset, you can configure the following factors:

• The sample size to test
• Which metrics to include
• The threshold value for each metric

General metrics provide the following types of information about the machine learning model:

• How well the model performs compared to a labeled test set.
• Whether the model produces biased outcomes.
• Changes in accuracy and data.
• How efficiently the model processes transactions.

You view the current results and the results over time. The results of each evaluation are added to the use case for the prompt.

To evaluate a machine learning model, open the Insights dashboard from Watson OpenScale. If you run an AutoAI for machine learning experiment, the candidate models are evaluated and ranked automatically.

Deploying your solution

Deploying an asset makes it available for testing or for productive use by an endpoint. After you create deployments, you can test and manage them, and prepare your assets to deploy into pre-production and production environments.

You create deployments in deployment spaces, which are separate workspaces from projects and to which you can add a different set of collaborators.

To deploy machine learning model assets, you promote the asset to a deployment space and create a deployment that contains an endpoint. You can then call the endpoint from your application. You can create separate deployment spaces for testing, staging, and production deployments to support your ModelOps workflow.

Automating machine learning pipelines

After you train and deploy your machine learning model, you can automate the process of training, deploying, and evaluating the model. You can compare the results of mulitple machine learning models to determine the best one. You can configure automated batch scoring at scale.

You can configure a pipeline to perform the following tasks:

• Add assets to your pipeline or to export pipeline assets
• Create or delete assets, deployments, or deployment spaces
• Add error handling and logic, such as loops
• Set user variables
• Replace or update assets to improve performance
• Run a job to train an experiment, execute a script, or run a data flow
• Specify conditions for running the flow
• Run a script that you write

To design a pipeline, drag nodes onto the Pipelines editor canvas, specify objects and parameters, then run and monitor the pipeline.

Monitoring and maintaining your solution

After you embed your solution into your application and put it into production, you must maintain your solution. You can also monitor model performance. Maintaining your solution can include retraining your machine learning model with recent data. Monitoring your solution evaluates the performance of the model in your production environment.

Your organization might require you to monitor your solution and ensure that performance does not fall below specified thresholds.

To monitor your solution, open the deployment of your solution in the deployment space and activate evaluations. You can use the payload logging endpoint to send scoring requests for fairness and drift evaluations and use the feedback logging endpoint to provide feedback data for quality evaluations.