Quick start: Evaluate and track a prompt template

To preview this task, watch the video beginning at 00:09.

A model inventory is for storing and reviewing AI use cases. AI use cases collect governance facts for AI assets that your organization tracks. You can view all the AI use cases in an inventory.

To preview this task, watch the video beginning at 00:51.

You need a project to store the prompt template and the evaluation. If you have already created the project, then skip step 1 in this task, and next associate the Watson Machine Learning service with the sample project. Otherwise, follow these steps to create a project based on a sample:

1. Access the Getting started with watsonx governance project in the Resource hub.

   1. Click Create project.

   2. Accept the default values for the project name, and click Create.

   3. Click View new project when the project is successfully created.

2. Associate a Watson Machine Learning service with the project. For more information, see Watson Machine Learning.

   1. When the project opens, click the Manage tab, and select the Services and integrations page.

   2. On the IBM services tab, click Associate service.

   3. Select your Watson Machine Learning instance. If you don't have a Watson Machine Learning service instance provisioned yet, follow these steps:

      1. Click New service.

      2. Select Watson Machine Learning.

      3. Click Create.

      4. Select the new service instance from the list.

   4. Click Associate service.

   5. If necessary, click Cancel to return to the Services & Integrations page.

3. Click the Assets tab in the project to see the sample assets.

For more information or to watch a video, see Creating a project.

For more information on associated services, see Adding associated services.

To preview this task, watch the video beginning at 01:28.

The sample project contains a few prompt templates and CSV files used as test data. Follow these steps to download the test data and evaluate one of the sample prompt templates:

1. Download the test data from the sample project. You need to provide a local file for the test data during evaluation.

   1. Click the Assets tab.
   2. For the Insurance claim summarization test data.csv file, click the Overflow menu , and choose Download.
   3. Save the CSV file locally.

2. Click Insurance claim summarization to open the prompt template in Prompt Lab, and then click Edit.

3. Click the Prompt variables icon .

   Note: To run evaluations, you must create at least one prompt variable.

4. Scroll to the Try section. Notice the {input} variable in the Input field. You must include the prompt variable as input for testing your prompt. A prompt variable is a placeholder keyword that you include in the static text of your prompt at creation time and replace with text dynamically at run time.

5. Click the Evaluate icon .

6. Expand the Generative AI Quality section to see a list of dimensions. The available metrics depend on the task type of the prompt. For example, summarization has different metrics than classification.

7. Click Next.

8. Select the test data:

   1. Click Browse.
   2. Select the Insurance claim summarization test data.csv file that you previously downloaded.
   3. Click Open.
   4. For the Input column, select Insurance_Claim.
   5. For the Reference output column, select Summary.
   6. Click Next.

9. Click Evaluate. Evaluations can take a few minutes to complete. When the evaluation completes, you see the test results on the Evaluate tab. This page shows detailed information about this evaluation run so you can gain insights about your model performance. The summary provides an overview of metric scores and violations of default score thresholds for your prompt template evaluations.

10. Click the AI Factsheet tab.

    1. View the information on each of the sections on the tab.
    2. Click Evaluation > Develop > Test to see the test results again.

To preview this task, watch the video beginning at 02:37.

You can track your prompt template in an AI use case to report the development and test process to your peers. Follow these steps to start tracking the prompt template:

1. From the Navigation Menu , choose Projects > View all projects.
2. Select the Getting started with watsonx governance project.
3. Click the Assets tab.
4. From the Overflow menu for the Insurance claim summarization prompt template, select Go to AI Factsheet. Every AI asset has an AI factsheet; which includes detailed information about how the asset was built, it’s evaluation results across the AI lifecycle, and additional attachments.
5. On the AI Factsheet tab, click the Governance page.
6. Click Track an AI use case.
7. Select the Insurance claims processing AI use case.
8. Click Next.
9. Select an approach. An approach is one facet of the solution to the business problem represented by the AI use case. For example, you might create approaches to track several prompt templates in a use case.
10. Click Next.
11. For the model version, select Experimental.
12. Accept the default value for the version number.
13. Click Next.
14. Review the information, and then click Track asset.
15. When model tracking successfully begins, click the View details icon to open the AI use case.
16. Click the Lifecycle tab to see the prompt template is in the Develop phase. As the prompt template moves through the AI lifecycle, it will move through these phases:
    • Develop phase: AI assets that have been developed in a project environment.
    • Test phase: ML models that have been deployed in a space for testing.
    • Validate phase: AI assets that have been deployed in a space or project for validation.
    • Operate phase: AI assets deployed in a space for operation.

To preview this task, watch the video beginning at 03:27.

Typically, the prompt engineer evaluates the prompt with test data, and the validation engineer validates the prompt. The validation engineer has access to the validation data that prompt engineers might not have. In this case, validation data occurs in a different project. Follow these steps to export the development project and import it as a new validation project to move the asset into the validation phase of the AI lifecycle:

1. From the Navigation Menu , choose Projects > View all projects.

2. Select the Getting started with watsonx governance project.

3. Click the Import/Export icon > Export project.

4. Check the box to select all assets.

5. For the project name, copy and paste the following text, and then click Save.

   validation project.zip
   ```1. Click **Export**.
   

6. When the project export completes, click Back to project.

7. From the Navigation Menu , choose Projects > View all projects.

8. Click New project.

9. Select Local file.

   1. Click Browse.

   2. Select the validation project.zip, and click Open.

   3. For the project name, copy and paste the following text:

      Validation project
      

   4. Click Create.

10. When the project is created, click View new project.1. Follow the same steps as in Step 1 to associate your Watson Machine Learning service with this project.

To preview this task, watch the video beginning at 04:22.

Now you are ready to evaluate the prompt template in this validation project using the same evaluation process as before. Use the same test data set for evaluation. And select the same Input and Output columns as before. Follow these steps to validate the prompt template:

1. Click the Assets tab in the Validation project.
2. Repeat the steps in Task 2 to evaluate the Claims processing summarization prompt template.
3. Click the AI Factsheet tab when the evaluation is complete.
4. View both sets of test results:
   1. Click Evaluation > Develop > Test.
   2. Click Evaluation > Validate > Test.

To preview this task, watch the video beginning at 05:04.