To preview this task, watch the video beginning at 00:16.

The Resource hub includes a sample project that contains sample prompt templates that you can compare in the Evaluation Studio. Follow these steps to create the project based on a sample:

1. From the home screen, click the Create a new project icon .

2. Select Sample.

3. Search for Getting started with watsonx.governance, select that sample project, and click Next.

4. Choose an existing object storage service instance or create a new one.

5. Click Create.

6. Wait for the project import to complete, and then click View new project.

7. Associate a watsonx.ai Runtime service with the project. For more information, see watsonx.ai Runtime.

   1. When the project opens, click the Manage tab, and select the Services and integrations page.

   2. On the IBM services tab, click Associate service.

   3. Select your watsonx.ai Runtime instance. If you don't have a watsonx.ai Runtime service instance provisioned yet, follow these steps:

      1. Click New service.

      2. Select watsonx.ai Runtime.

      3. Click Create.

      4. Select the new service instance from the list.

   4. Click Associate service.

   5. If necessary, click Cancel to return to the Services & Integrations page.

8. Click the Assets tab in the project to see the sample assets.

For more information or to watch a video, see Creating a project. For more information on associated services, see Adding associated services.

To preview this task, watch the video beginning at 01:11.

To compare prompt peformance, you need to create an Evaluation Studio experiment. Follow these steps to create the experiment:

1. From the Assets tab, click New asset > Evaluate and compare prompts.

2. On the Setup page, type Summarization Evaluation experiment for the name.

3. Select a task type. In this case, you want to compare summarization prompt templates, so select Summarization.

4. Click Next to continue to the Prompt templates page.

5. Select the Insurance claim summarization, 2 Insurance claim summarization and 3 Insurance claim summarization prompt templates.

   Notice that all three of these prompt templates include Input variables, which is a requirement for the Evaluation Studio.

6. Click Next to continute to the Metrics page.

7. Expand the Generative AI Quality and Model health sections to review the metrics that will be used in the evaluation.

8. Click Next to continute to the Test data page.

9. Select the test data:

   1. Click Select data from project.

   2. Select Project file > Insurance claim summarization test data.csv.

      The test data that you upload must contain reference output and input columns for each prompt variable. Reference output columns are used to calculate reference-based metrics such as ROUGE and BLEU.

   3. Click Select.

   4. For the Input column, select Insurance_Claim.

   5. For the Reference output column, select Summary.

10. Click Next to continute to the Review and run page.

11. Review the configuration, and click Run evaluation. Evaluations can take a few minutes to complete.

To preview this task, watch the video beginning at 02:26.

Now, you are ready to evaluate and compare the AI assets. Follow these steps to review the results in Evaluation Studio:

1. When your evaluation completes, view the Metric comparison visualizations.

   The charts compare the results for each prompt template that you selected. The visualization display whether scores violate the thresholds for each metric.

2. Click the Records list to select a different metric. For example, select Content Analysis to see the chart updates based on the selected metric.

   

3. Hover over a bar in the chart to see the details.

4. Review the table below the visualization that shows the three prompt templates. Notice that each of the prompts use a different foundation model.

5. To make comparisons, click the Set as reference icon next to a prompt template.

   Setting the reference template highlights columns in the table to show whether other assets are performing better or worse than the asset that you select.

6. Click the Custom ranking icon .

   

   To analyze results, you can also create a custom ranking of metrics across different groups by specifying weight factors and a ranking formula to determine which prompt templates have the best performance. When you create a custom ranking, you can select metrics that are relevant for your ranking and provide a weight factor to them. Click Cancel.

7. To run the evaluations again, click the Adjust settings icon . Use the Evaluation details pane to update the test data or reconfigure the metrics.

8. To edit the experiment, click the Assets icon to remove or add assets to your evaluation to change your comparison.

9. From the table, click the Overflow menu next to a prompt template, and choose View AI Factsheet. Factsheets capture details about the asset for each stage of the AI lifecycle to help you meet governance and compliance goals.

10. Close the AI Factsheet page to return to the Evaluation Studio.

11. From here, you can start tracking a prompt template in an AI use case. From the table, click the Overflow menu next to a prompt template, and choose Track in AI use case.