Track a prompt template in an AI use case to capture and share facts about the asset to help you meet governance and compliance goals.
Tracking prompt templates
A prompt template is the saved prompt input for a foundation model. A prompt template can include variables so that it can be run with different options. For example, if you have a prompt that summarizes meeting notes for project-X, you can define a variable so that the same prompt can run for project-Y.
You can add a saved prompt template to an AI use case to track the details for the prompt template. In addition to recording details about the prompt template creation information and source model details, the factsheet tracks information from prompt template evaluations to capture performance metrics. You can evaluate prompt templates before or after you start tracking a prompt template.
Before you begin
Before you can track a prompt template, these conditions must be met.
- Be an administrator or editor for the project that contains the prompt template.
- The prompt template must include at least one variable. For more information, see Building reusable prompts.
Watch this video to see how to track a prompt template in an AI use case.
This video provides a visual method to learn the concepts and tasks in this documentation.
Tracking a prompt template in an AI use case
You can add a prompt template to an AI use case from a project or space.
- Open the project or space that contains the prompt template that you want to govern.
- From the action menu for the asset, click View AI use case.
- If this prompt template is not already part of an AI use case, you are prompted to Track in AI use case. When you start tracking a prompt template, it is locked and you can no longer edit it. To make changes, you must create
a new prompt template.
- Select an existing AI use case or follow the prompts to create a new one.
- Choose an existing approach or create a new approach. An approach represents one facet of a complete solution. Each approach creates a version set for all assets in the same approach.
- Choose a version numbering scheme. All the assets in an approach share a common version. Choose from:
- Experimental if you plan to update frequently.
- Stable if the assets are not changing rapidly.
- Custom if you want to start a new version number. Version numbering must follow a schema of major.minor.patch.
When tracking is enabled, all collaborators for the use case can review details for the prompt template.
Details are captured for each lifecycle stage for a prompt template.
- Develop provides information about how the prompt is defined, including the prompt itself, creation date, foundation model that is used, prompt parameters set, and variables defined.
- Evaluate displays the dimension metrics from evaluating your prompt template.
- Operate provides details that are related to how the prompt template is deployed for productive use.
Requirements for tracking prompt templates
Consider these requirements for managing prompt templates for governance from the user interface.
- You can only create deployments of prompt templates that reference model deployments in the same project or space where you created the model deployment. If you promote a template that references a deployed model to a space, a copy of the model asset is automatically added to the space. You must deploy the model in the space before you deploy the prompt template.
- If you export a prompt template that references a deployed model, the associated model asset is automatically exported with the prompt template. When you import the template into a new container, the model is also imported. The model must be deployed in the new container before you deploy the prompt template.
Viewing the factsheet for a tracked prompt template
Click the name of the prompt template in an AI use case to view the associated factsheet.
The factsheet for a prompt template collects this type of data:
- Governance collects basic information such as the name of the AI use case, the description, and the approach name and version data.
- Foundation model displays the name of the foundation model, the license ID, and the model publisher.
- Prompt template shows the prompt name, ID, prompt input, and variables.
- Prompt parameters collect the configuration options for the prompt template, including the decoding method and stopping criteria.
- Evaluation displays the data from evaluation, including alerts, and metric data from the evaluation. For example, this prompt template shows the metrics data for quality evaluations on the prompt template. One threshold alert
was triggered by the evaluation:
- Validate shows the data for how the prompt template was evaluated, including the data set used for the validation, alerts triggered, and evaluation metric data.
- Attachments shows information about attachments that support the use case.
Moving a prompt template through lifecycle stages
When a prompt template is tracked, you can see details from creating the prompt template, and evaluating performance against appropriate metrics. The next stage in the lifecycle is to _validate the prompt template. This involves testing the prompt template with new data. If you are the prompt engineer who is tasked with validating the asset, follow these steps to validate the prompt template and capture the validation data in the associated factsheet.
- From the project containing the prompt template, export the project to a compressed ZIP file.
- Create a new project and populate it with the exported ZIP file.
- Upload validation data, evaluate the prompt template, and save the results to the validation project.
- From the project, promote the prompt template to a new or existing deployment space that is designated as a Production stage. The stage is assigned when the space is created and cannot be updated, so create a new space if
you do not have a production space available.
- After you promote the prompt template to a deployment space, you can configure continuous monitoring.
- Details from monitoring the prompt template in a production space are displayed in the Operate lifecycle stage of the AI use case.
Learn more
- Follow the tutorial Quick start: Evaluate and track a prompt template to evaluate and track a sample prompt template.
- See Deploying a prompt template for details on preparing a prompt template for production.
- See Evaluating prompt templates for details on evaluating a prompt template for dimensions such as accuracy or to test for the presence of hateful or abusive speech.
Parent topic: Tracking assets in an AI use case