Programmatically inferencing a foundation model by using a prompt template
Last updated: Nov 27, 2024
Programmatically inferencing a foundation model by using a prompt template
You can use a saved prompt template to prompt foundation models in IBM watsonx.ai programmatically.
A prompt template is an asset that you can create to capture a combination of prompt static text, model parameters, and prompt variables that generate the results you want from a specific model so that you can reuse it.
Inference a foundation model with a deployed prompt template
Copy link to section
Sending an inference request to a deployed foundation model involves the following steps:
Configure the APIClient. When you inference a deployed prompt template, you must specify the deployment space for the deployment where the prompt template is hosted.
from ibm_watsonx_ai import APIClient, Credentials
from ibm_watsonx_ai.metanames import GenTextParamsMetaNames
credentials=Credentials(
url=<URL>,
api_key=<API-KEY>,
)
client = APIClient(credentials)
client.set.default_space(space_id)
Copy to clipboardCopied to clipboard
Submit an inference request to the deployed template.
In this example the generate_text method is used and the question prompt variable value is being specified.
generated_response = client.deployments.generate_text(
deployment_id={deployment-id},
params={
GenTextParamsMetaNames.PROMPT_VARIABLES: {
"question": "What plans do you offer?"
}
}
)
Copy to clipboardCopied to clipboard
Deploy and inference a prompt template
Copy link to section
You can use the Python library to deploy a prompt template, and then inference a foundation model by using the deployed prompt template. The following high-level steps are involved. For the complete steps and more options, see the sample notebook.
from ibm_watsonx_ai import APIClient
from ibm_watsonx_ai.foundation_models.prompts import PromptTemplate
from ibm_watsonx_ai.foundation_models.utils.enums import DecodingMethods
from ibm_watsonx_ai.metanames import GenTextParamsMetaNames as GenParams
my_credentials = {
"url": "https://{region}.ml.cloud.ibm.com",
"apikey": {my-IBM-Cloud-API-key},
}
client = APIClient(my_credentials)
prompt_template = PromptTemplate(
name="New prompt",
model_id=client.foundation_models.TextModels.FLAN_T5_XXL,
model_params = {GenParams.DECODING_METHOD: DecodingMethods.SAMPLE},
description="My example",
task_ids=["generation"],
input_variables=["object"],
instruction="Answer the following question",
input_prefix="Human",
output_prefix="Assistant",
input_text="What is {object} and how does it work?",
examples=[
["What is a loan and how does it work?",
"A loan is a debt that is repaid with interest over time."]
]
)
Copy to clipboardCopied to clipboardShow more
Store the prompt template in your project to generate a prompt template ID.
Inference the foundation model. Be sure to specify values for any prompt variables that are defined in the prompt template.
from ibm_watsonx_ai.foundation_models.utils.enums import DecodingMethods
model_inference.generate_text(
params={
"prompt_variables": {"object": "a mortgage"},
GenParams.DECODING_METHOD: DecodingMethods.GREEDY,
GenParams.STOP_SEQUENCES: ['\n\n'],
GenParams.MAX_NEW_TOKENS: 50
}
)
Copy to clipboardCopied to clipboard
Use a prompt template to draft prompt text for an inference request
Copy link to section
Inferencing a foundation model by using a prompt template that is not deployed involves the following steps. For the complete steps and more options, see the sample notebook.
About cookies on this siteOur websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising.For more information, please review your cookie preferences options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.