Inferencing a foundation model programmatically (Python)
Last updated: Feb 21, 2025
Inferencing a foundation model programmatically (Python)
You can prompt foundation models in IBM watsonx.ai programmatically by using the Python library.
After you create a prompt in the Prompt Lab, you can save the prompt as a notebook, and then edit the notebook. Starting from a generated notebook is useful because it handles the initial setup steps, such as getting credentials and the project
ID, for you.
Alternatively, you can work with foundation models directly from a new notebook that you create in watsonx.ai.
For more information, see the Model Inference class of the watsonx.ai Python library.
Sample notebooks
Copy link to section
To find sample notebooks that prompt various foundation models available in watsonx.ai, see Python sample notebooks.
Most of these notebooks are also available from the Resource hub.
For example, the Use watsonx to analyze car rentals reviews is a sample notebook
that you can run to learn the steps involved in inferencing a foundation model in watsonx.ai.
The following sample code shows you how to prompt the flan-t5-xxl-11b model by using different methods so that you can compare the type of output that is generated by each method.
Although only the example that shows how to generate a text stream includes model parameters for prompting, you can specify parameters for any of the methods. The validate=False parameter speeds up the request by skipping the step
of checking that the model ID is valid and references an available foundation model.
Sample Python code for prompting a foundation model
Copy link to section
import json
from ibm_watsonx_ai import APIClient
from ibm_watsonx_ai.foundation_models import ModelInference
from ibm_watsonx_ai.metanames import GenTextParamsMetaNames as GenParams
from ibm_watsonx_ai.foundation_models.utils.enums import DecodingMethods
my_credentials = {
"url": "https://{region}.ml.cloud.ibm.com",
"apikey": {my-IBM-Cloud-API-key},
}
client = APIClient(my_credentials)
gen_parms = {
GenParams.DECODING_METHOD: DecodingMethods.SAMPLE,
GenParams.MAX_NEW_TOKENS: 100
}
model_id = client.foundation_models.TextModels.FLAN_T5_XXL
project_id = {my-project-ID}
space_id = None
verify = False
model = ModelInference(
model_id=model_id,
credentials=my_credentials,
params=gen_parms,
project_id=project_id,
space_id=space_id,
verify=verify,
)
prompt_txt = "In today's sales meeting, we "
gen_parms_override = gen_parms = {
GenParams.DECODING_METHOD: DecodingMethods.SAMPLE,
GenParams.MAX_NEW_TOKENS: 50
}
# GENERATE
generated_response = model.generate(prompt=prompt_txt, params=gen_parms_override)
print("Output from generate() method:")
print(json.dumps(generated_response, indent=2))
# GENERATE TEXT
generated_text_response = model.generate_text(prompt=prompt_txt, params=gen_parms_override)
print("Output from generate_text() method:")
print(generated_text_response)
# GENERATE STREAM
gen_stream_params = {
GenParams.DECODING_METHOD: DecodingMethods.GREEDY,
GenParams.TEMPERATURE: 0.5,
GenParams.MIN_NEW_TOKENS: 10,
GenParams.MAX_NEW_TOKENS: 20
}
print("Output from generate_text_stream() method:")
stream_response = model.generate_text_stream(prompt=prompt_txt, params=gen_stream_params)
for chunk in stream_response:
print(chunk, end='')
Copy to clipboardCopied to clipboardShow more
Note:
To use the sample code, you must replace {region}, {my-IBM-Cloud-API-key}, and {my-project-ID} with valid values for your environment.
will discuss the new product line. Let's start with marketing plans
Copy to clipboardCopied to clipboard
Removing harmful content
Copy link to section
When you submit a prompt to a foundation model, the hate, abuse, and profanity (HAP) filter and the personally-identifiable information (PII) filter are disabled by default. You can enable them and specify the sensitivity of the HAP filter by
using the modifications field in the API.
About cookies on this siteOur websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising.For more information, please review your cookie preferences options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.