This notebook contains the steps and code to demonstrate Sequential Chain using langchain integration with watsonx models.
Some familiarity with Python is helpful. This notebook uses Python 3.10.
The goal of this notebook is to demonstrate how to chain google/flan-ul2
and google/flan-t5-xxl
models to generate a sequence of creating a random question on a given topic and an answer to that question and also to make the user friends with LangChain framework, using simple chain (LLMChain) and the extended chain (SequentialChain) with the WatsonxLLM.
This notebook contains the following parts:
Before you use the sample code in this notebook, you must perform the following setup tasks:
datasets
and dependecies¶!pip install "ibm-watsonx-ai" | tail -n 1
!pip install "pydantic>=1.10.0" | tail -n 1
!pip install "langchain==0.1.8" | tail -n 1
!pip install "langchain_ibm==0.0.1" | tail -n 1
This cell defines the WML credentials required to work with watsonx Foundation Model inferencing.
Action: Provide the IBM Cloud user API key. For details, see documentation.
import getpass
credentials = {
"url": "https://us-south.ml.cloud.ibm.com",
"apikey": getpass.getpass("Please enter your WML api key (hit enter): ")
}
The Foundation Model requires project id that provides the context for the call. We will obtain the id from the project in which this notebook runs. Otherwise, please provide the project id.
import os
try:
project_id = os.environ["PROJECT_ID"]
except KeyError:
project_id = input("Please enter your project_id (hit enter): ")
watsonx.ai
¶All avaliable models are presented under ModelTypes
class.
from ibm_watsonx_ai.foundation_models.utils.enums import ModelTypes
print([model.name for model in ModelTypes])
['FLAN_T5_XXL', 'FLAN_UL2', 'MT0_XXL', 'GPT_NEOX', 'MPT_7B_INSTRUCT2', 'STARCODER', 'LLAMA_2_70B_CHAT', 'LLAMA_2_13B_CHAT', 'GRANITE_13B_INSTRUCT', 'GRANITE_13B_CHAT', 'FLAN_T5_XL', 'GRANITE_13B_CHAT_V2', 'GRANITE_13B_INSTRUCT_V2', 'ELYZA_JAPANESE_LLAMA_2_7B_INSTRUCT']
You need to specify model_id
's that will be used for inferencing:
model_id_1 = ModelTypes.FLAN_UL2.value
model_id_2 = ModelTypes.FLAN_T5_XXL.value
You might need to adjust model parameters
for different models or tasks, to do so please refer to documentation under GenTextParamsMetaNames
class.
Action: If there are any complications, refer to the documentation.
from ibm_watsonx_ai.metanames import GenTextParamsMetaNames as GenParams
from ibm_watsonx_ai.foundation_models.utils.enums import DecodingMethods
parameters = {
GenParams.DECODING_METHOD: DecodingMethods.SAMPLE.value,
GenParams.MAX_NEW_TOKENS: 100,
GenParams.MIN_NEW_TOKENS: 1,
GenParams.TEMPERATURE: 0.5,
GenParams.TOP_K: 50,
GenParams.TOP_P: 1
}
WatsonxLLM
is a wrapper around watsonx.ai models that provide chain integration around the models.
Action: For more details about CustomLLM
check the LangChain documentation
WatsonxLLM
class.¶from langchain_ibm import WatsonxLLM
flan_ul2_llm = WatsonxLLM(
model_id=model_id_1,
url=credentials["url"],
apikey=credentials["apikey"],
project_id=project_id,
params=parameters
)
flan_t5_llm = WatsonxLLM(
model_id=model_id_2,
url=credentials["url"],
apikey=credentials["apikey"],
project_id=project_id
)
You can print all set data about the WatsonxLLM object using the dict()
method.
flan_ul2_llm.dict()
{'model_id': 'google/flan-ul2', 'deployment_id': '', 'params': {'decoding_method': 'sample', 'max_new_tokens': 100, 'min_new_tokens': 1, 'temperature': 0.5, 'top_k': 50, 'top_p': 1}, 'project_id': '00f1747d-e5ba-4ce6-bdb7-409e8486e18a', 'space_id': '', '_type': 'IBM watsonx.ai'}
The simplest type of sequential chain is called a SequentialChain
, in which each step has a single input and output and the output of one step serves as the input for the following step.
The experiment will consist in generating a random question about any topic and answer the following question.
An object called PromptTemplate
assists in generating prompts using a combination of user input, additional non-static data, and a fixed template string.
In our case we would like to create two PromptTemplate
objects which will be responsible for creating a random question and answering it.
from langchain import PromptTemplate
prompt_1 = PromptTemplate(
input_variables=["topic"],
template="Generate a random question about {topic}: Question: "
)
prompt_2 = PromptTemplate(
input_variables=["question"],
template="Answer the following question: {question}",
)
We would like to add functionality around language models using LLMChain
chain.
prompt_to_flan_ul2
chain formats the prompt template whose task is to generate random question, passes the formatted string to LLM and returns the LLM output.
from langchain.chains import LLMChain
prompt_to_flan_ul2 = LLMChain(llm=flan_ul2_llm, prompt=prompt_1, output_key='question')
flan_to_t5
chain formats the prompt template whose task is to answer the question we got from prompt_to_flan_ul2
chain, passes the formatted string to LLM and returns the LLM output.
flan_to_t5 = LLMChain(llm=flan_t5_llm, prompt=prompt_2, output_key='answer')
This is the overall chain where we run prompt_to_flan_ul2
and flan_to_t5
chains in sequence.
from langchain.chains import SequentialChain
qa = SequentialChain(chains=[prompt_to_flan_ul2, flan_to_t5], input_variables=["topic"], output_variables=['question', 'answer'], verbose=True)
Generate random question and answer to topic.
qa.invoke({"topic": "life"})
> Entering new SequentialChain chain... > Finished chain.
{'topic': 'life', 'question': 'What is the main reason a person would need a DNA test?', 'answer': 'to prove paternity'}
Let's wrap the chain code within python function that can be used as a content of inference endpoint.
Let's wrap the above chain code into function.
ai_params = {
"url": credentials["url"],
"apikey": credentials["apikey"],
"project_id": project_id,
"generation_parameters": parameters
}
def chain_text_generator(params=ai_params):
from ibm_watsonx_ai.foundation_models.utils.enums import ModelTypes
from langchain import PromptTemplate
from langchain.chains import LLMChain, SequentialChain
from langchain_ibm import WatsonxLLM
url = params["url"]
apikey = params["apikey"]
project_id = params['project_id']
parameters = params['generation_parameters']
flan_ul2_llm = WatsonxLLM(model_id=ModelTypes.FLAN_UL2.value, url=url, apikey=apikey, project_id=project_id, params=parameters)
flan_t5_llm = WatsonxLLM(model_id=ModelTypes.FLAN_T5_XXL.value, url=url, apikey=apikey, project_id=project_id)
prompt_1 = PromptTemplate(input_variables=["topic"], template="Generate a random question about {topic}: Question: ")
prompt_2 = PromptTemplate(input_variables=["question"], template="Answer the following question: {question}")
prompt_to_flan_ul2 = LLMChain(llm=flan_ul2_llm, prompt=prompt_1, output_key='question')
flan_to_t5 = LLMChain(llm=flan_t5_llm, prompt=prompt_2, output_key='answer')
chain = SequentialChain(chains=[prompt_to_flan_ul2, flan_to_t5], input_variables=["topic"], output_variables=['question', 'answer'])
def score(payload):
"""Generates question based on provided topic and returns the answer."""
answer = chain.invoke({"topic": payload["input_data"][0]['values'][0][0]})
return {'predictions': [{'fields': ['topic', 'question', 'answer'], 'values': [answer['topic'], answer['question'], answer['answer']]}]}
return score
It is good practice to validate the code locally first.
sample_payload = {
"input_data": [
{
"fields": ["topic"],
"values": [["life"]]
}
]
}
inference = chain_text_generator()
inference(sample_payload)
{'predictions': [{'fields': ['topic', 'question', 'answer'], 'values': ['life', 'What is the most common type of bacteria?', 'gram positive']}]}
Create new software specification based on default Python 3.10 environment extended by langchain package.
config_yml =\
"""
name: python310
channels:
- empty
dependencies:
- pip:
- pydantic>=1.10.0
- langchain==0.1.8
- ibm-watsonx-ai
- langchain_ibm==0.0.1
prefix: /opt/anaconda3/envs/python310
"""
with open("config.yaml", "w", encoding="utf-8") as f:
f.write(config_yml)
space_id = input("Please enter your space_id (hit enter): ")
Now you need to store new package extension with APIClient.
from ibm_watsonx_ai import APIClient
client = APIClient(credentials)
client.set.default_space(space_id)
base_sw_spec_uid = client.software_specifications.get_uid_by_name("runtime-23.1-py3.10")
meta_prop_pkg_extn = {
client.package_extensions.ConfigurationMetaNames.NAME: "langchain watsonx.ai env",
client.package_extensions.ConfigurationMetaNames.DESCRIPTION: "Environment with langchain",
client.package_extensions.ConfigurationMetaNames.TYPE: "conda_yml"
}
pkg_extn_details = client.package_extensions.store(meta_props=meta_prop_pkg_extn, file_path="config.yaml")
pkg_extn_uid = client.package_extensions.get_uid(pkg_extn_details)
pkg_extn_url = client.package_extensions.get_href(pkg_extn_details)
Creating package extensions SUCCESS
Create new software specification and add created package extension to it.
meta_prop_sw_spec = {
client.software_specifications.ConfigurationMetaNames.NAME: "langchain watsonx.ai custom software specification",
client.software_specifications.ConfigurationMetaNames.DESCRIPTION: "Software specification for statsmodels",
client.software_specifications.ConfigurationMetaNames.BASE_SOFTWARE_SPECIFICATION: {"guid": base_sw_spec_uid}
}
sw_spec_details = client.software_specifications.store(meta_props=meta_prop_sw_spec)
sw_spec_uid = client.software_specifications.get_uid(sw_spec_details)
client.software_specifications.add_package_extension(sw_spec_uid, pkg_extn_uid)
SUCCESS
'SUCCESS'
meta_props = {
client.repository.FunctionMetaNames.NAME: "SequenceChain LLM function",
client.repository.FunctionMetaNames.SOFTWARE_SPEC_UID: sw_spec_uid
}
function_details = client.repository.store_function(meta_props=meta_props, function=chain_text_generator)
function_id = client.repository.get_function_id(function_details)
metadata = {
client.deployments.ConfigurationMetaNames.NAME: "Deployment of LLMs chain function",
client.deployments.ConfigurationMetaNames.ONLINE: {}
}
function_deployment = client.deployments.create(function_id, meta_props=metadata)
####################################################################################### Synchronous deployment creation for uid: '284e9dff-4ddd-4f42-983a-f7b947633fa9' started ####################################################################################### initializing Note: online_url and serving_urls are deprecated and will be removed in a future release. Use inference instead. ...................... ready ------------------------------------------------------------------------------------------------ Successfully finished deployment creation, deployment_uid='d996c3ff-e26d-4b8c-922c-8a9d12a6c793' ------------------------------------------------------------------------------------------------
deployment_id = client.deployments.get_id(function_deployment)
client.deployments.score(deployment_id, sample_payload)
{'predictions': [{'fields': ['topic', 'question', 'answer'], 'values': ['life', 'In what state is the highest concentration of volcanoes in the United States?', 'hawaii']}]}
You successfully completed this notebook!.
You learned how to use Sequential Chain using custom llm WastonxLLM.
Check out our Online Documentation for more samples, tutorials, documentation, how-tos, and blog posts.
Lukasz Cmielowski, PhD, Senior Technical Staff Member at Watson Machine Learninig.
Mateusz Szewczyk, Software Engineer at Watson Machine Learning.
Copyright © 2024 IBM. This notebook and its source code are released under the terms of the MIT License.