Use watsonx Granite Model Series, Chroma, and LangChain to answer questions (RAG)¶
Disclaimers¶
- Use only Projects and Spaces that are available in watsonx context.
Notebook content¶
This notebook contains the steps and code to demonstrate support of Retrieval Augumented Generation in watsonx.ai. It introduces commands for data retrieval, knowledge base building & querying, and model testing.
Some familiarity with Python is helpful. This notebook uses Python 3.11.
About Retrieval Augmented Generation¶
Retrieval Augmented Generation (RAG) is a versatile pattern that can unlock a number of use cases requiring factual recall of information, such as querying a knowledge base in natural language.
In its simplest form, RAG requires 3 steps:
- Index knowledge base passages (once)
- Retrieve relevant passage(s) from knowledge base (for every user query)
- Generate a response by feeding retrieved passage into a large language model (for every user query)
Contents¶
This notebook contains the following parts:
Set up the environment¶
Before you use the sample code in this notebook, you must perform the following setup tasks:
- Create a Watson Machine Learning (WML) Service instance (a free plan is offered and information about how to create the instance can be found here).
Install and import the dependecies¶
!pip install wget | tail -n 1
!pip install -U "langchain>=0.3,<0.4" | tail -n 1
!pip install -U "ibm_watsonx_ai>=1.1.22" | tail -n 1
!pip install -U "langchain_ibm>=0.3,<0.4" | tail -n 1
!pip install -U "langchain_chroma>=0.1,<0.2" | tail -n 1
import os, getpass
watsonx API connection¶
This cell defines the credentials required to work with watsonx API for Foundation Model inferencing.
Action: Provide the IBM Cloud user API key. For details, see documentation.
from ibm_watsonx_ai import Credentials
credentials = Credentials(
url="https://us-south.ml.cloud.ibm.com",
api_key=getpass.getpass("Please enter your WML api key (hit enter): "),
)
Defining the project id¶
The API requires project id that provides the context for the call. We will obtain the id from the project in which this notebook runs. Otherwise, please provide the project id.
Hint: You can find the project_id
as follows. Open the prompt lab in watsonx.ai. At the very top of the UI, there will be Projects / <project name> /
. Click on the <project name>
link. Then get the project_id
from Project's Manage tab (Project -> Manage -> General -> Details).
try:
project_id = os.environ["PROJECT_ID"]
except KeyError:
project_id = input("Please enter your project_id (hit enter): ")
Create an instance of APIClient with authentication details.
from ibm_watsonx_ai import APIClient
api_client = APIClient(credentials=credentials, project_id=project_id)
import wget
filename = 'state_of_the_union.txt'
url = 'https://raw.github.com/IBM/watson-machine-learning-samples/master/cloud/data/foundation_models/state_of_the_union.txt'
if not os.path.isfile(filename):
wget.download(url, out=filename)
Build up knowledge base¶
The most common approach in RAG is to create dense vector representations of the knowledge base in order to calculate the semantic similarity to a given user query.
In this basic example, we take the State of the Union speech content (filename), split it into chunks, embed it using an open-source embedding model, load it into Chroma, and then query it.
from langchain.document_loaders import TextLoader
from langchain.text_splitter import CharacterTextSplitter
from langchain_chroma import Chroma
loader = TextLoader(filename)
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
texts = text_splitter.split_documents(documents)
The dataset we are using is already split into self-contained passages that can be ingested by Chroma.
Create an embedding function¶
Note that you can feed a custom embedding function to be used by chromadb. The performance of Chroma db may differ depending on the embedding model used. In following example we use watsonx.ai Embedding service. We can check available embedding models using get_embedding_model_specs
api_client.foundation_models.EmbeddingModels.show()
{'SLATE_125M_ENGLISH_RTRVR': 'ibm/slate-125m-english-rtrvr', 'SLATE_125M_ENGLISH_RTRVR_V2': 'ibm/slate-125m-english-rtrvr-v2', 'SLATE_30M_ENGLISH_RTRVR': 'ibm/slate-30m-english-rtrvr', 'SLATE_30M_ENGLISH_RTRVR_V2': 'ibm/slate-30m-english-rtrvr-v2', 'MULTILINGUAL_E5_LARGE': 'intfloat/multilingual-e5-large', 'ALL_MINILM_L12_V2': 'sentence-transformers/all-minilm-l12-v2', 'ALL_MINILM_L6_V2': 'sentence-transformers/all-minilm-l6-v2'}
from langchain_ibm import WatsonxEmbeddings
embeddings = WatsonxEmbeddings(
model_id="ibm/slate-30m-english-rtrvr",
url=credentials["url"],
apikey=credentials["apikey"],
project_id=project_id
)
docsearch = Chroma.from_documents(texts, embeddings)
Compatibility watsonx.ai Embeddings with LangChain¶
LangChain retrievals use embed_documents
and embed_query
under the hood to generate embedding vectors for uploaded documents and user query respectively.
help(WatsonxEmbeddings)
Foundation Models on watsonx.ai
¶
IBM watsonx foundation models are among the list of LLM models supported by Langchain. This example shows how to communicate with Granite Model Series using Langchain.
Defining model¶
You need to specify model_id
that will be used for inferencing:
from ibm_watsonx_ai.foundation_models.utils.enums import ModelTypes
model_id = ModelTypes.GRANITE_13B_CHAT_V2
Defining the model parameters¶
We need to provide a set of model parameters that will influence the result:
from ibm_watsonx_ai.metanames import GenTextParamsMetaNames as GenParams
from ibm_watsonx_ai.foundation_models.utils.enums import DecodingMethods
parameters = {
GenParams.DECODING_METHOD: DecodingMethods.GREEDY,
GenParams.MIN_NEW_TOKENS: 1,
GenParams.MAX_NEW_TOKENS: 100,
GenParams.STOP_SEQUENCES: ["<|endoftext|>"]
}
LangChain CustomLLM wrapper for watsonx model¶
Initialize the WatsonxLLM
class from Langchain with defined parameters and ibm/granite-13b-chat-v2
.
from langchain_ibm import WatsonxLLM
watsonx_granite = WatsonxLLM(
model_id=model_id.value,
url=credentials.get("url"),
apikey=credentials.get("apikey"),
project_id=project_id,
params=parameters
)
Generate a retrieval-augmented response to a question¶
Build the RetrievalQA
(question answering chain) to automate the RAG task.
from langchain.chains import RetrievalQA
qa = RetrievalQA.from_chain_type(llm=watsonx_granite, chain_type="stuff", retriever=docsearch.as_retriever())
Select questions¶
Get questions from the previously loaded test dataset.
query = "What did the president say about Ketanji Brown Jackson"
qa.invoke(query)
{'query': 'What did the president say about Ketanji Brown Jackson', 'result': ' The president said, "One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence." This statement was made in reference to Ketanji Brown Jackson, who was nominated by the president to serve on the United States Supreme Court.'}
Summary and next steps¶
You successfully completed this notebook!.
You learned how to answer question using RAG using watsonx and LangChain.
Check out our Online Documentation for more samples, tutorials, documentation, how-tos, and blog posts.
Copyright © 2023, 2024 IBM. This notebook and its source code are released under the terms of the MIT License.