0 / 0
Quick start: Prompt a foundation model with the retrieval-augmented generation pattern

Quick start: Prompt a foundation model with the retrieval-augmented generation pattern

Take this tutorial to learn how to use foundation models in IBM watsonx.ai to generate factually accurate output grounded in information in a knowledge base by applying the retrieval-augmented generation pattern. Foundation models can generate output that is factually inaccurate for a variety of reasons. One way to improve the accuracy of generated output is to provide the needed facts as context in your prompt text. This tutorial uses a sample notebook using the retrieval-augmented generation pattern method to improve the accuracy of the generated output.

Required services
Watson Studio
Watson Machine Learning

Your basic workflow includes these tasks:

  1. Open a project. Projects are where you can collaborate with others to work with data.
  2. Add a notebook to your project. You can create your own notebook, or add a sample notebook to your project.
  3. Add and edit code, then run the notebook.
  4. Review the notebook output.

Read about retrieval-augmented generation pattern

You can scale out the technique of including context in your prompts by leveraging information in a knowledge base. The retrieval-augmented generation pattern involves three basic steps:

  • Search for relevant content in your knowledge base
  • Pull the most relevant content into your prompt as context
  • Send the combined prompt text to the model to generate output

Read more about the retrieval-augmented generation pattern

Watch a video about using the retrieval-augmented generation pattern

Watch Video Watch this video to preview the steps in this tutorial. There might be slight differences in the user interface shown in the video. The video is intended to be a companion to the written tutorial.

This video provides a visual method to learn the concepts and tasks in this documentation.


Try a tutorial to prompt a foundation model with the retrieval-augmented generation pattern

In this tutorial, you will complete these tasks:




  • Use the video picture-in-picture

    Tip: Start the video, then as you scroll through the tutorial, the video moves to picture-in-picture mode. Close the video table of contents for the best experience with picture-in-picture. You can use picture-in-picture mode so you can follow the video as you complete the tasks in this tutorial. Click the timestamps for each task to follow along.

    The following animated image shows how to use the video picture-in-picture and table of contents features:

    How to use picture-in-picture and chapters

    Get help in the community

    If you need help with this tutorial, you can ask a question or find an answer in the watsonx.ai Community discussion forum.

    Set up your browser windows

    For the optimal experience completing this tutorial, open Cloud Pak for Data in one browser window, and keep this tutorial page open in another browser window to switch easily between the two applications. Consider arranging the two browser windows side-by-side to make it easier to follow along.

    Side-by-side tutorial and UI

    Tip: If you encounter a guided tour while completing this tutorial in the user interface, click Maybe later.

    Back to the top


  • You need a project to store the sample notebook.

    Watch a video to see how to create a sandbox project and associate a service. Then follow the steps to verify that you have an existing project or create a sandbox project.

    This video provides a visual method to learn the concepts and tasks in this documentation.

    Follow the steps to verify that you have an existing project or create a project.

    1. From the watsonx home screen, scroll to the Projects section. If you see any projects listed, then skip to Associate the Watson Machine Learning service. If you don't see any projects, then follow these steps to create a project.

    2. Click Create a sandbox project. When the project is created, you will see the sandbox in the Projects section.

    3. Open an existing project or the new sandbox project.

    Associate the Watson Machine Learning service with the project

    You will use Watson Machine Learning to prompt the foundation model, so follow these steps to associate your Watson Machine Learning service instance with your project.

    1. In the project, click the Manage tab.

    2. Click the Services & Integrations page.

    3. Check if this project has an associated Watson Machine Learning service. If there is no associated service, then follow these steps:

      1. Click Associate service.

      2. Check the box next to your Watson Machine Learning service instance.

      3. Click Associate.

      4. If necessary, click Cancel to return to the Services & Integrations page.

    For more information or to watch a video, see Creating a project.

    Checkpoint icon Check your progress

    The following image shows the Manage tab with the associated service. You are now ready to add the sample notebook to your project.

    Manage tab in the project


    Back to the top


  • The sample notebook uses a small knowledge base and a simple search component to demonstrate the basic pattern. The scenario used in this notebook is for a company that sells seeds for planting in a garden. The website for an online seed catalog has many articles to help customers plan their garden and ultimately select which seeds to purchase. The new widge is being added to the website to answer customer questions on the contents of the articles.

    Watch this video to see how to add a sample notebook to a project, and then follow the steps to add the notebook to your project.

    This video provides a visual method to learn the concepts and tasks in this documentation.

    1. Access the Simple introduction to retrieval-augmented generation with watsonx.ai in the Resource hub.

    2. Click Add to project.

    3. Select your project from the list, and click Add.

    4. Type the notebook name and description (optional).

    5. Select a runtime environment for this notebook.

    6. Click Create. Wait for the notebook editor to load.

    7. From the menu, click Kernel > Restart & Clear Output, then confirm by clicking Restart and Clear All Outputs to clear the output from the last saved run.


    For more information on associated services, see Adding associated services.

    Checkpoint icon Check your progress

    The following image shows the notebook open in Edit mode. Now you are ready to set up the prerequisites for running the notebook.

    Notebook open in Edit mode


    Back to the top


  • preview tutorial video To preview this task, watch the video beginning at 00:57.

    Before you can run the notebook, you need to set up the environment. Follow these steps to verify the notebook prerequisites:

    1. Scroll to the For IBM watsonx on IBM Cloud section in the notebook to see the two prerequisites to run the notebook.

    2. Under the Create an IBM Cloud API key section, you need to pass your credentials to the Watson Machine Learning API using an API key. If you don't already have a saved API key, then follow these steps to create an API key.

      1. Access the IBM Cloud console API keys page.

      2. Click Create an IBM Cloud API key. If you have any existing API keys, the button may be labelled Create.

      3. Type a name and description.

      4. Click Create.

      5. Copy the API key.

      6. Download the API key for future use.

    3. Review the Associate an instance of the Watson Machine Learning service with the current project section. You completed this prerequisite in Task 1.

    4. Scroll to the Run the cell to provide the IBM Cloud API key section:

      1. Click the Run icon Run icon to run the cell.

      2. Paste the API key, and press Enter.

    5. Under Run the cell to set the credentials for IBM watsonx on IBM Cloud, click the Run icon Run icon to run the cell and set the credentials.

    Checkpoint icon Check your progress

    The following images shows the notebook with the prerequisites completed. Now you are ready to run the notebook and review the output.

    Notebook with the prerequisites completed


    Back to the top


  • preview tutorial video To preview this task, watch the video beginning at 01:03.

    The sample notebook includes information about the retrieval-augmented generation and how you can adapt the notebook for your specific use case. Follow these steps to run the notebook and review the output:

    1. Scroll to the Step 2: Create a Knowledge Base section in the notebook:

      1. Click the Run icon Run icon for each of the three cells in that section.

      2. Review the output for the three cells in the section. The code in these cells sets up the knowledge base as a collection of two articles. These articles were written as samples for watsonx.ai, they are not real articles published anywhere else. The authors and publication dates are fictional.

    2. Scroll to the Step 3: Build a simple search component section in the notebook:

      1. Click the Run icon Run icon for each of the two cells in that section.

      2. Review the output for the two cells in the section. The code in these cells builds a simple search component. Many articles that discuss retrieval-augmented generation assume the retrieval component uses a vector database. However, to perform the general retrieval-augmented generation pattern, any search-and-retrieve method that can reliably return relevant content from the knowledge base will do. In this notebook, the search component is a trivial search function that returns the index of one or the other of the two articles in the knowledge base, based on a simple regular expression match.

    3. Scroll to the Step 4: Craft prompt text section in the notebook:

      1. Click the Run icon Run icon for each of the two cells in that section.

      2. Review the output for the two cells in the section. The code in these cells crafts the prompt text. There is no one, best prompt for any given task. However, models that have been instruction-tuned, such as bigscience/mt0-xxl-13b, google/flan-t5-xxl-11b, or google/flan-ul2-20b, can generally perform this task with a sample prompt. Conservative decoding methods tend towards succinct answers. In the prompt, notice two string placeholders (marked with %s) that will be replaced at generation time:

        • The first placeholder will be replaced with the text of the relevant article from the knowledge base
        • The second placeholder will be replaced with the question to be answered
    4. Scroll to the Step 5: Generate output using the foundation models Python library section in the notebook:

      1. Click the Run icon Run icon for each of the three cells in that section.

      2. Review the output for the three cells in the section. The code in these cells generates output by using the Python library. You can prompt foundation models in watsonx.ai programmatically using the Python library. For more information about the library, see the following topics:

    5. Scroll to the Step 6: Pull everything together to perform retrieval-augmented generation section in the notebook:

      1. Click the Run icon Run icon for each of the two cells in that section. This code pulls everything together to perform retrieval-augmented generation.

      2. Review the output for the first cell in the section. The code in this cell sets up the user input elements.

      3. For the second cell in the section, type a question related to tomatoes or cucumbers to see the answer and the source. For example, Do I use mulch with tomatoes?.

      4. Review the answer to your question.

    Checkpoint icon Check your progress

    The following image shows the completed notebook.

    The completed notebook


    Back to the top

Next steps

  • Watch Video Watch the video beginning at 02:55 to learn about considerations for applying the retrieval-augmented generation pattern to a production solution.
  • Try the Prompt a foundation model tutorial using Prompt Lab.

Additional resources

  • Foundation models

  • Prompt Lab

  • Python library

  • View more videos.

  • Find sample data sets, projects, models, prompts, and notebooks in the Resource hub to gain hands-on experience:

    Notebook icon Notebooks that you can add to your project to get started analyzing data and building models.

    Project icon Projects that you can import containing notebooks, data sets, prompts, and other assets.

    Data set icon Data sets that you can add to your project to refine, analyze, and build models.

    Prompt icon Prompts that you can use in the Prompt Lab to prompt a foundation model.

    Model icon Foundation models that you can use in the Prompt Lab.

Parent topic: Quick start tutorials

Generative AI search and answer
These answers are generated by a large language model in watsonx.ai based on content from the product documentation. Learn more