Building agent-driven workflows with the chat API

Last updated: Dec 06, 2024

Use the watsonx.ai chat API with foundation models that support tool calling to build agent-driven applications.

Agentic applications allow a foundation model to function as an agent that controls the flow of interaction with the user. You define the parameters of the interaction, including the tools that the foundation model can use, but you allow the foundation model to decide the next best step based on the current state of the interaction. Tool calling is sometimes referred to as function calling.

In a real-world workflow, you might want to always call one tool at the start of every interaction. Later in the exchange, you might want to use different prompts based on how the user answers follow-up questions about a task. Or you might want to call various third-party tools that are needed to complete the workflow. You can define these options, and then let the foundation model pick the prompt to return or tool to use when the time is right.

For example, a restaurant reservation app might need to collect the following information, and then create a reservation:

Location
Number of guests
Date and time

The order in which the information is collected might matter only in some locations, where the number of guests allowed per table is limited, or where more seating is available outside in fine weather. You can account for these conditions in the code, and let the foundation model determine when to call extra tools, such as a third-party weather app, before the agent completes a reservation request.

Supported foundation models

When you build an agentic workflow, choose a foundation model that meets the following requirements:

Handles chat tasks
Supports tool calling
Can choose the next action

Foundation models with the following model IDs can be used to call various tools in the context of a conversational workflow with the chat API:

ibm/granite-3-2b-instruct
ibm/granite-3-8b-instruct
meta-llama/llama-3-1-70b-instruct
meta-llama/llama-3-1-8b-instruct
meta-llama/llama-3-2-11b-vision-instruct
meta-llama/llama-3-2-1b-instruct
meta-llama/llama-3-2-90b-vision-instruct
meta-llama/llama-3-3-70b-instruct
meta-llama/llama-3-405b-instruct
mistralai/mistral-large

To programmatically get a list of foundation models that support tool calling from the chat API, specify the filters=task_function_calling parameter when you submit a List the available foundation models method request.

See the API reference.

For example:

curl -X GET \
  'https://{region}.ml.cloud.ibm.com/ml/v1/foundation_model_specs?version=2024-10-10&filters=task_function_calling'

You can inference these foundation models from watsonx.ai or from third-party tools, such as CrewAI. CrewAI is a Python framework for building agentic applications. For more information, see the CrewAI documentation.

Sample notebooks

The following sample notebooks are available:

Use the REST API to build a chat that calls third-party tools: Use watsonx, and mistral-large to make simple chat conversation and tool calls
Use the Python SDK to have a conversation among various user types, such as system prompt, assistant, and user, and then call various tools to do math equations based on user input. See Use watsonx, and mistralai/mistral-large to make simple chat conversation and tool calls
Use the Python SDK to call tools that get information to include in a conversation, including by using LangChain to write and implement calculator functions, and illustrating the workflow that is followed with LangGraph. See Use watsonx, and mistralai/mistral-large with support for tools to perform simple calculations.

API details

For API method details, see the API reference documentation.

Example tool-calling request

The following example defines two tools, a function for addition and a function for multiplication. The example submits user input to the foundation model and lets the model choose which tool to use to answer the question.

In the following example, replace the {url} variable with the right value for your instance, such as us-south.ml.cloud.ibm.com. Add your own bearer token and project ID.

curl -X POST \
  'https://{url}/ml/v1/text/chat?version=2024-10-08' \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer eyJraWQiOi...'
  --data '{
      "model_id": "mistralai/mistral-large",
      "project_id": "4947c695-a374-428c-acca-332c1a1dc9e9",
      "messages": [
        {
          "role": "user",
          "content": [
            {
              "type": "text",
              "text": "What is 2 plus 4?"
            }
          ]
        }
      ],
      "tools": [
        {
          "type": "function",
          "function": {
            "name": "add",
            "description": "Adds the values a and b to get a sum.",
            "parameters": {
              "type": "object",
              "properties": {
                "a": {
                  "description": "A number value",
                  "type": "float"
                },
                "b": {
                  "description": "A number value",
                  "type": "float"
                }
              },
              "required": [
                "a",
                "b"
              ]
            }
          }
        },
        {
          "type": "function",
          "function": {
          "name": "multiply",
            "description": "Multiplies the values a and b.",
            "parameters": {
              "type": "object",
              "properties": {
                "a": {
                  "description": "A number value",
                  "type": "float"
                },
                "b": {
                  "description": "A number value",
                  "type": "float"
                }
              },
              "required": [
                "a",
                "b"
              ]
            }
          }
        }
      ],
      "tool_choice_option": "auto",
      "max_tokens": 300,
      "time_limit": 1000
      }'

The sample output shows that the model, mistral-large in this case, is able to choose the correct tool to use for the task, the add function.

{
  "id": "chat-a00942a130e84f83bc0090c38c2f419f",
  "model_id": "mistralai/mistral-large",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "tool_calls": [
          {
            "id": "chatcmpl-tool-77cbe4e94d88489383a0c6ed1b644674",
            "type": "function",
            "function": {
              "name": "add",
              "arguments": "{\"a\": 2, \"b\": 4}"
            }
          }
        ]
      },
      "finish_reason": "tool_calls"
    }
  ],
  "created": 1728413314,
  "model_version": "2.0.0",
  "created_at": "2024-10-08T18:48:34.818Z",
  "usage": {
    "completion_tokens": 25,
    "prompt_tokens": 189,
    "total_tokens": 214
  },
  "system": {
    "warnings": [
      {
        "message": "This model is a Non-IBM Product governed by a third-party license that may impose use restrictions and other obligations. By using this model you agree to its terms as identified in the following URL.",
        "id": "disclaimer_warning",
        "more_info": "https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models.html?context=wx"
      }
    ]
  }
}

Example of a tool-calling request with granite-3-2b-instruct

The following request defines two tools, a function for addition and a function for multiplication, and asks the granite-3-2b-instruct foundation model which tool to use.

The Granite models can call tools better when you provide a system prompt with the request. The following system prompt is used:

You are a helpful assistant with access to the following function calls. Your task is to produce a sequence of function calls necessary to generate response to the user utterance. Use the following function calls as required. The tool calling output MUST adhere to the following JSON format: <|tool_call|>[{"name": "func_name1", "arguments": {"argument1": "value1", "argument2": "value2"}},... (more tool calls as required)]. If no function call is needed, please directly reply to the user message.

In the following example, replace the {url} variable with the right value for your instance, such as us-south.ml.cloud.ibm.com. Add your own bearer token and project ID.

curl -X POST \
  'https://{url}/ml/v1/text/chat?version=2024-10-08' \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer eyJraWQiOi...'

The body of the request contains the following JSON snippet:

{
  "model_id": "ibm/granite-3-2b-instruct",
  "project_id": "4947c695-a374-428c-acca-332c1a1dc9e9",
  "messages": [
    {
      "role":"system",
      "content":[
        {
          "type":"text",
          "text":"<|start_of_role|>system<|end_of_role|>You are a helpful assistant with access to the following function calls. Your task is to produce a sequence of function calls necessary to generate response to the user utterance. Use the following function calls as required. The tool calling output MUST adhere to the following JSON format: <|tool_call|>[{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]. If no function call is needed, please directly reply to the user message.<|end_of_text|>"
      }
      ]
    },
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What is 2 plus 4?"
        }
      ]
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "add",
        "description": "Adds the values a and b to get a sum.",
        "parameters": {
          "type": "object",
          "properties": {
            "a": {
              "description": "A number value",
              "type": "float"
            },
            "b": {
              "description": "A number value",
               "type": "float"
            }
          },
          "required": [
            "a",
            "b"
          ]
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "multiply",
        "description": "Multiplies the values a and b.",
        "parameters": {
          "type": "object",
          "properties": {
            "a": {
              "description": "A number value",
              "type": "float"
            },
            "b": {
              "description": "A number value",
               "type": "float"
            }
          },
          "required": [
            "a",
            "b"
          ]
        }
      }
    }
  ],
  "tool_choice_option": "auto",
  "max_tokens": 300,
  "time_limit": 10000
}

The granite-3-2b-instruct foundation model is able to choose the correct tool to answer the query.

The response looks as follows:

{
  "id": "chat-9eb3284f12c94256a3271180bdf6b6d2",
  "model_id": "ibm/granite-3-2b-instruct",
  "model": "ibm/granite-3-2b-instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "tool_calls": [
          {
            "id": "chatcmpl-tool-6cd1758c5caf4f058c63188a7c7b78fb",
            "type": "function",
            "function": {
              "name": "add",
              "arguments": "{\"a\": 2, \"b\": 4}"
            }
          }
        ]
      },
      "finish_reason": "tool_calls"
    }
  ],
  "created": 1733504510,
  "model_version": "1.0.0",
  "created_at": "2024-12-06T17:01:50.321Z",
  "usage": {
    "completion_tokens": 23,
    "prompt_tokens": 383,
    "total_tokens": 406
  }
}

Learn more

Parent topic: Coding generative AI solutions