Set up the state

The ibm-watsonx-gov Python SDK provides a pydantic based state class that you can extend:

from ibm_watsonx_gov.entities.state import EvaluationState

class AppState(EvaluationState):
    pass

Set up the evaluator

To evaluate agentic AI applications, you must instantiate the AgenticEvaluation class to define evaluators to compute different metrics:

from ibm_watsonx_gov.evaluate.agentic_evaluation import AgenticEvaluation

evaluator = AgenticEvaluation()

Add your evaluators

Compute the context relevance metric by defining the retrieval_node tool and decorate it with the evaluate_context_relevance evaluator tool:

@evaluator.evaluate_context_relevance
def retrieval_node(state: AppState, config: RunnableConfig):
    # do something
    pass

You can also stack evaluators to compute multiple metrics with a tool. The following example shows the generate_node tool decorated with the evaluate_faithfulness and evaluate_answer_similarity tools to compute answer quality metrics:

@evaluator.evaluate_faithfulness
@evaluator.evaluate_answer_similarity
def generate_node(state: AppState, config: RunnableConfig):
    # do something
    pass

Make an invocation

When you invoke an application for a row of data, a record_id key is added to the inputs to track individual rows and associate metrics with each row:

result = rag_app.invoke({"input_text": "What is concept drift?", "ground_truth": "Concept drift occurs when the statistical properties of the target variable change over time, causing a machine learning model’s predictions to become less accurate.", "record_id": "12"})
evaluator.get_metrics_df()

The invocation generates a result as shown in the following example:

Invoke the graph on multiple rows

To complete batch invocation, you can define a dataframe with questions and ground truths for those questions:

import pandas as pd

question_bank_df = pd.read_csv("https://raw.githubusercontent.com/IBM/ibm-watsonx-gov/refs/heads/samples/notebooks/data/agentic/medium_question_bank.csv")
question_bank_df["record_id"] = question_bank_df.index.astype(str)
result = rag_app.batch(inputs=question_bank_df.to_dict("records"))
evaluator.get_metrics_df()

The dataframe index is used as a record_id to uniquely indentify each row.

The invocation generates a result as shown in the following example:

For more information, see the sample notebook.

Parent topic: Metrics computation using Python SDK