After your AutoAI RAG experiment completes, view the details of a pattern to understand the composition and performance.
From the pipeline leaderboard, you can view a table with the following information for each of the patterns the experiment generates:
Column
Description
Rank
Rank of placement according to best performance for the optimized metric
Name
Pattern name
Model name
Name of the embedding model used to vectorize and index the documents
Optimize metric
Performance result for the optimized metric selected for the experiment. A RAG pattern can be optimized for Answer faithfulness and Answer correctness
Chunk size
Amount of data retrieved from the indexed documents
Retrieval method
Method (Window or Simple) used to retrieve indexed documents
Vector store distance metric
Metric (Cosine or Euclidean) used to measure how relevant the stored vectors are to the input vectors
Click on a pattern name to review the following configuration details for a pattern. Some of the configuration settings are editable when you create the experiment. For details, see Customizing RAG experiment settings.
Column
Description
Pattern
Pattern name
Score type
How the performance metric is calculated and presented. - Mean: The average score of the metric. - CI High: The upper bound of the confidence interval (CI) for the mean score, showing the highest value that the metric is likely
to achieve. - CI Low: The lower bound of the confidence interval (CI) for the mean score, showing the lowest value that the metric is likely to achieve.
Answer correctness
Correctness of the generated response including both the relevance of the retrieved context and the quality of the generated response.
Answer faithfulness
Accuracy of the generated response to the retrieved text.
Context correctness
Relevancy of the generated response to the input.
Vector store datasource type
Datasource that connects to the vector store.
Vector store distance metric
Metric used to measure how relevant the stored vectors are to the input vector.
Vector store index name
Index used as a data structure for storing and retrieving vector stores.
Vector store operation
Vector store operation for retrieving data. For example, upsert for inserting or updating rows in the database table.
Vector store schema fields
Schema for storing the index of vectored documents.
Vector store schema ID
Unique identifier of the vector store schema.
Vector store schema name
Name of the schema used for structuring the database table
Vector store schema type
Determines the kind of data that can be stored in the schema and the operations that can be carried out on the data. The schema type struct is used for storing structured data.
Chunk overlap
Number of chunks that share common words or phrases.
Chunk size
Amount of data retrieved from the index documents.
Chunk augmentation
How relevant text is retrieved for each chunk of the input after splitting the input into multiple chunks.
Embedding model
Embedding model used to vectorize and index the document collection.
Truncate input tokens
Maximum number of tokens accepted as input.
Truncate strategy
Strategy used to determine how to process retrieved documents to optimize the model performance.
Retrieval method
Method used to retrieve indexed documents: Window or Simple.
Number of chunks
Number of chunks retrieved from the indexed documents.
Context template text
Structured template for the generated response.
Foundation model
Foundation model used in the experiment. Click the preview icon for details about the model.
Decoding method
Process that a model uses to choose the tokens in the generated output.
Maximum new tokens
Maximum number of new tokens that can be generated in addition to the tokens retrieved from indexed documents.
Minimum new tokens
Minimum number of new tokens that can be generated.
Prompt template text
Structure with guidelines for the model to use when retrieving text from indexed documents.
Word to token ratio
Number of words in a text divided by the number of generated tokens.
Sample Q&A
Questions and correct answers provided as test data to measure the performance of the pattern.
About cookies on this siteOur websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising.For more information, please review your cookie preferences options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.