Supported Languages

English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese. Users may fine tune the granite-embedding-107m-multilingual model for languages beyond these 12 languages.

Intended use

The model is designed to produce fixed length vector representations for a given text, which can be used for text similarity, retrieval, and search applications.

Usage with Sentence Transformers

The model is compatible with SentenceTransformer library and is very easy to use:

First, install the sentence transformers library.

pip install sentence_transformers

The model can then be used to encode pairs of text and find the similarity between their representations.

from sentence_transformers import SentenceTransformer, util

model_path = "ibm-granite/granite-embedding-107m-multilingual"
# Load the Sentence Transformer model
model = SentenceTransformer(model_path)

input_queries = [
    ' Who made the song My achy breaky heart? ',
    'summit define'
    ]

input_passages = [
    "Achy Breaky Heart is a country song written by Don Von Tress. Originally titled Don't Tell My Heart and performed by The Marcy Brothers in 1991. ",
    "Definition of summit for English Language Learners. : 1 the highest point of a mountain : the top of a mountain. : 2 the highest level. : 3 a meeting or series of meetings between the leaders of two or more governments."
    ]

# encode queries and passages
query_embeddings = model.encode(input_queries)
passage_embeddings = model.encode(input_passages)

# calculate cosine similarity
print(util.cos_sim(query_embeddings, passage_embeddings))

Usage with Huggingface Transformers

This is a simple example of how to use the granite-embedding-107m-multilingual model with the Transformers library and PyTorch.

First, install the required libraries.

pip install transformers torch

The model can then be used to encode pairs of text.

import torch
from transformers import AutoModel, AutoTokenizer

model_path = "ibm-granite/granite-embedding-107m-multilingual"

# Load the model and tokenizer
model = AutoModel.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)
model.eval()

input_queries = [
    ' Who made the song My achy breaky heart? ',
    'summit define'
    ]

# tokenize inputs
tokenized_queries = tokenizer(input_queries, padding=True, truncation=True, return_tensors='pt')

# encode queries
with torch.no_grad():
    # Queries
    model_output = model(**tokenized_queries)
    # Perform pooling. granite-embedding-107m-multilingual uses CLS Pooling
    query_embeddings = model_output[0][:, 0]

# normalize the embeddings
query_embeddings = torch.nn.functional.normalize(query_embeddings, dim=1)

Evaluation

The average performance of the granite-embedding-107m-multilingual model on Multilingual Miracl (across 18 langauges), Mintaka Retrieval (across 8 languages) and MTEB Retrieval for English (across 15 tasks), German (across 4 tasks), Spanish (across 2 tasks), Frenc (across 5 tasks), Japanese (across 2 tasks), Arabic (1 task), Korean (1 task) and Chinese (across 8 tasks) is reported below. The granite-embedding-107m-multilingual model is twice as fast as other models with similar embedding dimensions.

Model Architecture

The granite-embedding-107m-multilingual model is based on an encoder-only XLM-RoBERTa like transformer architecture, trained internally at IBM Research.

Training Data

Overall, the training data consists of four key sources: (1) unsupervised title-body paired data scraped from the web, (2) publicly available paired with permissive, enterprise-friendly license, (3) IBM-internal paired data targetting specific technical domains, and (4) IBM-generated synthetic data. The data is listed below:

Notably, we do not use the popular MS-MARCO retrieval dataset in our training corpus due to its non-commercial license, while other open-source models train on this dataset due to its high quality.

Infrastructure

We train Granite Embedding Models using IBM's computing cluster, Cognitive Compute Cluster, which is outfitted with NVIDIA A100 80gb GPUs. This cluster provides a scalable and efficient infrastructure for training our models over multiple GPUs.

Ethical Considerations and Limitations

The English data used to train the base language model was filtered to remove text containing hate, abuse, and profanity.

Resources

• Learn about the latest updates with Granite: IBM Granite website
• Get started with tutorials, best practices, and prompt engineering advice: IBM Granite Documentation
• Learn about the latest Granite learning resources: IBM Developer