Downloading a custom foundation model and setting up storage
Downloading a custom foundation model and setting up storage
Last updated: Nov 28, 2024
Downloading a custom foundation model and setting up storage
To deploy a custom foundation model for inferencing with watsonx.ai, you must upload the model to a cloud object storage. You can use the bucket in the IBM Cloud Object Storage that is associated with your deployment space or an external cloud
storage.
Downloading models from public repositories
Copy link to section
You can use public model repositories to download foundation models. Public model repositories might require you to set up an account before you download models from them.
For example, you can use Hugging Face, a public model repository, to download custom foundation models for your use case. You must set up a Hugging Face account to download the model from Hugging Face.
Downloading a model
Copy link to section
These steps demonstrate how to download a custom foundation model by using a Hugging Face model. Follow these steps to download a custom foundation model by using the Hugging Face command-line interface:
Install the huggingface-cli package with pip:
pip install -U "huggingface_hub[cli]"
Copy to clipboardCopied to clipboard
Check that the huggingface-cli is correctly set up:
huggingface-cli --help
Copy to clipboardCopied to clipboard
Configure the required environment variables:
export HF_TOKEN="<your Hugging Face token>"export MODEL_NAME="<name of the model>"export MODEL_DIR="<directory to download the model to>"
Copy to clipboardCopied to clipboard
Set up a directory on your local disk to download the model to:
mkdir ${MODEL_DIR}
Copy to clipboardCopied to clipboard
Log in to Hugging Face command-line interface and download the model:
Before you add a model to object storage, you must make sure that your model is compatible with the Text Generation Inference (TGI) standard and is built with a supported model architecture and model type. For more information, see Planning to deploy a custom foundation model.
If your model was fine-tuned with InstructLab, conversion to the safetensors format might not be possible:
Models that were fine-tuned in Linux environment require conversion.
Models that were fine-tuned on a Mac can't be converted.
Models that were fine-tuned and saved to the .gguf format (in any environment) cannot be converted.
To convert a model that was fine-tuned with InstructLab, use the code in This repository to convert your model.
For all the other models, if your model is not in the safetensors format and does not contain the tokenizer.json file, follow these steps to convert your model to the required format. Otherwise, skip to the section
for Setting up cloud object storage and adding the model.
You can upload your model to a bucket in the IBM Cloud Object Storage or any other storage buckets, such as Amazon Simple Storage Service (Amazon S3). Here are some of the cloud object storages offered by IBM:
Uploading the model to IBM Cloud Object Storage by using the IBM Aspera Transfer SDK
Copy link to section
Prerequisites:
Download the IBM Aspera Transfer SDK. Click this link.
Set these environment variables:
Path to IBM Aspera Transfer SDK as the path-to-aspera
Path to the folder with the model as path-to-model-folder
Bucket name to transfer the model to as bucket-name
Your cloud object storage API key as api-key
Your cloud object storage service instance ID as cos-service-instance-id
Your cloud object storage service endpoint as cos-service-endpoint
Use this script to upload your model to IBM Cloud Object Storage by using the IBM Aspera Transfer SDK:
import grpc
import sys
import time, subprocess
import json
import os.path
import os.environ
import transfer_pb2 as transfer_manager
import transfer_pb2_grpc as transfer_manager_grpc
path_to_aspera = os.environ['path-to-aspera']
sys.path.append(f'{path_to_aspera}/connectors/python')
defstart_aspera_daemon():
config_path = f'/{path_to_aspera}/config/asperatransferd.json'
daemon_path = f'/{path_to_aspera}/bin/asperatransferd'# Start the daemon
process = subprocess.Popen([daemon_path, '--config', config_path])
print(f"Started Aspera Transfer SDK daemon with PID {process.pid}")
time.sleep(5) # Increased wait time for the daemon to start properlyreturn process
defrun():
try:
# create a connection to the transfer manager daemon
client = transfer_manager_grpc.TransferServiceStub(grpc.insecure_channel('localhost:55002'))
# create transfer spec
transfer_spec = {
"session_initiation":{
"icos":{
"bucket": os.environ['bucket-name'],
"api_key": os.environ['cos-api-key'],
"ibm_service_instance_id": os.environ['cos-service-instance-id'],
"ibm_service_endpoint": os.environ['cos-service-endpoint'] # example: https://s3.us-south.cloud-object-storage.appdomain.cloud
}
},
"direction": "send",
"title": "COS Upload",
"assets": {
"destination_root": "/model",
"paths": [
{
"source": os.environ['path-to-model-folder']
}
]
}
}
transfer_spec = json.dumps(transfer_spec)
# create a transfer request
transfer_request = transfer_manager.TransferRequest(
transferType=transfer_manager.FILE_REGULAR,
config=transfer_manager.TransferConfig(),
transferSpec=transfer_spec
)
# send start transfer request to transfer manager daemontry:
transfer_response = client.StartTransfer(transfer_request)
transfer_id = transfer_response.transferId
print(f"Transfer started with id {transfer_id}")
except grpc.RpcError as e:
print(f"Error starting transfer: {e.code()}: {e.details()}")
return# monitor transfer statustry:
# Monitor transfer statuswhileTrue:
response = client.MonitorTransfers(
transfer_manager.RegistrationRequest(
filters=[transfer_manager.RegistrationFilter(
transferId=[transfer_id]
)]))
for transfer_info in response:
print(f"Transfer info: {transfer_info}")
# Check transfer status in response
status = transfer_info.status
if status == transfer_manager.FAILED or status == transfer_manager.COMPLETED:
print(f"Transfer finished with status {status}")
return# Wait before polling again
time.sleep(5)
except grpc.RpcError as e:
print(f"Error monitoring transfer: {e.code()}: {e.details()}")
except Exception as e:
print(f"Unexpected error: {str(e)}", file=sys.stderr)
if __name__ == '__main__':
# Start the Aspera Transfer SDK daemon
daemon_process = start_aspera_daemon()
# Run the file transfer
run()
# Optionally, stop the Aspera daemon after transfer
daemon_process.terminate()
print("Aspera Transfer SDK daemon stopped")
Copy to clipboardCopied to clipboardShow more
Uploading the model to IBM Cloud Object Storage by using tools that are provided by third parties
Copy link to section
You can use third-party software to upload your model to IBM Cloud Object Storage.
Follow these example steps to upload your model to IBM Cloud Object Storage by using the Amazon Web Services command-line interface:
Install the Amazon Web Services command-line interface with pip:
pip install awscli
Copy to clipboardCopied to clipboard
Set the required environment variables:
export AWS_ACCESS_KEY_ID="<your access key>"export AWS_SECRET_ACCESS_KEY="<your secret access key>"export ENDPOINT="<s3 endpoint URL>"export BUCKET_NAME="<name of the bucket to upload the model>"
MODEL_FOLDER=${MODEL_NAME//\//-}# The name of the created folder is based on model name.export MODEL_FOLDER=${MODEL_FOLDER//./-}# Just in case, we're removing slashes and dots from it.
Copy to clipboardCopied to clipboard
Add the model to the IBM Cloud Object Storage bucket by using the Amazon Web Services command-line interface:
About cookies on this siteOur websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising.For more information, please review your cookie preferences options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.