When you create an online deployment for a model or function from a deployment space or programmatically, a single copy of the asset is deployed by default. To increase scalability and availability, you can increase the number of copies (replicas) by editing the configuration of the deployment. More copies allow for a larger volume of scoring requests.
Deployments can be scaled in the following ways:
- Update the configuration for a deployment in a deployment space.
- Programmatically, using the watsonx.ai Runtime Python client library, or the watsonx.ai Runtime REST APIs.
Before you begin
You must set up your task credentials by generating an API key. For more information, see Managing task credentials.
Changing the number of copies of an online deployment from a space
- Click the Deployment tab of your deployment space.
- From the action menu for your deployment name, click Edit.
- In the Edit deployment dialog box, change the number of copies and click Save.
Increasing the number of replicas of a deployment programmatically
To view or run a working sample of scaling a deployment programmatically, you can increase the number of replicas in the metadata for a deployment.
Python example
This example uses the Python client to set the number of replicas to 3.
change_meta = {
client.deployments.ConfigurationMetaNames.HARDWARE_SPEC: {
"name":"S",
"num_nodes":3}
}
client.deployments.update(<deployment_id>, change_meta)
The HARDWARE_SPEC value includes a name because the API requires a name or an ID to be provided.
REST API example
curl -k -X PATCH -d '[ { "op": "replace", "path": "/hardware_spec", "value": { "name": "S", "num_nodes": 2 } } ]' <Deployment end-point URL>
You must specify a name for the hardware_spec value, but the argument is not applied for scaling.
Parent topic: Managing predictive deployments