0 / 0
Adding very large objects to a project's Cloud Object Storage

Adding very large objects to a project's Cloud Object Storage

The amount of data you can load to a project's Cloud Object Storage at any one time depends on where you load the data from. If you are loading the data in the product UI, the limit is 5 GB. To add larger objects to a project's Cloud Object Storage, you can use an API or an FTP client.

Load data in multiple parts by using the Cloud Object Storage API

With the Cloud Object Storage API, you can load data objects as large as 5 GB in a single PUT, and objects as large as 10 TB by loading the data into object storage as a set of parts which can be loaded independently in any order and in parallel. After all of the parts have been loaded, they are presented as a single object in Cloud Object Storage.

You can load files with these formats and mime types in multiple parts:

  • application/xml
  • application/pdf
  • text/plain; charset=utf-8

To load a data object in multiple parts:

  1. Initiate a multipart upload:
curl -X "POST" "https://(endpoint)/(bucket-name)/(object-name)?uploads"
 -H "Authorization: bearer (token)"

The value for bucket-name can be found on your project's General page on the Manage tab. Click Manage in IBM Cloud on the Watson Studio for the endpoint value. The value for the bearer token is retrieved via the IBM Cloud IAM API.

  1. Load the parts by specifying arbitrary sequential part numbers and an UploadId for the object:
curl -X "PUT" "https://(endpoint)/(bucket-name)/(object-name)?partNumber=(sequential-integer)&uploadId=(upload-id)"
 -H "Authorization: bearer (token)"
 -H "Content-Type: (content-type)"

Replacecontent-type with application/xml, application/pdf or text/plain; charset=utf-8.

  1. Complete the multipart load:
  curl -X "POST" "https://(endpoint)/(bucket-name)/(object-name)?uploadId=(upload-id)"
 -H "Authorization: bearer (token)"
 -H "Content-Type: text/plain; charset=utf-8"
 -d \$'<CompleteMultipartUpload>
         <Part>
           <PartNumber>1</PartNumber>
           <ETag>(etag)</ETag>
         </Part>
         <Part>
           <PartNumber>2</PartNumber>
           <ETag>(etag)</ETag>
         </Part>
  1. Add your file to the project as an asset. From the Assets page of your project, click the Import assets button. Then from the Project files page, select the file you uploaded to your Cloud Object Storage and click Import.

Next steps

Learn more

Parent topic: Adding connections to projects

Generative AI search and answer
These answers are generated by a large language model in watsonx.ai based on content from the product documentation. Learn more