Adding very large objects to a project’s Cloud Object Storage

The amount of data you can load to a project’s Cloud Object Storage at any one time depends on where you load the data from. If you are using the Watson Studio app to load the data, the limit is 5 GB. To add larger objects to a project’s Cloud Object Storage, you can use any of the following methods:

Load data in multiple parts by using the Cloud Object Storage API

With the Cloud Object Storage API, you can load data objects as large as 5 GB in a single PUT, and objects as large as 5 TB by loading the data into object storage as a set of parts which can be loaded independently in any order and in parallel. After all of the parts have been loaded, they are presented as a single object in Cloud Object Storage.

You can load files with these formats and mime types in multiple parts:

  • application/xml
  • application/pdf
  • text/plain; charset=utf-8

To load a data object in multiple parts:

  1. Initiate a multipart load:
    curl -X "POST" "https://(endpoint)/(bucket-name)/(object-key)?uploads"
     -H "Authorization: bearer (token)"
    

    The values for bucket-name and token are on the Watson Studio project’s Settings page. Click Services > Data Services and Actions > Manage in IBM Cloud on the Cloud Object Storage for the endpoint value.

  2. Load the parts by specifying arbitrary sequential part numbers and an UploadId for the object:
    curl -X "PUT" "https://(endpoint)/(bucket-name)/(object-key)?partNumber=(sequential-integer)&uploadId=(upload-id)"
     -H "Authorization: bearer (token)"
     -H "Content-Type: (content-type)"
    

    content-type can be application/xml, application/pdf or text/plain; charset=utf-8.

  3. Complete the multipart load:
      curl -X "POST" "https://(endpoint)/(bucket-name)/(object-key)?uploadId=(upload-id)"
     -H "Authorization: bearer (token)"
     -H "Content-Type: text/plain; charset=utf-8"
     -d \$'<CompleteMultipartUpload>
          <Part>
            <PartNumber>1</PartNumber>
            <ETag>(etag)</ETag>
          </Part>
          <Part>
            <PartNumber>2</PartNumber>
            <ETag>(etag)</ETag>
          </Part>
    
  4. Add your file to the project as an asset. From the Assets page of your project, click the Find and add data icon. Then, from the Files pane, click the action menu and select Add as data set.

Next steps

Learn more