0 / 0
Watson Pipelines requirements and limits

Watson Pipelines requirements and limits

Consider these requirements, limits, and best practices when planning a pipeline.

Limits per configuration

Limit on pipeline operations depend on the resources available with your configuration.

Small configuration

A SMALL configuration supports 600 standard nodes (across all active pipelines) or 300 nodes run in a loop. For example:

  • 30 standard pipelines with 20 nodes run in parallel = 600 standard nodes
  • single pipeline containing a loop with 30 iterations and 10 nodes in each iteration = 300 nodes in a loop

Medium configuration

A MEDIUM configuration supports 1200 standard nodes (across all active pipelines) or 600 nodes run in a loop. For example:

  • 30 standard pipelines with 40 nodes run in parallel = 1200 standard nodes
  • single pipeline containing a loop with 60 iterations and 10 nodes in each iteration = 600 nodes in a loop

Large configuration

A LARGE configuration supports 4800 standard nodes (across all active pipelines) or 2400 nodes run in a loop. For example:

  • 80 standard pipelines with 60 nodes run in parallel = 4800 standard nodes
  • 4 pipelines containing a loop with 60 iterations and 10 nodes in each iteration = 2400 nodes in a loop

Single pipeline limits

These limitation apply to a single pipeline, regardless of configuration.

  • Any single pipeline cannot contain more than 120 standard nodes
  • Any pipeline with a loop cannot contain more than 600 nodes across all iterations (for example, 60 iterations - 10 nodes each)

Imported pipelines might require configuration changes

Pipeline assets can be exported as an asset when a project is exported. When you create a new project from an export file, pipelines in the import package are added to your project but might require some configuration updates. Because pipeline assets can be configured with references to many different scopes and dependencies, you might must configure assets after importing them for the pipeline to run successfully.

These are some considerations for working with imported pipeline assets:

  • When you create a project from a project export file that contains a pipeline, the imported project updates the GUID path of the newly created project and local artifacts. For example, paths for notebooks, Data Refinery flows, and data assets are updated.
  • If the pipeline referenced artifacts outside of the project those paths will reference the original scope. For example, if you copy an asset from Space_x with GUID 123, the asset still references the same artifact. You might encounter errors about lack of permissions when you try to run the pipeline. In that case, you might have to manually reconfigure the assets.
  • If a pipeline is imported on the same Cloud Pak for Data cluster but for a different user, you might have to add that user as a collaborator to resolve access problems.
  • If a pipeline is imported to a different Cloud Pak for Data cluster, you might have to manually update some path configurations to successfully run the pipeline.
  • You have to manually update any pipeline parameter that references a hardcoded CPDPath.

Input and output names must follow pipeline parameter naming requirements

Like pipeline parameters, input and output names must be lower snake case with lowercase letters, numbers, and underscores. For example, lower_snake_case_with_numbers_123 is a valid name. The name must begin with a letter. If the name does not comply, you get a 404 error when trying to run the pipeline.

Parent topic: Watson Pipelines

Generative AI search and answer
These answers are generated by a large language model in watsonx.ai based on content from the product documentation. Learn more