Configuring global objects for Watson Pipelines

Last updated: Oct 09, 2024

Use global objects to create configurable constants to configure your pipeline at run time. Use parameters or user variables in pipelines to specify values at run time, rather than hard coding the values. Unlike pipeline parameters, user variables can be dynamically set during the flow.

Learn about creating:

Pipeline parameters
Parameter sets
User variables

Pipeline parameters

Use pipeline parameters to specify a value at pipeline runtime. For example, if you want a user to enter a deployment space for pipeline output, use a parameter to prompt for the space name to use when the pipeline runs. Specifying the value of the parameter each time that you run the job ensures that you use the correct resources.

Note the following about pipeline parameters:

You can assign a parameter as a node value or assign it for the pipeline job.
A status indicator alerts you if the pipeline parameter is assigned to any node.
You can use the same pipeline parameter for multiple nodes.

Defining a pipeline parameter

Create a pipeline parameter from the configuration panel for a node or from the toolbar for the Pipeline editor.
Enter a name and optional description. Note that the name must be lower snake case with lower-case letters, numbers, and underscores. For example, lower_snake_case_with_numbers_123 is a valid name. The name must begin with a letter. If the name does not comply, you will get a 404 error when trying to run the pipeline.
Assign a parameter type. Depending on the parameter type, you might need to provide more details or assign a default value.
Click Add to list to save the pipeline parameter.

Notes:

You can define an individual parameter, or create a parameter set. Parameters you define individually can also be added to a set.
It is possible to use a pipeline parameter in ways that can produce unsupported results in a pipeline. For example, if you assign a parameter that supplies a generic asset as the value for a Copy asset node, you can select an asset as the parameter value that is not supported by the Copy asset node, resulting in an error.

Parameter types

Parameter types are categorized as:

Basic: including data types to structure input to a pipeline or options for handling the creation of a duplicate space or asset.
CPD Path: for selecting a project, catalog, space, or asset.
Instance: for selecting a machine learning instance or a Cloud Object Storage instance
Other: for specifying details such as creation mode or error policy.

Example of using pipeline types

This is an example of using the parameter type Path.

Create a parameter set called MASTER_PARAMETER_SET.
Create a parameter called file_path and set the type to Path.
Set the value of file_path to mnts/workspace/masterdir.
Drag the node Wait for file onto the canvas and set the File location value to MASTER_PARAMETER_SET.file_path.
Connect the Wait for file with the Run Bash script node so that the latter node runs after the former.
Optional: Test your parameter variable:
1. Add the environment variable parameter to your MASTER_PARAMETER_SET parameter set, for example $FILE_PATH.
2. Paste the following command into the Script code of the Run Bash script:
```
echo File: $FILE_PATH
cat $FILE_PATH
```1. Run the pipeline. You should see the path `mnts/workspace/masterdir` in both of the nodes' execution logs to see they passed successfully.
```

Configuring a node with a pipeline parameter

When you configure a node with a pipeline parameter, you can choose an existing pipeline parameter or create a new one as part of configuring a node.

For example:

Create a pipeline parameter called creationmode and save it to the parameter list.
Configure a Create deployment space node and click to open the configuration panel.
For the Creation mode option, choose Pipeline parameter as the input.
Choose the creationmode pipeline parameter and save the configuration.

When you run the flow, the pipeline parameter is assigned when the space is created.

Parameter sets

Parameter sets are a group of related parameters to use in a pipeline. For example, you could create one set of parameters to use in a test environment and another for use in a production environment.

Parameter sets can be created as a project asset. Parameter sets created in the project are then available for use in pipelines in that project.

Creating a parameter set as a project asset

You can create a parameter set as a reusable project asset to use in pipelines.

Open an existing project or create a project.
Click New task > Collect multiple job parameters with specified values to reuse in jobs from the available tasks.
Assign a name for the set, and specify the details for each parameter in the set, including:
- Name for the parameter
- Data type
- Prompt
- Default value
Optionally create value sets for the parameters in the parameter set. The value sets can be the different values for different contexts. For example, you can create a Test value set with values for a test environment, and a production set for production values.
When you have created all of the parameters, save the parameter set. It will then be available for use in pipelines created in that project.

Adding a parameter set for use in a pipeline

To add a parameter set from a project:

Click the Global objects icon and switch to the Parameter sets tab.
Click Add parameter set to add parameter sets from your project that you want to use in your pipeline.
You can add or remove parameter sets from the list. The parameter sets you specify for use in your pipeline will be available when you assign parameters as input in the pipeline.

Creating a parameter set from the parameters list in your pipeline

You can create a parameter set from the parameters list for your pipeline

Click the Global objects icon and open the Pipeline Parameters.
Select the parameters you want in the set, then click the Save as parameter set icon.
Enter a name and optional description for the set.
Save to add the parameter set for use in your pipeline.

Using a parameter set in a pipeline

To use a parameter set:

From a node property sheet, choose Assign pipeline parameter as an input type.
A list displays all available parameters of the type required for that input. This includes individual parameters, and parameters defined as part of a set. Choose the parameter to assign. Note that the parameter set name precedes the name of the parameter. For example, Parameter_set_name.Parameter_name.
When you run the pipeline, you can select a value set for the corresponding value (if available), assign a value for the parameter, or accept the default value.

Note:

You can use a parameter set in the expression builder using the format param_sets.<MyParamSet.BRParam>. If a parameter set value contains an environment variable, you must use this syntax in the expression builder: param_sets.MyParamSet["$ICU_DATA"].

Attention: If you delete a parameter, ensure that you remove the references to the parameter from your job design. If you do not remove the references, your job might fail.

Editing a parameter set in a job

If you use a parameter set when you define a job, you can choose a value set to populate variables with the values in that set. Note that if you change and save the values, then edit the job and save changes, the parameter set values will reset to the defaults.

User variables

Create user variables to assign values when the flow runs. Unlike pipeline parameters, user variables can be modified during processing.

Defining a user variable

You can create user variables for use in your pipeline. Note that user variables, like parameters, are defined on the global level and are not specific to any node. The initial value for a user variable must be set when you define it and cannot be set dynamically as the result of any node output. Once you define a user variable, you can use the Set user variables node to update it with node output.

To create a user variable:

Create a variable from the configuration panel for an Update variable node or from the toolbar for the Pipeline editor.
Enter a name and optional description. Note that the name must be lower snake case with lower-case letters, numbers, and underscores. For example, lower_snake_case_with_numbers_123 is a valid name. The name must begin with a letter. If the name does not comply, you will get a 404 error when trying to run the pipeline.
Complete the definition of the variable, including choosing a variable type and input type.
Click Add to add the variable to the list. It is now available for use in a node.

Parent topic: Creating a pipeline