Configuring global objects for Orchestration Pipelines

Last updated: Sep 23, 2024

Use global objects to create configurable constants to configure your pipeline at run time. Use parameters or user variables in pipelines to specify values at run time, rather than hardcoding the values. Unlike pipeline parameters, user variables can be dynamically set during the flow.

Learn about creating:

Pipeline parameters
Parameter sets
User variables

Pipeline parameters

Use pipeline parameters to specify a value at pipeline runtime. For example, if you want a user to enter a deployment space for pipeline output, use a parameter to prompt for the space name to use when the pipeline runs. Specifying the value of the parameter each time that you run the job helps you use the correct resources.

About pipeline parameters:

can be assigned as a node value or assign it for the pipeline job.
can be assigned to any node, and a status indicator alerts you.
can be used for multiple nodes.

Defining a pipeline parameter

Create a pipeline parameter from the node configuration panel from the toolbar.
Enter a name and an optional description. The name must be lower snake case with lowercase letters, numbers, and underscores. For example, lower_snake_case_with_numbers_123 is a valid name. The name must begin with a letter. If the name does not comply, you get a 404 error when you try to run the pipeline.
Assign a parameter type. Depending on the parameter type, you might need to provide more details or assign a default value.
Click Add to list to save the pipeline parameter.

Note:

- You can define an individual parameter, or create a parameter set. Parameters that you define individually can also be added to a set.
- It is possible to use a pipeline parameter in ways that can produce unsupported results in a pipeline. If you assign a parameter with a generic asset as its value, you might select an asset as the parameter value that is unsupported by the Copy asset node.

Parameter types

Parameter types are categorized as:

Basic: including data types to structure input to a pipeline or options for handling the creation of a duplicate space or asset.
Resource: for selecting a project, catalog, space, or asset.
Instance: for selecting a machine learning instance or a Cloud Object Storage instance.
Other: for specifying details, such as creation mode or error policy.

Example of using pipeline types

To create a parameter of the type Path:

Create a parameter set called MASTER_PARAMETER_SET.
Create a parameter called file_path and set the type to Path.
Set the value of file_path to mnts/workspace/masterdir. There are several ways you can set the value:
- Select resource: Browse for the file in your scopes with the UI.
- Enter the value: A valid text for the relative path of the file, for example /mnts/sender.txt.
Drag the node Wait for file onto the canvas and set the File location value to MASTER_PARAMETER_SET.file_path.
Connect the Wait for file with the Run Bash script node so that the latter node runs after the former.
Optional: Test your parameter variable:
1. Add the environment variable parameter to your MASTER_PARAMETER_SET parameter set, for example FILE_PATH.
2. Paste the following command into the Script code of the Run Bash script:
```
echo File: $FILE_PATH
cat $FILE_PATH
```
Run the pipeline. The path mnts/workspace/masterdir is in both of the nodes' execution logs to see they passed successfully.

Configuring a node with a pipeline parameter

When you configure a node with a pipeline parameter, you can choose an existing pipeline parameter or create a new one as part of configuring a node.

For example:

Create a pipeline parameter called creationmode and save it to the parameter list.
Configure a Create deployment space node and click to open the configuration panel.
Choose the Pipeline parameter as the input for the Creation mode option.
Choose the creationmode pipeline parameter and save the configuration.

When you run the flow, the pipeline parameter is assigned when the space is created.

Parameter sets

Parameter sets are a group of related parameters to use in a pipeline. For example, you might create one set of parameters to use in a test environment and another for use in a production environment.

Parameter sets can be created as a project asset. Parameter sets created in the project are then available for use in pipelines in that project.

Creating a parameter set as a project asset

You can create a parameter set as a reusable project asset to use in pipelines.

Open an existing project or create a project.
Click New asset > Define resuable sets of parameters.
Assign a name for the set, and specify the details for each parameter in the set, including:
- Name for the parameter
- Data type
- Prompt
- Default value
Optionally create value sets for the parameters in the parameter set. The value sets can be the different values for different contexts. For example, you can create a Test value set with values for a test environment, and a production set for production values.
Save the parameter set after you create all the parameters, s. It becomes available for use in pipelines that are created in that project.

Adding a parameter set for use in a pipeline

To add a parameter set from a project:

Click the global objects icon and switch to the Parameter sets tab.
Click Add parameter set to add parameter sets from your project that you want to use in your pipeline.
You can add or remove parameter sets from the list. The parameter sets you specify for use in your pipeline becomes available when you assign parameters as input in the pipeline.

Creating a parameter set from the parameters list in your pipeline

You can create a parameter set from the parameters list for your pipeline

Click the global objects icon and open the Pipeline Parameters.
Select the parameters that you want in the set, then click the Save as parameter set icon.
Enter a name and optional description for the set.
Save to add the parameter set for use in your pipeline.

Using a parameter set in a pipeline

To use a parameter set:

Choose Assign pipeline parameter as an input type from a node property sheet.
Choose the parameter to assign. A list displays all available parameters of the type for that input. Available parameters can be individual parameters, and parameters defined as part of a set. The parameter set name precedes the name of the parameter. For example, Parameter_set_name.Parameter_name.
Run the pipeline and select a value set for the corresponding value (if available), assign a value for the parameter, or accept the default value.

Note: You can use a parameter set in the expression builder by using the format `param_sets. `. If a parameter set value contains an environment variable, you must use this syntax in the expression builder: `param_sets.MyParamSet["$ICU_DATA"]`.

Using project default parameter sets

A project can have a parameter set named PROJDEF. This parameter set contains the a global set of properties that can then be referenced from a DataStage flow or Pipelines flow. Because it is a parameter set, you can use the existing parameter set UI to manage these variables.

Be wary that PROJDEFs:

You can reference a PROJDEF parameter set value with the value PROJDEF of the pipeline parameter.
The value PROJDEF is supported for parameters of all types excepted encrypted parameters.
The PROJDEF parameter set's parameters must use the same name as the pipeline parameters that have the PROJDEF value.

To use PROJDEF in a project:

Create the parameter set PROJDEF.
Add any parameter to your PROJDEF parameter set, for example SOME_DATE.
Click Global objects in the toolbar.
Click Add PROJDEF parameter.
Select the parameter from the list.

Attention: If you delete a parameter, make sure that you remove the references to the parameter from your job design. If you do not remove the references, your job might fail.

Editing a parameter set in a job

If you use a parameter set when you define a job, you can choose a value set to populate variables with the values in that set. If you change and save the values, then edit the job and save changes, the parameter set values reset to the defaults.

User variables

Create user variables to assign values when the flow runs. Unlike pipeline parameters, user variables can be modified during processing.

Defining a user variable

You can create user variables for use in your pipeline. User variables, like parameters, are defined on the global level and are not specific to any node. The initial value for a user variable must be set when you define it and cannot be set dynamically as the result of any node output. When you define a user variable, you can use the Set user variables node to update it with node output.

To create a user variable:

Create a variable from the Update variable node configuration panel or from the toolbar.
Enter a name and an optional description. The name must be lower snake case with lowercase letters, numbers, and underscores. For example, lower_snake_case_with_numbers_123 is a valid name. The name must begin with a letter. If the name does not comply, you get a 404 error when you try to run the pipeline.
Complete the definition of the variable, including choosing a variable type and input type.
Click Add to add the variable to the list. It is now available for use in a node.

Set a time zone

You can set the time zone in your configmap.

Open your configmap by running the command kubectl edit -n <namespace> configmap watson-pipelines-config where namespace is your Cloud Pak for Data instance namespace.

Set the value for your time zone with the following code example:

apiVersion: v1
kind: ConfigMap
metadata:
   name: watson-pipelines-config
   namespace: zen
data:
   default_container_env: |
      - name: TZ
        value: America/New_York

Set the value for your time zone in Pipelines CR:

oc patch wspipelines wspipelines-cr -p '{"spec":{"timezone":"America/New_York"}}' --type=merge

Wait for reconciliation to complete with:
```
oc get wspipelines wspipelines-cr
```

Parent topic: Creating a pipeline