Configuring pipeline nodes

Input parameters

Output parameters

Input parameters

Output parameters

Notes:

• If you export a project that contains a notebook, the latest version of the notebook is included in the export file. If the Pipeline with the Run notebook job node was configured to use a different notebook version other than the latest version, the exported Pipeline is automatically reconfigured to use the latest version when imported. This might produce unexpected results or require some reconfiguration after the import.
• If assets are self-contained in the exported project, they are retained when you import a new project. Otherwise, some configuration might be required following an import of exported assets.

Input parameters

Notes: After you import a file, paths and references to the imported assets are updated, following these rules:

• References to assets from the exported project or space are updated in the new project or space after the import.
• If assets from the exported project refer to external assets (included in a different project), the reference to the external asset will persist after the import.
• If the external asset no longer exists, the parameter is replaced with an empty value and you must reconfigure the field to point to a valid asset.

Input parameters

Output parameters

Input parameters

Output parameters

Input parameters

Output parameters

Input parameters

Output parameters

Input parameters

Output parameters

Input parameters

Output parameters

Input parameters

Output parameters

Loops in parallel

Add a parallel looping construct to the pipeline. A parallel loop runs the iterating nodes independently and possibly simultaneously.

For example, to train a machine learning model with a set of hyperparameters to find the best performer, you can use a loop to iterate over a list of hyperparameters to train the notebook variations in parallel. The results can be compared later in the flow to find the best notebook. To see limits on the number of loops you can run simultaneously, see Limitations.

In the following example, a Run Bash script node searches for and retrieves notebooks that match specified criteria. A Run DataStage job node retrieves data from a Git repository. When input from each node is available, the loop process begins, running each notebook retrieved by the search and processing the data retrieved from the Git repository.

Click Expand to add nodes or the outgoing icon on the node to view the full loop process. As the notebooks run, any errors in the notebook are captured in a condition called Poor quality. The condition triggers a Bash script to increment a user variable that is named Increase error count. When the value of the Increase error count variable meets the specified threshold, the loop is terminated.

Since the flow is executed in parallel for each notebook, it returns results faster than a sequential loop.

Loops in sequence

Add a sequential loop construct to the pipeline. Loops can iterate over a numeric range, a list, or text with a delimiter.

A use case for sequential loops is if you want to try an operation 3 time before you determine whether an operation failed.

Terminate loop

In a parallel or sequential loop process flow, you can add a Terminate pipeline node to end the loop process anytime. You must customize the conditions for terminating.

Input parameters

x

• For expressions, use the built-in Expression Builder to create a variable that results from a custom expression.
• For pipeline parameters, assign a pipeline parameter and use the parameter value as input for the user variable.

Input parameters

Terminator mode can be:

• Terminate pipeline run and all running jobs stops all jobs and stops the pipeline.
• Cancel all running jobs then terminate pipeline cancels any running jobs before stopping the pipeline.
• Terminate pipeline run after running jobs finish waits for running jobs to finish, then stops the pipeline.
• Terminate pipeline that is run without stopping jobs stops the pipeline but allows running jobs to continue.

Input parameters

Output parameters

Input parameters

Output parameters

Input parameters

Output parameters

Input parameters

Output parameters

Input parameters

Output parameters

Input parameters

Output parameters

Rules for Bash script output

The output for a Bash script is often the result of a computed expression and can be large. When you are reviewing the properties for a script with valid large output, you can preview or download the output in a viewer.

These rules govern what type of large output is valid.

• The output of a list_expression is a calculated expression, so it is valid a large output.
• String output is treated as a literal value rather than a calculated expression, so it must follow the size limits that govern inline expressions. For example, you are warned when a literal value exceeds 1 KB and values of 2 KB and higher result in an error.
• 4.8.4 and later You can save standard error (standard_error) messages as a separate output and use it as input for other nodes or use it to conditionalize executing the next node.

Referencing a variable in a Bash script

The way that you reference a variable in a script depends on whether the variable was created as an input variable or as an output variable. Output variables are created as a file and require a file path in the reference. Specifically:

• Input variables are available using the assigned name
• Output variable names require that _PATH be appended to the variable name to indicate that values have to be written to the output file pointed by the {output_name}_PATH variable.

Using SSH in Bash scripts

The following steps describe how to use ssh to run your remote Bash script.

1. Create a private key and public key.
   

   ssh-keygen -t rsa -C "XXX"
   

2. Copy the public key to the remote host.
   

   ssh-copy-id USER@REMOTE_HOST
   

3. On the remote host, check whether the public key contents are added into /root/.ssh/authorized_keys.
   
4. Copy the public and private keys to a new directory in the Run Bash script node.

   mkdir -p $HOME/.ssh
   
   #copy private key content
   echo "-----BEGIN OPENSSH PRIVATE KEY-----
   ... ...
   -----END OPENSSH PRIVATE KEY-----" > $HOME/.ssh/id_rsa
   
   #copy public key content
   echo "ssh-rsa ...... " > $HOME/.ssh/id_rsa.pub
   
   chmod 400 $HOME/.ssh/id_rsa.pub
   chmod 400 $HOME/.ssh/id_rsa
   
   ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o GlobalKnownHostsFile=/dev/null -i $HOME/.ssh/id_rsa USER@REMOTE_HOST "cd /opt/scripts; ls -l; sh 1.sh"
   

Using SSH utilities in Bash scripts

The following steps describe how to use sshpass to run your remote Bash script.

1. Put your SSH password file in your system path, such as the mounted storage volume path.
2. Use the SSH password directly in the Run Bash script node:

   cd /mnts/orchestration
   ls -l sshpass
   chmod 777 sshpass
   ./sshpass -p PASSWORD ssh -o StrictHostKeyChecking=no USER@REMOTE_HOST "cd /opt/scripts; ls -l; sh 1.sh"
   

Input parameters

Output parameters

Output parameters

Input parameters

Output parameters

Input parameters

Notes:

• Environment variables that you define in a pipeline cannot be used for notebook jobs you run outside of Watson Pipelines.
• You can run a notebook from a code package in a regular package.

Output parameters

Input parameters

Output parameters

Notes for running nested pipeline jobs

If you create a pipeline with nested pipelines and run a pipeline job from the top-level, the pipelines are named and saved as project assets that use this convention:

• The top-level pipeline job is named "Trial job - pipeline guid".
• All subsequent jobs are named "pipeline_ pipeline guid".

Input parameters

Output parameters