0 / 0
Migrating and constructing pipeline flows for DataStage

Migrating and constructing pipeline flows for DataStage

The following steps and limitations apply to migrated Sequence Jobs and flows that are constructed directly with the pipeline canvas.

Migrated flows

Wait for file
Manually reselect or configure the file path. As a helper node for cross loop, the default timeout value is 23:59:59. Manually update value or set to 00:00:00 for no timeout.
Wait for all
Replaces Sequencer (all) and Nested condition.
Wait for any
Replaces Sequencer (any).
Terminate pipeline
Replaces Terminator.
Final message text is not supported.
Terminate loop
Not supported during migration, must be manually added to a loop.
Loop in sequence
Replaces Start/end loop.
Run DataStage job
Replaces Job activity.
The List type is mapped to Enum. Path is mapped to the File type. For information see Configuring global objects for Watson Pipelines.
Run Bash script
Set user variables
Replaces User variable.
Error handling
Replaces Exception handler.
Use error.status and error.status_message to get the failed node's information. Use ds.GetErrorSource() and ds.GetErrorNumber() to get the error source and error number.
General pipeline issues
  • Unsupported functions return "1" or "unsupported."
  • When outside nodes are accessed inside a loop or exception handler, migration adds an extra Set user variables node.
  • When the main pipeline has an Exception handler node or when a Loop node has links that point out to the main pipeline, migration creates extra nodes and a local parameter, MigrationTempFolder. A Run Bash script node is created inside the Loop or Exception handler. This node creates a linkage file under the mounted path provided in MigrationTempFolder, which must include a / at the end. Outside of the Loop or Exception handler, a Wait for file node is created which waits for the linkage file to be created. If the mounted path is incorrect the node will wait 24 hours. Another Run Bash script node deletes the linkage file afterward.

  • When Automatically handle activities that fail is selected, all migrated run nodes will be set to Fail on pipeline error, unless a condition has been defined on a link for when an error is thrown. If a condition has been defined or Automatically handle activities that fail is not defined, nodes will be set to Continue pipeline on error.

Set and get user status

To set user status in a DataStage job, you can call the built-in function SetUserStatus from the Expression builder in the Transformer stage. To get the status in a pipeline that calls the DataStage job with a Run DataStage job node, you can use the built-in function ds.GetUserStatus(tasks.<node name>) with the name of the Run DataStage job node. You can also access it in the job results with tasks.<node name>.user_status. To set user status in a pipeline, you must add it as a variable with the Set user variables node and select Make user variable value available as a pipeline result, which makes it an output parameter that other pipelines can access. Another pipeline can use a Run pipeline job node to call the pipeline that set the user status, and then get the user status using tasks.<node name>.results.output_parameters.<user status parameter name>.

When you go to Triggers in the Transformer and call SetUserStatus, it cannot be used on input column derivations.

Constructed flows

Run DataStage job
The Environment Variables parameter value is overwritten if the DataStage flow includes the same environment. The List type is mapped to Enum. Path is mapped to the File type. For information see Configuring global objects for Watson Pipelines.
Run Bash script
Echo statements must use double quotes to access the value of a variable. For example, echo "variablename" will replace "variablename" with the value of the variable. echo 'variablename' will just echo the name of the variable.
Generative AI search and answer
These answers are generated by a large language model in watsonx.ai based on content from the product documentation. Learn more