Best practice: Adding assets to a pipeline

When you create a pipeline, you add assets, such as data, notebooks, or Data Refinery jobs to the pipeline to orchestrate a sequential process. The strongly recommended method for adding assets to a pipeline is to collect the assets in the project containing the pipeline and use the asset browser to select project assets for the pipeline.

Copy assets

Copy selected assets from a project or space to a non-empty space. You can copy these assets to a space:

• Data asset
• Model
• ML Pipeline
• Function
• Connection
• Software specification
• Data Refinery flow
• Environment
• Script

Export assets

Export selected assets from the scope, for example, a project or deployment space. The operation exports all the assets by default. You can limit asset selection by building a list of resources to export.

Import Assets

Import assets from a ZIP file containing exported assets.

Create AutoAI experiment

Use this node to train an AutoAI classification or regression experiment and generate model-candidate pipelines.

Create AutoAI time series experiment

Use this node to train an AutoAI time series experiment and generate model-candidate pipelines.

Create batch deployment

Use this node to create a batch deployment for a machine learning model.

Create data asset

Use this node to create a data asset.

Create deployment space

Use this node to create and configure a space that you can use to organize and create deployments.

Create online deployment

Use this node to create an online deployment where you can submit test data directly to a web service REST API endpoint.

Wait for all results

Use this node to wait until all results from the previous nodes in the pipeline are available so the pipeline can continue.

This node takes no inputs and produces no output. When the results are all available, the pipeline continues automatically.

Wait for any result

Use this node to wait until any result from the previous nodes in the pipeline is available so the pipeline can continue. Run the downstream nodes as soon as any of the upstream conditions have been met.

This node takes no inputs and produces no output. When any results are available, the pipeline continues automatically.

Wait for file

Wait for an asset to be created or updated in the location that is specified in the path from a job or process earlier in the pipeline. Specify a timeout length to wait for the condition to be met. If 00:00:00 is the specified timeout length, the flow waits indefinitely.

Loops

Loops are a node in a Pipeline that operates like a coded loop.

There are two types of loops: parallel and sequential.

You can use loops when the number of iterations for an operation is dynamic. For example, if you don't know the number of notebooks to process, or you want to choose the number of notebooks at run time, you can use a loop to iterate through the list of notebooks.

You can also use a loop to iterate through the output of a node or through elements in a data array.

Referencing a variable in a Bash script

The way you reference a variable in a script depends on whether the variable was created as an input variable or as an output variable. Output variables are created as a file and require a file path in the reference. Specifically:

• Input variables are available using the assigned name
• Output variable names require that _PATH be appended to the variable name to indicate that values have to be written to the output file pointed by the {output_name}_PATH variable.

Set user variables

Configure a user variable with a key/value pair, then add the list of dynamic variables for this node.

For details on creating a user variable, see Configuring global objects.

Terminate pipeline

You can initiate and control the termination of a pipeline with a Terminate pipeline node from the Control category. When the error flow runs, you can optionally specify how to handle notebook or training jobs that were initiated by nodes in the pipeline. You must specify whether to wait for jobs to finish, cancel the jobs then stop the pipeline, or just stop everything without canceling. Specify the options for the Terminate pipeline node.

Update AutoAI experiment

Update the training details for an AutoAI experiment.

Update batch deployment

Use these parameters to update a batch deployment.

Update deployment space

Update the details for a space.

Update online deployment

Use these parameters to update an online deployment (web service).

Run AutoAI experiment

Trains and stores AutoAI experiment pipelines and models.

Run Bash script

Run an inline Bash script to automate a function or process for the pipeline. You can enter the Bash script code manually, or you can import the bash script from a resource, pipeline parameter, or the output of another node.

You can also use a Bash script to process large output files. For example, you can generate a large, comma-separated list that you can then iterate over using a loop.

In the following example, the user entered the inline script code manually. The script uses the cpdctl tool to search all notebooks with a set variable tag and aggregates the results in a JSON list. The list can then be used in another node, such as running the notebooks returned from the search.

Run batch deployment job

Configure this node to run selected deployment jobs.

Run Data Refinery job

This node runs a specified Data Refinery job.

Run Notebook job

Use these configuration options to specify how to run a Jupyter Notebook in a pipeline.

Run Pipelines component

Run a reusable pipeline component created using a Python script. For details, see Creating a custom component.

• If a pipeline component is available, configuring the node presents a list of available components.
• The component you choose specifies the input and output for the node.
• Once you assign a component to a node, you cannot delete or change the component. You must delete the node and create a new one.

Run Pipelines job

Add a pipeline to run a nested pipeline job as part of a containing pipeline. This is a way of adding reusable processes to multiple pipelines. You can use the output from a nested pipeline run as input for a node in the containing pipeline.