0 / 0
Funnel Stage in DataStage

Funnel Stage

The Funnel stage copies multiple input data sets to a single output data set. You can use this operation for combining separate data sets into a single large data set.

The Funnel stage is a processing stage. It can have any number of input links and a single output link.

The Funnel stage can operate in one of three modes:

  • Continuous Funnel combines the records of the input data in no guaranteed order. It takes one record from each input link in turn. If data is not available on an input link, the stage skips to the next link rather than waiting.
  • Sort Funnel combines the input records in the order defined by the value(s) of one or more key columns and the order of the output records is determined by these sorting keys.
  • Sequence copies all records from the first input data set to the output data set, then all the records from the second input data set, and so on.

For all methods the meta data of all input data sets must be identical.

The sort funnel method has some particular requirements about its input data. All input data sets must be sorted by the same key columns as to be used by the Funnel operation.

Typically all input data sets for a sort funnel operation are hash-partitioned before they're sorted. Hash partitioning guarantees that all records with the same key column values are located in the same partition and so are processed on the same node. If sorting and partitioning are carried out on separate stages before the Funnel stage, this partitioning must be preserved.

The sortfunnel operation allows you to set one primary key and multiple secondary keys. The Funnel stage first examines the primary key in each input record. For multiple records with the same primary key value, it then examines secondary keys to determine the order of records it will output.

The stage editor has three tabs:

  • Stage. This is always present and is used to specify general information about the stage.
  • Input. This is where you specify details about the data sets being joined.
  • Output. This is where you specify details about the joined data being output from the stage.

Watch the following video for an example of how to work with the DataStage® Funnel stage.

This video provides a visual method to learn the concepts and tasks in this documentation.

Generative AI search and answer
These answers are generated by a large language model in watsonx.ai based on content from the product documentation. Learn more