Wave Generator stage in DataStage

Use the Wave Generator stage to break large data into more manageable sets. For example, use the Wave Generator stage to break down data into sets before you use stages like the Aggregator or Sort stages. Then, the other stages will not exceed memory limits because the stages will not run on an entire data set at once.

The Wave Generator stage is for jobs that are meant to always be up and running. Sometimes there is a need to determine end-of-wave or end-of-data marker on every interval for some stages to start processing the data. For example, Sort stage and Aggregator stage. Sort stage cannot work when the data is still incoming data. To handle this, an end of wave marker is generated on every interval. End-of-wave indicates that the data can be pushed to the next level. In most "real time always" jobs, the end of the wave is introduced by the source stage. However, you could use the Wave Generator stage in cases where the source stage did not support generating waves like Sequential File. In this case, use the Wave Generator stage.

This stage is helpful for aggregators when there is no sorting (aggregate on the entire set of rows flowing through) and XML targeting (write the XML for the entire set of rows flowing through). The source with an always on job is generally something like MQ series, javaPack using JMS, a named pipe, or other custom resource that never stops feeding data.