0 / 0
Runtime column propagation (Java Integration stage in DataStage)

Runtime column propagation (Java Integration stage in DataStage)

IBM® DataStage® allows you to define part of your schema and specify that if your job encounters extra columns that are not defined in the meta data when it actually runs, it will adopt these extra columns and propagate them through the rest of the job. This is known as runtime column propagation.

You can enable runtime column propagation for a project and set for individual links in the Output Page Columns tab. To enable runtime column propagation, select the Runtime column propagation check box .

Adding extra columns to output link

When runtime column propagation is enabled on a output link, your user code can add extra columns to this output link. The extra columns can be specified by Processor.getAdditionalOutputColumns() method, which is called for each output link whose runtime column propagation is enabled.

Following example shows how to add columns named "charCol" and "intCol" to a given output link.
public List<ColumnMetadata> getAdditionalOutputColumns
Link outputLink, List<Link> inputLinks, 
Properties stageProperties)
{
List<ColumnMetadata> addtionalColumns = new 
ArrayList<ColumnMetadata>();
ColumnMetadata charCol = new ColumnMetadataImpl
("charCol",ColumnMetadata.SQL_TYPE_CHAR);
addtionalColumns.add(charCol);
ColumnMetadata intCol = new ColumnMetadataImpl
("intCol",ColumnMetadata.SQL_TYPE_INTEGER);
addtionalColumns.add(intCol);
return addtionalColumns;
}
If you want to add columns that are in input link 0, but not in the output link, you can use the following helper method named the subtractColumnList() of the InputLink interface:

public List<ColumnMetadata> getAdditionalOutputColumns(Link outputLink, 
List<Link> inputLinks, Properties stageProperties)
{
return inputLinks.get(0).subtractColumnList(outputLink);^M
}

Alternatively, you can use JavaBean to specify column definitions on the output links at runtime. If your user code uses JavaBean on the output links whose runtime column propagation is enabled, and both the JavaBean class and the Column mapping property is empty, the Java™ Integration stage automatically creates columns from the bean properties in a given JavaBean class, and add them to an output link.

The following table summarizes the SQL type created by the Java Integration stage. You can overwrite SQL type of columns by manually adding the corresponding column definition on the output link.
Java type SQL type
byte[] Binary
boolean/java.lang.Boolean Bit
short/java.lang.Short TinyInt
int/java.lang.Integer SmallInt
double/java.lang.Double Double
float/java.lang.Float Float
long/java.lang.Long Integer
java.lang.String VarChar
java.math.BigInteger BigInt
java.math.BigDecimal Decimal
java.sql.Date Date
java.sql.Time Time
com.ibm.is.cc.javastage.api.TimeMicroseconds.class Time
java.sql.Timestamp Timestamp
If the stage has a single input link and one or more output links, the Java Integration stage automatically adds all columns on the input link that are not present in the output link whose RCP is enabled.

Transferring the column data from input to output

Java Integration stage API provides the functionality to query column metadata dynamically at runtime, and access the data. You need to create code to read the data of the propagated columns on the input link, and write them to the output link for which the RCP is enabled.

Java Integration stage also provides the functionality to automatically transfer the column data from an input link to output link if the stage has a single input link and one or more output links. In this case, your user code is not required to transfer the data of the propagated columns on the input link. For more information, see the com.ibm.is.cc.javastage.api.ColumnTransferBehavior class in the Javadoc information for the Java Integration stage API.

Generative AI search and answer
These answers are generated by a large language model in watsonx.ai based on content from the product documentation. Learn more