Runtime column propagation (Java Integration stage in DataStage)
IBM® DataStage® allows you to define part of your schema and specify that if your job encounters extra columns that are not defined in the meta data when it actually runs, it will adopt these extra columns and propagate them through the rest of the job. This is known as runtime column propagation.
You can enable runtime column propagation for a project and set for individual links in the Output Page Columns tab. To enable runtime column propagation, select the Runtime column propagation check box .
Adding extra columns to output link
When runtime column propagation is enabled on a output link, your
user code can add extra columns to this output link. The extra columns
can be specified by Processor.getAdditionalOutputColumns()
method,
which is called for each output link whose runtime column propagation
is enabled.
public List<ColumnMetadata> getAdditionalOutputColumns
Link outputLink, List<Link> inputLinks,
Properties stageProperties)
{
List<ColumnMetadata> addtionalColumns = new
ArrayList<ColumnMetadata>();
ColumnMetadata charCol = new ColumnMetadataImpl
("charCol",ColumnMetadata.SQL_TYPE_CHAR);
addtionalColumns.add(charCol);
ColumnMetadata intCol = new ColumnMetadataImpl
("intCol",ColumnMetadata.SQL_TYPE_INTEGER);
addtionalColumns.add(intCol);
return addtionalColumns;
}
If you want to add columns that are in input link 0,
but not in the output link, you can use the following helper method
named the subtractColumnList()
of the InputLink
interface:
public List<ColumnMetadata> getAdditionalOutputColumns(Link outputLink,
List<Link> inputLinks, Properties stageProperties)
{
return inputLinks.get(0).subtractColumnList(outputLink);^M
}
Alternatively, you can use JavaBean to specify column definitions on the output links at runtime. If your user code uses JavaBean on the output links whose runtime column propagation is enabled, and both the JavaBean class and the Column mapping property is empty, the Java™ Integration stage automatically creates columns from the bean properties in a given JavaBean class, and add them to an output link.
Java type | SQL type |
---|---|
byte[] | Binary |
boolean/java.lang.Boolean | Bit |
short/java.lang.Short | TinyInt |
int/java.lang.Integer | SmallInt |
double/java.lang.Double | Double |
float/java.lang.Float | Float |
long/java.lang.Long | Integer |
java.lang.String | VarChar |
java.math.BigInteger | BigInt |
java.math.BigDecimal | Decimal |
java.sql.Date | Date |
java.sql.Time | Time |
com.ibm.is.cc.javastage.api.TimeMicroseconds.class | Time |
java.sql.Timestamp | Timestamp |
Transferring the column data from input to output
Java Integration stage API provides the functionality to query column metadata dynamically at runtime, and access the data. You need to create code to read the data of the propagated columns on the input link, and write them to the output link for which the RCP is enabled.
Java Integration stage also provides the functionality to automatically transfer the column data from an input link to output link if the stage has a single input link and one or more output links. In this case, your user code is not required to transfer the data of the propagated columns on the input link. For more information, see the com.ibm.is.cc.javastage.api.ColumnTransferBehavior class in the Javadoc information for the Java Integration stage API.