Looking up data in the Sparse lookup mode
The Java™ Integration stage supports sparse lookup. You can implement the lookup logic in your Java code to read the input record and write to the output record one by one.
When the connector is used to complete a sparse lookup
operation, the connector has one input link, one output link, and
an optional reject link. The connector with a single output reference
link is internally converted to contain one input link, one output
link and optionally one reject link, if the Lookup stage in the job
contains a reject link. You must add an implementation in your Java code to run in sparse lookup
mode in the process()
method.
process()
mode, call the getSparseLookupMode()
method
for the Configuration interface as follows: SparseLookupMode m_sparseLookupMode = m_configuration.getSparseLookupMode()
The return value indicates the lookup mode that is configured. In the Lookup Stage Conditions pane, the Lookup Failure column indicates the expected behavior when the lookup operation fails. The Java code must contain the logic to create a loop to repeatedly read the data from the input link, and write to the output link or reject link. The schema for the input link and reject link is the same as the schema for the input link for the Lookup stage. The schema for the output link is the same as the schema for the output reference link for the Java Integration stage. The Java code must read the key columns on the input link and write the output records that have the lookup data.
To
identify the key columns on the input link, use the isKey()
method
for the ColumnMetadata interface as follows:
List<ColumnMetadata> inputColumns = m_inputLink.getColumnMetadata();
ColumnMetadata cm = inputColumns.get(columnIndex);
boolean isKey = cm.isKey();
After the Java code writes output records, the Java Integration stage and the framework add or copy the additional columns according to the mapping configuration in the Lookup stage so that the schema of the output link is the same as the schema of the output link from the Lookup stage.
When
you implement the logic for sparse lookup, ensure that the user code
calls the writeRecord()
method for OutputLink or
RejectLink, before it calls the InputLink.readRecord()
method
to fetch the next incoming record. The framework also writes to the
column for sparse lookup output along with the connector. The connector
framework holds the cursor for the input record that the user code
reads and when you call the InputLink.readRecord()
method,
it changes the cursor position to a different record. To write the
output record correctly, the writeRecord()
method
must be called before the InputLink.readRecord()
method
for the next record.
getSparseLookupMode()
methods
and process failed records. The following behavior is expected for
each mode:
Mode | Expected behavior |
---|---|
Continue | The lookup operation continues after the empty
data is sent to columns in the looked up data. You can implement Java code to pass empty records
or record with no values to the writeRecord() method. |
Drop | Drop the records that don't match the records
on lookup table. When the code makes the next call for the readRecord() method,
the input record is discarded automatically. There is nothing to be
implemented in the Java code. |
Fail | The job ends when the lookup fails. The Java code throws an exception when the job ends. |
Reject | The records that don't match the records on
the lookup table are rejected. The input record are passed to the RejectLink.writeRecord() method. |
The following sample code shows the process()
method
and the implementation of expected behavior for sparse lookup on lookup
failure:
public void process() throws Exception
{
if (m_sparseLookupMode == Configuration.SparseLookupMode.NOT_SPARSE)
{
// regular implementation for process()
// ...
// ...
}
else
{
// Sparse lookup implementation
do
{
InputRecord inputRecord = m_inputLink.readRecord();
if (inputRecord == null)
{
// No more input
return;
}
OutputRecord outputRecord = m_outputLink.getOutputRecord();
OutputRecord emptyRecord = m_outputLink.getOutputRecord();
// -- do the lookup and fill output record
boolean fLookupFail = lookup(inputRecord, outputRecord);
if (!fLookupFail)
{
// lookup success - put the output record populated as you like
m_outputLink.writeRecord(outputRecord);
}
else
{
switch(m_sparseLookupMode)
{
case Configuration.SparseLookupMode.CONTINUE:
// "Continue" mode - send out the empty record
m_outputLink.writeRecord(emptyRecord);
break;
case Configuration.SparseLookupMode.DROP:
// "Drop" mode - do nothing and proceed to read next record.
break;
case Configuration.SparseLookupMode.FAIL:
// "Fail" mode - throw exception to abort the job
throw new ConnectorException(9998, "Lookup failed.");
case Configuration.SparseLookupMode.REJECT:
// "Reject" mode - copy input record into reject link
RejectRecord rejectRecord = m_rejectLink.getRejectRecord(inputRecord);
m_rejectLink.writeRecord(rejectRecord);
break;
}
}
}
while(true);
}
}
In the sample code, users implement their own
lookup logic in the lookup()
function by looking
at the input data on the inputRecord
argument and
populate the outputReocrd
argument with the extracted
data. The return value indicates whether the lookup is a success or
a failure.
The sparse lookup mode runs with the special link topology. In the sparse lookup mode, the Java Integration Stage does not enforce the capability check on the number of input and reject links.