Looking up data in the Sparse lookup mode

The Java™ Integration stage supports sparse lookup. You can implement the lookup logic in your Java code to read the input record and write to the output record one by one.

When the connector is used to complete a sparse lookup operation, the connector has one input link, one output link, and an optional reject link. The connector with a single output reference link is internally converted to contain one input link, one output link and optionally one reject link, if the Lookup stage in the job contains a reject link. You must add an implementation in your Java code to run in sparse lookup mode in the process() method.

To identify the lookup mode in the process() mode, call the getSparseLookupMode() method for the Configuration interface as follows:
SparseLookupMode m_sparseLookupMode = m_configuration.getSparseLookupMode()

The return value indicates the lookup mode that is configured. In the Lookup Stage Conditions pane, the Lookup Failure column indicates the expected behavior when the lookup operation fails. The Java code must contain the logic to create a loop to repeatedly read the data from the input link, and write to the output link or reject link. The schema for the input link and reject link is the same as the schema for the input link for the Lookup stage. The schema for the output link is the same as the schema for the output reference link for the Java Integration stage. The Java code must read the key columns on the input link and write the output records that have the lookup data.

To identify the key columns on the input link, use the isKey() method for the ColumnMetadata interface as follows:

List<ColumnMetadata> inputColumns = m_inputLink.getColumnMetadata();
ColumnMetadata cm = inputColumns.get(columnIndex);
boolean isKey = cm.isKey();

After the Java code writes output records, the Java Integration stage and the framework add or copy the additional columns according to the mapping configuration in the Lookup stage so that the schema of the output link is the same as the schema of the output link from the Lookup stage.

When you implement the logic for sparse lookup, ensure that the user code calls the writeRecord() method for OutputLink or RejectLink, before it calls the InputLink.readRecord() method to fetch the next incoming record. The framework also writes to the column for sparse lookup output along with the connector. The connector framework holds the cursor for the input record that the user code reads and when you call the InputLink.readRecord() method, it changes the cursor position to a different record. To write the output record correctly, the writeRecord() method must be called before the InputLink.readRecord() method for the next record.

It is not mandatory to implement an expected behavior for sparse lookup on lookup failure. However, it is a good practice to implement an expected behaviour as the user code can check the return value for the getSparseLookupMode() methods and process failed records. The following behavior is expected for each mode:
Table 1. Expected behaviour for sparse lookup on lookup failure
Mode Expected behavior
Continue The lookup operation continues after the empty data is sent to columns in the looked up data. You can implement Java code to pass empty records or record with no values to the writeRecord() method.
Drop Drop the records that don't match the records on lookup table. When the code makes the next call for the readRecord() method, the input record is discarded automatically. There is nothing to be implemented in the Java code.
Fail The job ends when the lookup fails. The Java code throws an exception when the job ends.
Reject The records that don't match the records on the lookup table are rejected. The input record are passed to the RejectLink.writeRecord() method.

The following sample code shows the process() method and the implementation of expected behavior for sparse lookup on lookup failure:

public void process() throws Exception
{
      
    if (m_sparseLookupMode == Configuration.SparseLookupMode.NOT_SPARSE)
    {
         // regular implementation for process() 
         // ...
         // ...
    }
    else
    {
         // Sparse lookup implementation
         do
         {
            InputRecord inputRecord = m_inputLink.readRecord();
            if (inputRecord == null)
            {
               // No more input
               return;
            }
            
            OutputRecord outputRecord = m_outputLink.getOutputRecord();
            OutputRecord emptyRecord = m_outputLink.getOutputRecord();
   
            // -- do the lookup and fill output record
            boolean fLookupFail = lookup(inputRecord, outputRecord);
   
            if (!fLookupFail)
            {
               // lookup success - put the output record populated as you like
               m_outputLink.writeRecord(outputRecord);
            }
            else 
            {
               switch(m_sparseLookupMode)
               {
               case Configuration.SparseLookupMode.CONTINUE:
                  // "Continue" mode - send out the empty record
                  m_outputLink.writeRecord(emptyRecord);
                  break;
               case Configuration.SparseLookupMode.DROP: 
                  // "Drop" mode - do nothing and proceed to read next record. 
                  break;
               case Configuration.SparseLookupMode.FAIL: 
                  // "Fail" mode - throw exception to abort the job
                  throw new ConnectorException(9998, "Lookup failed.");
               case Configuration.SparseLookupMode.REJECT: 
                  // "Reject" mode - copy input record into reject link
                  RejectRecord rejectRecord = m_rejectLink.getRejectRecord(inputRecord);
                  m_rejectLink.writeRecord(rejectRecord);
                  break;
               }
            }
         }
         while(true);
    }
}

In the sample code, users implement their own lookup logic in the lookup() function by looking at the input data on the inputRecord argument and populate the outputReocrd argument with the extracted data. The return value indicates whether the lookup is a success or a failure.

The sparse lookup mode runs with the special link topology. In the sparse lookup mode, the Java Integration Stage does not enforce the capability check on the number of input and reject links.