Congestion

A streams flow can consume large volumes of data, but if an operator cannot handle the large volume, the flow backs up to the operators that feed into it. Congestion occurs when tuples are being delivered to an operator faster than they can be processed and sent to the next operator.

What you can do

You can take a number of steps to improve the performance of a congested operator. Some steps can be done only if you have Admin privileges.

Consider implementing the following measures:

  • Modify the operator that has the Congestion icon.
    • Cloud Object Storage - To optimize partition design and file handling, see Preferred practices for partition design.

    • Code and Filter - Python performance considerations are applicable. The longer the code takes to execute, the more likely congestion will occur. For details about performance tuning, see Python Performance Tips.

    • Python Model

      • The longer the code takes to execute, the more likely congestion will occur. For more information about performance tuning, see Python Performance Tips.

      • If any required resources are not available, the operation cannot process incoming events. So, if it takes some time to upload missing resources, congestion is likely to occur. Ensure that all required resources are available.

    • Relocate the operator to a different host if the host where the operator is running is busy.

  • Design the streams flow to reduce network communication resource overload.

    • Use parallel operators to improve throughput. If order is important, consider adding a sequence identifier to the data so it can be joined in proper order later in the flow.

    • Consider using multiple hosts to take advantage of more cores, memory, and disk space.

    • For more information, see Optimizing Streams Applications.

  • Implement limits outside of Streams Designer to control the rate of data that are ingested by the streams flow.