Kafka (Source and Target) operators

Kafka is a distributed streaming platform that provides a publish and a subscribe messaging transport protocol using topics.

In a streams flow, the Kafka operator can be of type Source or of type Target.

  • Kafka as a Source operator is a subscriber to a topic in the Kafka cluster and processes the stream of records produced to that topic.

  • Kafka as a Target operator is a publisher to a topic in the Kafka cluster and publishes a stream of records to that topic.

Required parameters per security protocol

When creating a connection to Kafka, there are many different security configurations that can be used and each have their own set of requirements. The following is a list of supported configurations:

SASL_PLAINTEXT

  • Brokers
  • SASL mechanism
  • Username
  • Password

SASL_SSL

  • Brokers
  • SASL mechanism
  • Username
  • Password
  • Cluster CA certificate (optional)

SSL

  • Brokers
  • Client SSL certificate
  • Client SSL private key
  • Cluster CA certificate (optional)

SASL

SASL can be used with PLAINTEXT (SASL_ PLAINTEXT) or SSL (SASL_SSL) as the transport layer using the security protocol:

  • SASL_PLAINTEXT:
    • Simple username/password authentication
    • No SSL encryption.
  • SASL_SSL supports the following SASL mechanisms:

    • SASL/PLAIN:
      Simple username/password authentication mechanism that is used with TLS for encryption.

    • SASL/SCRAM-SHA-256, SASL/SCRAM-SHA-512:
      Stores SCRAM credentials in ZooKeeper and is suitable for production use where ZooKeeper is secure and on a private network.

Learn more