Kafka (Source and Target) operators
Kafka is a distributed streaming platform that provides a publish and a subscribe messaging transport protocol using topics.
In a streams flow, the Kafka operator can be of type Source or of type Target.
-
Kafka as a Source operator is a subscriber to a topic in the Kafka cluster and processes the stream of records produced to that topic.
-
Kafka as a Target operator is a publisher to a topic in the Kafka cluster and publishes a stream of records to that topic.
Required parameters per security protocol
When creating a connection to Kafka, there are many different security configurations that can be used and each have their own set of requirements. The following is a list of supported configurations:
SASL_PLAINTEXT
- Brokers
- SASL mechanism
- Username
- Password
SASL_SSL
- Brokers
- SASL mechanism
- Username
- Password
- Cluster CA certificate (optional)
SSL
- Brokers
- Client SSL certificate
- Client SSL private key
- Cluster CA certificate (optional)
SASL
SASL can be used with PLAINTEXT (SASL_ PLAINTEXT) or SSL (SASL_SSL) as the transport layer using the security protocol:
- SASL_PLAINTEXT:
- Simple username/password authentication
- No SSL encryption.
- SASL_SSL supports the following SASL mechanisms:
-
SASL/PLAIN:
Simple username/password authentication mechanism that is used with TLS for encryption. -
SASL/SCRAM-SHA-256, SASL/SCRAM-SHA-512:
Stores SCRAM credentials in ZooKeeper and is suitable for production use where ZooKeeper is secure and on a private network.
-