Kafka setup guide

Kafka doesn’t manage users directly; it relies on the underlying authentication system. So if you’re using SASL/PLAIN for authentication, you would add the user to the JAAS configuration file.

1. Open the JAAS configuration file (e.g., kafka_server_jaas.conf) in a text editor.
2. Add the following entry to create the user “propel” with the password:

KafkaServer {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="propel"
  password="YOUR_SUPER_SECURE_PASSWORD"
  user_propel="YOUR_SUPER_SECURE_PASSWORD";
};

Replace YOUR_SUPER_SECURE_PASSWORD with a secure password.

3. Save the file.

Set the KAFKA_OPTS environment variable to point to your JAAS config file:

export KAFKA_OPTS="-Djava.security.auth.login.config=/path/to/your/kafka_server_jaas.conf"

Restart the Kafka server for the changes to take effect.

Now, you’ll use Kafka’s Access Control Lists (ACLs) to grant permissions to the “propel” user.

Use the kafka-acls CLI to add ACLs for the “propel” user so that it can operate on the propel-* consumer groups.

bin/kafka-acls.sh \
  --authorizer-properties zookeeper.connect=localhost:2181 \
  --add \
  --allow-principal 'User:propel' \
  --operation Describe \
  --operation Read \
  --operation Delete \
  --group 'propel-' --resource-pattern-type prefixed

For each topic you need to ingest to Propel, run the following command:

bin/kafka-acls.sh \
  --authorizer-properties zookeeper.connect=localhost:2181 \
  --add \
  --allow-principal 'User:propel' \
  --operation Describe \
  --operation Read \
  --topic 'YOUR_TOPIC'

Make sure to replace localhost:2181 with your Zookeeper server.

These commands grant Describe and Read access to the topics for the user “propel”.

Verify that the ACLs have been correctly set by listing the ACLs for the topics you authorized.

bin/kafka-acls.sh \
  --authorizer-properties zookeeper.connect=localhost:2181 \
  --list \
  --topic YOUR_TOPIC

You should see the ACLs you added for the user “propel”.

Go to the “Data Pools” section in the Console, click “Create Data Pool” and click on the “Kafka” tile.

If you create a Kafka Data Pool for the first time, you must create your Kafka credentials for Propel to connect to your Kafka servers.

To create your Kafka credentials, you will need the following details:

• Bootstrap servers: The list of addresses for your Kafka cluster’s brokers.
• Authentication type: The authentication protocol used by your Kafka cluster: SASL/SCRAM-SHA-256, SASL/SCRAM-SHA-512, SASL/PLAIN, or NONE.
• TLS: Whether your Kafka cluster uses TLS for secure communication.
• Username: The username for the user you created in your Kafka cluster.
• Password: The password for the user you created in your Kafka cluster.

After entering your Kafka credentials, click “Create and test credentials” to ensure Propel can successfully connect to your Kafka cluster. If the connection is successful, you will see a confirmation message. If not, check your entered credentials and try again.

Here, you will see a list of topics available to ingest. If you don’t see the topic you want to ingest, make sure your user has the right permissions to access the topic.

Here, you will see a list of topics available to ingest. Select the topic you want to ingest into this Data Pool. You will see the schema of the Data Pool.

Next, you need to select the timestamp column. This is the column that will be used to order the data in the Data Pool. By default, Propel selects the _timestamp generated by Kafka.

After you’ve selected the topic, provide a name for your Data Pool. This name will be used to identify the Data Pool in Propel. Once you’ve named your Data Pool, click “Create Data Pool”. Propel will then start ingesting data from the selected Kafka topics into your Data Pool.

Once you’ve started ingesting data, you can view the data in your Data Pool. Go to the “Data Pools” section in the Console, click on your Kafka Data Pool, and click on the “Preview Data” tab. Here, you can see the data that has been ingested from your Kafka topic.