Welcome to the Kafka Toolkit. This toolkit enables SPL applications to integrate with Apache Kafka.
This version of the toolkit currently supports: Apache Kafka 0.10.2, 0.11, 1.0, 1.1, and 2.0 to 2.7. The older Kafka versions 0.9 and 0.10.0, are not supported any more. Using the Kafka operators from the com.ibm.streamsx.messaging toolkit for these server versions is not recommended.
For building applications that connect to the IBM Cloud Event Streams service, it is recommended that the com.ibm.streamsx.messagehub toolkit be used. This toolkit provides functionality on top of the Kafka toolkit to simplify the configuration needed to connect to the public cloud service.
To migrate applications using the old Kafka operators in the com.ibm.streamsx.messaging toolkit, refer to the Migration Document.
- Toolkit documentation can be found here: SPLDoc
- Standard use cases for Consumers and Partitions can be found on the overview page of the user documentation
- Toolkit reference for latest and older versions
cd com.ibm.streamsx.kafka
../gradlew build
./gradlew spldoc
NOTE: SPLDocs will be generated in the docs/spldoc directory.
./gradlew release
NOTE: The release will be available in the build/release/output folder.
cd tests/KafkaTests
./setup.sh -b <list_of_bootstrap_servers>
../../gradlew test
NOTE 1: setup.sh will add a domain-level app config called "kafka-tests", as well as create a properties file containing the bootstrap.servers property.
NOTE 2: Tests will run using the local domain specified by the STREAMS_DOMAIN_ID env var. All tests run in Distributed mode.
| topic name | number of partitions |
|---|---|
| test | 3 |
| other1 | 1 |
| other2 | 1 |
| position | 1 |
Create them by hand before or make sure that they are automatically created with the required number of partitions.
Hint: The test case kafkaStartPositionTest requires that the position topic is re-created before every test run.
Each sample contains a build.gradle file and a Makefile. The samples can be built/compiled by running ../../gradlew build or make from the sample directory.