Kafka Toolkit v1.3.0

The release 1.3.0 of the Kafka Toolkit contains following new features:

• Kafka Client upgrade to version 1.0 - The toolkit supports Kafka Brokers v0.10.2, v0.11 and v1.0.
• support for SPL float32 data type for key and message
• different Consistent Region policies for KafkaProducer configurable via new operator parameter consistentRegionPolicy: AtLeastOnce and Transactional. AtLeastOnce is the legacy behaviour and is the default value.
• new operator parameter groupId for the KafkaConsumer operator. The group ID can now given as a submission time parameter.
• new operator parameter startOffset for the KafkaConsumer to specify the offset where the operator should start consuming records from.
• The clientId operator parameter is now also available for the KafkaProducer.

Restrictions:

1. The value Transactional of the consistentRegionPolicy parameter can only be used when the Kafka broker is at version 0.11 or higher. Older broker versions do not support transactions. Runtime errors will occur when older brokers are used.
2. The KafkaConsumer can consume from one topic only when the startOffset parameter is used.

Issues solved with this release:

#11 - Kafka Toolkit v1.3.0
#13 - Add support for transactional messaging
#15 - Add support for float32 data type
#37 - Update SPLDoc to include 'namespace-info.spl' containing info common to both operators
#62 - KafkaProducer: reset handling is confusing
#63 - KafkaProducer: reset handling cancel records

Known issues

• When Kafka's group management is enabled (KafkaConsumer not in consistent region and startPosition parameter unset or Default), the KafkaConsumer can silently stop consuming messages when committing Kafka offsets fails. #98. As a workaround, the consumer property enable.auto.commit=true can be used in a property file or app Option.

Kafka Toolkit v1.2.4

Release v1.2.4 is a bugfix release that contains following fixes:

• issue #60 - KafkaConsumer: Potential to lose connection: auto-commit replaced by explicit commit
• issue #67 - KafkaConsumer parameter startTime must document units of measurement. - spldoc update
• issue #70 - Error acquiring permit - avoid false errors on PE shutdown

Both samples and SPL documentation are now included in the release. Now, all samples have a makefile, what allows them to be built without the need to install gradle.

Known issues (28.06.2018)

• When Kafka's group management is enabled (KafkaConsumer not in consistent region and startPosition parameter unset or Default), the KafkaConsumer can silently stop consuming messages when committing Kafka offsets fails. #98. As a workaround, the consumer property enable.auto.commit=true can be used in a property file or app Option.

Kafka Toolkit v1.2.3

This release contains fixes for issue #53, #57, #58, #61, #65.

Kafka Toolkit v1.2.2

This version of the toolkit includes fixes for issue #51.

Kafka Toolkit v1.2.1

This version of the toolkit includes fixes for issues #46 & #48.

Kafka Toolkit v1.2.0

This version of the toolkit includes:

• Optional input port added to the KafkaConsumer operator to allow for dynamically adding and remove topics partitions. When adding a topic partition, the start offset can be specified to allow applications to begin consuming from specific offsets (rather than only the beginning or end of a topic-partition). The KafkaConsumerInputPortSample demonstrates how to use this functionality.
• The KafkaConsumer operator can now output the offset, partition and timestamp for each message it consumes. This functionality adds the following new parameters:
  • outputOffsetAttributeName
  • outputPartitionAttributeName
  • outputTimestampAttributeName
• The KafkaConsumer operator's startPosition parameter now includes a Time option. This option, when used in conjunction with the new startTime parameter, allows for specifying the timestamp that the consumer should begin consuming from.
• The KafkaProducer operator now provides the ability to specify the timestamp value to add to the records being written to the topics. This is supported via the new timestampAttribute parameter.

This release also fixes a number of bugs, including a critical memory issue - #42

Kafka Toolkit v1.1.1

This is a minor release that includes the fix for #34.

Kafka Toolkit v1.1.0

This version of the Kafka Toolkit includes the following:

• The KafkaConsumer operator now includes a clientId parameter that allows for programatically setting Kafka's client.id configuration property. This parameter is useful when implementing load balancing using Streams' UDP feature. Take a look at the KafkaClientIdSample for an example of how to use this parameter.
• Bug fixes (Issues fixed in this release)

Kafka Toolkit v1.0.0

This is the first release of the Kafka Toolkit. It contains operators to both consume from and produce to Kafka topics. This version of the toolkit supports Apache Kafka v0.10.x.

The toolkit contains the following features:

• At least message delivery semantics
• Support for dynamic and static topics when writing messages to Kafka topics
• Specify Kafka properties via a properties file or application configuration
• Begin consuming from either the beginning or the end of a topic
• Support for user-defined partitioners and serializers/deserializers
• Automatic determination of serializer/deserializer for keys and messages
• Support for non-string keys and messages