Kafka Toolkit v1.6.0
This Release of the Kafka toolkit contains following enhancements:
KafkaProducer
1. Metric reporting
The Kafka producer in the client collects various performance metrics. A subset has been exposed as custom metrics to the operator (issue #112).
| Custom Metric name | Description |
|---|---|
| connection-count | The current number of active connections. |
| compression-rate-avg | The average compression rate of record batches (as percentage, 100 means no compression). |
| topic:compression-rate | The average compression rate of record batches for a topic (as percentage, 100 means no compression). |
| record-queue-time-avg | The average time in ms record batches spent in the send buffer. |
| record-queue-time-max | The maximum time in ms record batches spent in the send buffer. |
| record-send-rate | The average number of records sent per second. |
| record-retry-total | The total number of retried record sends |
| topic:record-send-total | The total number of records sent for a topic. |
| topic:record-retry-total | The total number of retried record sends for a topic |
| topic:record-error-total | The total number of record sends that resulted in errors for a topic |
| records-per-request-avg | The average number of records per request. |
| requests-in-flight | The current number of in-flight requests awaiting a response. |
| request-rate | The number of requests sent per second |
| request-size-avg | The average size of requests sent. |
| request-latency-avg | The average request latency in ms |
| request-latency-max | The maximum request latency in ms |
| batch-size-avg | The average number of bytes sent per partition per-request. |
| outgoing-byte-rate | The number of outgoing bytes sent to all servers per second |
| bufferpool-wait-time-total | The total time an appender waits for space allocation. |
| buffer-available-bytes | The total amount of buffer memory that is not being used (either unallocated or in the free list). |
2. Default producer configs
Previous releases of the toolkit have used the Kafka default producer configs unless otherwise configured by the user. For optimum throughput these settings had to be tuned. Now, the important producer configs have default values, which result in higher throughput to the broker and reliability (issue #113):
| Property name | Kafka default | New operator default |
|---|---|---|
| retries | 0 |
10. When 0 is provided as retries and consistentRegionPolicy parameter is Transactional retries is adjusted to 1. |
| compression.type | none |
lz4 |
| linger.ms | 0 |
100 |
| batch.size | 16384 |
32768 |
| max.in.flight.requests.per.connection | 5 |
1 when guaranteeOrdering parameter is true, limited to 5 when provided and consistentRegionPolicy parameter is Transactional, or 10 in all other cases. |
3. New optional operator parameter guaranteeOrdering
If set to true, the operator guarantees that the order of records within a topic partition is the same as the order of processed tuples when it comes to retries. This implies that the operator sets the max.in.flight.requests.per.connection producer config automatically to 1 if retries are enabled, i.e. when the retries config is unequal 0, what is the operator default value.
If unset, the default value of this parameter is false, which means that the order can change due to retries.
4. Queue time control
In previous releases including 1.5.1, the producer operator was easily damageable when the producer did not come up transferring the data to the broker nodes. Then records stayed to long in the accumulator (basically a buffer), what caused timeouts with subsequent restarts (issue #128).
The producer now has an adaptive control that monitors several producer metrics and flushes the producer on occasion to ensure that the maximum queue time of records stays below typically 5 seconds. This enhancement addresses the robustness of the producer operator.
KafkaConsumer
1. Metric reporting
The Kafka consumer in the client collects various performance metrics. A subset has been exposed as custom metrics to the operator:
| Custom Metric name | Description |
|---|---|
| connection-count | The current number of active connections. |
| incoming-byte-rate | The number of bytes read off all sockets per second |
| topic-partition:records-lag | The latest lag of the partition |
| records-lag-max | The maximum lag in terms of number of records for any partition in this window |
| fetch-size-avg | The average number of bytes fetched per request |
| topic:fetch-size-avg | The average number of bytes fetched per request for a topic |
| commit-rate | The number of commit calls per second |
| commit-latency-avg | The average time taken for a commit request |
One of the most interesting metric is the record lag for every consumed topic partition. The lag is the difference between the offset of the last inserted record and current reading position.