Table of Contents |
---|
Status
Current state: Under Discussion
...
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
We support a new type of OffsetSpce
OffsetSpec
in KIP-734 which is max-timestamp, and it's preferable to extend it to GetOffsetShell. In the future, maybe more OffsetSpce
OffsetSpec
types will be added to it.
Currently, we use KafkaConsumer to get offsets in GetOffsetShell, whereas the new OffsetSpec is only supported in AdminClient, so we need to change the client from KafkaConsumer to AdminClient.
Public Interfaces
This KIP change 2 parameters for command line tool kafka-get-offsets.sh. These 2 arguments are:
...
Code Block | ||
---|---|---|
| ||
# get the latest offset of topic1 : bin/kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic topic1 --time -1 # get the offset of max timestamp of topic1 bin/kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic topic1 --time -3 # contents of kafka_admin_client.properties bootstrap.servers=localhost:9092 security.protocol=SASL_PLAINTEXT sasl.mechanism=SCRAM-SHA-256 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="root" password="123456"; # get offset from sasl kafka broker bin/kafka-get-offsets.sh --command-config kafka_admin_client.properties --topic topic1 --time -1 |
Proposed Changes
- Support max timestamp in GetOffsetShell
- Support All AdminClient config in the file specified by --command-config, the only new config is
retries
, which means we will resend any request that fails when getting offsets - Some old KafkaConsumer config will be ignored, for example, key.deserializer and value.deserializer
Compatibility, Deprecation, and Migration Plan
...
The only incompatible change is the --command-config param, currently the property file will be passed to KafkaConsumer Client, In this KIP, we change it to the property file of AdminClient, here are some differences:
- Some ConsumerConfig which can't work with
...
- AdminClient will be ignored, for example, group.id and key.deserializer, which has no influence on the client.
- Only one AdminClientConfig is not presented in ConsumerConfig which is
retries
, and it is not mandatory for AdminClient with default value=Integer.MaxValue, so this has a very little effect on the client. - The only mandatory config in AdminClient is bootstrap.servers, which is also mandatory in KafkaConsumer.
So we can conclude , so this is a compatible change and the transition won't be noticed.
Rejected Alternatives
...
Extend KafkaConsumer to support max-timestamp
Currently, we can get the earliest and latest offset using KafkaConsumer, we can also simply support max-timestamp in GetOffsetShell if we support it in KafkaConsumer.
Ultimately, we determine that admin client is a better way to implement this, or otherwise we need to extend KafkaConsumer every time we add a new OffsetSpec, in addition, AdminClient is more lightweight since we need to construct many unused components in KafkaConsumer, e.g. OffsetSpce.No