You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Next »


Status

Current state: Under Discussion

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here [Change the link from KAFKA-1 to your own ticket]

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

We support a new type of OffsetSpce in KIP-734 which is max-timestamp, and it's preferable to extend it to GetOffsetShell. In the future, maybe more OffsetSpce types will be added to it. 

Currently, we use KafkaConsumer to get offsets in GetOffsetShell, whereas the new OffsetSpec is only supported in AdminClient, so we need to change the client from KafkaConsumer to AdminClient.

Public Interfaces

This KIP change 2 parameters for command line tool kafka-get-offsets.sh. These 2 arguments are:

  • --time , we could pass -1(latest), -2(earliest) or a specified timestamp currently, in this KIP, we support -3(max-timestamp) which is introduced in KIP-734.
  • --command-config, currently the property file will be passed to KafkaConsumer Client, In this KIP, we change it to the property file of AdminClient.

here are some examples, 

# get the latest offset of topic1 : 
bin/kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic topic1 --time -1

# get the offset of max timestamp of topic1
bin/kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic topic1 --time -3 


# contents of kafka_admin_client.properties
bootstrap.servers=localhost:9092
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="root" password="123456";

# get offset from sasl kafka broker
bin/kafka-get-offsets.sh --command-config kafka_admin_client.properties --topic topic1 --time -1


Proposed Changes

  1. Support max timestamp in GetOffsetShell
  2. Support All AdminClient config in the file specified by --command-config, the only new config is retries , which means we will resend any request that fails when getting offsets
  3. Some old KafkaConsumer config will be ignored, for example, key.deserializer and value.deserializer

Compatibility, Deprecation, and Migration Plan

Those consumer configs which can't work with admin config will be ignored, and the only mandatory config in AdminClient is bootstrap.servers, which is also mandatory in KafkaConsumer, so this is a compatible change

Rejected Alternatives

No

  • No labels