Page History

...

• Move clients to Java to fix scala problems
  • Javadoc
  • Scala version non-compatability
  • Readability by non-scala users
  • Scary stack traces
  • Leakage of scala classes/interfaces into java api
• Code cleanup and embeddability
  • Both producer and consumer code are extremely hard to understand
  • Redo the request serialization layer to avoid all the custom request definition objects
  • Eliminate the "simple" consumer api and have only a single consumer API with the capabilities of both
  • Remove all threads from the consumer
  • Have a separate client jar with no depedencies
• Generalize APIs
  • Producer
    • Give back a return value containing error code, offset, etc
  • Consumer
    • Enable static partition assignment for stateful data systems
    • Enable consumer-driven offset changes.
• Better support non-java consumers
  • Move to a high-level protocol for consumer group management to centralize complexity on the server for all clients
• Improve performance and operability
  • Make the producer fully async to to allow issuing sends to all brokers simultaneously and having multiple in-flight requests simultaneously. This will dramatically reduce the impact of latency on throughput (which is important with replication).
  • Move to server-side offset management will allow us to scale this facility which is currently a big scalability problem for high-commit rate consumers due to zk non scalability.
  • Server-side group membership will be more scalable with number of partitions then the current consumer co-ordination protocol
  • Improve inefficiencies in compression code

The idea would be to roll out the new api as a separate jar, leaving the existing client intact but deprecated for one or two releases before removing the old client. This should allow a gradual migration.

...