...

Some of the good practices and ideas will be borrowed from existing tools.

Proposed Changes

Rest Server

...

The REST server is a separate server, sitting between Kafka cluster and client applications. It is wrapper around existing client libraries. 

...

Rest server can be easily scaled by deploying multiple proxy instances. This way we can spread the load across multiple proxy instance

Producer API

...

REST server accepts produce requests for specific topics or partitions. It internally uses java producer instance to write messages into Kafka. 

Consumer API

...

REST server uses the consumer API to consume the messages from the subscribed topic on behalf of the specified consumer group.
When a message is consumed on behalf of a consume group for the first time, then Kafka Consumer instance joins the consumer group
and subscribes to the topic. All Consumer instances that are currently members of that group and subscribed to that topic divide
topic partitions among themselves. If a Consumer instance has not consumed from a particular topic on behalf of a particular
consumer group for configured interval, then it unsubscribes from the topic on behalf of that group. Offset commit can be either
automatic or manual as requested by the user. We can also retrieve the messages for a consumer, from a specific partition,
starting with an offset.

...