You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 7 Next »

Status

Current state: DRAFT [One of "Under Discussion", "Accepted", "Rejected"]

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: KAFKA-3294 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The goal is to add Kafka Rest Server/Proxy to Kafka Repository. This will allow any language/tool that
can work with HTTP to produce and consume messages, and perform administrative actions with Kafka service.

Why add Kafka Rest Server/Proxy to Kafka?

There are already some open-source REST proxies are available. Some are specific to a particular user cases
Confluent REST Proxy got comprehensive interface. We want to add Kafka Rest Server to Kafka for the following reasons.

1) Many data Infra tools comes up with Rest Interface. It is useful to have inbuilt Rest API support for Produce,
   Consume messages and admin interface for integrating with external management tools.
2) Shipping Kafka Rest Server as part of a normal Kafka release makes it immediately available to every user that downloads Kafka.
3) Helps to maintain the version compatibility between Kafka and Rest Server.
 

Some of the good practices and ideas will be borrowed from existing tools.

Proposed Changes

Rest Server/Proxy:
The REST Proxy is a separate server, sitting between Kafka cluster and client applications. It is wrapper around existing client libraries. 

Request/Responses supports JSON format with embedded data format (JSON and Base64 encoded strings). Proxy uses vendor specific
content types in Content-Type and Accept headers to make the format of the data explicit. This approach is borrowed from Confluent Rest Proxy.
Supported Content-Types are: application/vnd.kafka.binary.v1+json, application/vnd.kafka.json.v1+json

Rest Proxy can be easily scaled by deploying multiple proxy instances. This way we can spread the load across multiple proxy instances.
Producer API:

REST Proxy accepts produce requests for specific topics or partitions. It internally uses java producer instance to write messages into Kafka. 

Consumer API:

Admin API and Security Integration:

This will be taken up as future work after the KIP-4 implementation.

Public Interfaces

 

Compatibility, Deprecation, and Migration Plan

  • This KIP only proposes additions. There should be no compatibility issues.


Rejected Alternatives

Make Kafka Rest Server an external third-party tool:

Main Reason for this KIP is to add Rest Server to Kafka. Shipping Kafka Rest Server as part of a normal Kafka release
makes it immediately available to every user that downloads Kafka. Also helps to maintain the version compatibility
between Kafka clients and Rest Server.


  • No labels