I am working on a proposal to satisfy the Restful Proxy project listed on the Projects page. (http://incubator.apache.org/kafka/projects.html)

My working knowledge of Kafka is about 3 hours old and I have actually written zero code yet, it is my job to do so for Box so I will update this as my knowledge increases. There may be some fundamental issues with this proposal at this time.

General Idea

Create a Restful API which will allow producers and consumers to communicate with a Kafka server via HTTP calls

Consumers

URL Format:

GET /messages/<topic>/<consumer>/<offset> - Retrieve the messages for a consumer, from the topic, starting with an offset

DELETE /messages/<topic>/<consumer>/<offset> - ACK consumption of message for a consumer, topic, offset

Producers

PUT /messages/<topic>

Request Body will become the message

Known Missing Items:

  • I know that this does not take into account partitions.
  • It would be nice to be able to implement the async producer and have an HTTP callback happen from within the RESTServer (producer specifies a URL to be requested as callback)
  • If an HTTP call back wasn't available, have the service return a task ID (MD5 of the message or something), and then allow the client to query the server to see if the callback from Kafka has happened yet.
  • No labels

2 Comments

  1. Jay Kreps

    This is great. A few comments:

    • I am not much of a REST person, but maybe the PUT should be a POST since it appends?
    • The PUT/POST will definitely need to support multiple messages in a single batch for efficiency. This could be done either with HTTP multipart perhaps or just by allowing one to post a full message set.
    • Another approach to the API would be to have the consumer stream messages to the client. This might be more natural.
    • It is worth considering whether offset management should be exposed in this api or hidden, or perhaps both depending on the options given.
    • It would be good to allow plugin APIs for optional serialization and deserialization by the REST server.

  2. Adolfo Cortés

    Hello, did you reach any results in your KAFKA HTTP REST API proposal?