Project Ideas

General

• Security and SSL support
• Quota - currently DL support per stream rate limiting. It is kind of throughput quota. It would be good to also have data size quota.
• Improve rate limiting algorithm - currently the rate limiting is a decentralized solution. It would be good to see how to evolve it into a more centralized solution and more controllable.

Core Library

• Metadata Abstraction to integrate with different consistent metadata stores. For example, Etcd.
• Log segment abstraction to integrate with different data stores. For example, HDFS, Memory, Amazon S3 and such.
• Endless stream
• Stream transformation, like filtering and projection
• Support large records over 1 megabyte.

Write Proxy

• Zero data copy enhancement
• Better stream placement. For example, traffic or network bandwidth usage aware algorithms.
• Be able to support different wire protocol. For example, gRPC.
• Support exclusive writer. Currently you can achieve fencing only if you use core library directly. In some use cases, fencing is still required even using write proxy.
• Exact once or transnational publish

Write Proxy Client

• Multiple language bindings. E.g C++ client.

Deployment

• Different deployment modes: local, distributed and broker-like.
  
  • Currently you can deploy the whole stack locally or separated in a different way. It might make sense to deploy both bookie and write proxy together in same instance, in a more 'broker'-like way. It would simply the deployment if users don't use a cluster management system like mesos.
• Easy to deploy and install to different environments
• Be able to run distributedlog on Windows

Documentation

• Documents written in different languages. For example, Chinese.
• Improve the documentation. For example, write proxy wire protocol, rate limiting algorithm.

Integration

• Integrate with ingestion systems
• Integrate with stream computing engines
• Integrate with use cases that require a WAL
• Libraries that make building state machines easily