THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
- Kafka Streams - the built-in stream processing library of the Apache Kafka project
- Kafka Streams Ecosystem:
- Complex Event Processing (CEP): https://github.com/fhussonnois/kafkastreams-cep.
- Fluent Kafka Streams Test: https://github.com/bakdata/fluent-kafka-streams-tests (blog post: https://medium.com/bakdata/fluent-kafka-streams-tests-e641785171ec)
- Storm - A stream-processing framework.
- Samza - A YARN-based stream processing framework.
- Storm Spout - Consume messages from Kafka and emit as Storm tuples
- Kafka-Storm - Kafka 0.8, Storm 0.9, Avro integration
- SparkStreaming - Kafka receiver supports Kafka 0.8 and above
- Flink - Apache Flink has an integration with Kafka
- IBM Streams - A stream processing framework with Kafka source and sink to consume and produce Kafka messages
- Spring Cloud Stream - a framework for building event-driven microservices, Spring Cloud Data Flow - a cloud-native orchestration service for Spring Cloud Stream applications
- Apache Apex - Stream processing framework with connectors for Kafka as source and sink.
- Logstash - Input and Output plugins to enrich events and optionally store in Elasticsearch
- Logagent - Kafka Input and Kafka Output plugins
Hadoop Integration
- Confluent HDFS Connector - A sink connector for the Kafka Connect framework for writing data from Kafka to Hadoop HDFS
- Camus - LinkedIn's Kafka=>HDFS pipeline. This one is used for all data at LinkedIn, and works great.
- Kafka Hadoop Loader A different take on Hadoop loading functionality from what is included in the main distribution.
- Flume - Contains Kafka source (consumer) and sink (producer)
- KaBoom - A high-performance HDFS data loader
...
- Kafka Manager - A tool for managing Apache Kafka.
- kafkat - Simplified command-line administration for Kafka brokers.
- Kafka Web Console - Displays information about your Kafka cluster including which nodes are up and what topics they host data for.
- Kafka Offset Monitor - Displays the state of all consumers and how far behind the head of the stream they are.
- Capillary – Displays the state and deltas of Kafka-based Apache Storm topologies. Supports Kafka >= 0.8. It also provides an API for fetching this information for monitoring purposes.
- Doctor Kafka - Service for cluster auto healing and workload balancing.
- Cruise Control - Fully automate the dynamic workload rebalance and self-healing of a Kafka cluster.
- Burrow - Monitoring companion that provides consumer lag checking as a service without the need for specifying thresholds.
- Chaperone - An audit system that monitors the completeness and latency of data stream.
- Sematext integration for Kafka monitoring that collects and charts 200+ Kafka metrics
AWS Integration
- Automated AWS deployment
- Kafka -> S3 Mirroring tool from Pinterest.
- Alternative Kafka->S3 Mirroring tool
...
- Mozilla Metrics Service - A Kafka and Protocol Buffers based metrics and logging system
- Ganglia Integration
- SPM Sematext integration for Kafka monitoring
- Coda Hale Metric Reporter to Kafka
- kafka-dropwizard-reporter - Register built-in Kafka client and stream metrics to Dropwizard Metrics
- Metricbeat Kafka Module - Capture and ship Kafka consumergroup and partition metrics to Elasticsearch (docs)
...