THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Kafka has a built-in framework called Kafka Connect for writing sources and sinks that either continuously ingest data into Kafka or continuously ingest data in Kafka into external systems. The connectors themselves for different applications or data systems are federated and maintained separately from the main code base. You can find a An externally hosted list of available connectors is maintained by Confluent at the Confluent Hub.
Distributions & Packaging
- Confluent: Confluent Cloud (SaaS, fully managed) and Confluent Platform (software download for self-managed/on-premise)
- Cloudera distribution - - http://confluent.io/product/. Downloads - http://confluent.io/downloads/.Cloudera Kafka https://www.cloudera.com/products/open-source/apache-hadoop/apache-kafka.htmlStratio Kafka source for ubuntu
- httpEmblocSoft distribution and CPFA training - https://repositorywww.stratioemblocsoft.com/sdspages/1.1en/ubuntusolutions/13.10/binary/ and for RHEL http://repository.stratio.com/sds/1.1/RHEL/kafka/support/
- IBM Event Streams - https://www.ibm.com/cloud/event-streams - Apache Kafka on premise and the public cloud
- Strimzi - http://strimzi.io/ - Apache Kafka Operator for Kubernetes and Openshift. Downloads and Helm Chart - https://github.com/strimzi/strimzi-kafka-operator/releases/latest
- TIBCO Messaging - Apache Kafka Distribution - https://www.tibco.com/products/apache-kafka Downloads - https://www.tibco.com/products/tibco-messaging/downloads
...
- Kafka Streams - the built-in stream processing library of the Apache Kafka project
- Kafka Streams Ecosystem:
- Complex Event Processing (CEP): https://github.com/fhussonnois/kafkastreams-cep.
- Fluent Kafka Streams Test: https://github.com/bakdata/fluent-kafka-streams-tests (blog post: https://medium.com/bakdata/fluent-kafka-streams-tests-e641785171ec)
- Azkarra Streams - A lightweight java framework to make it easy to build and manage streaming microservices based on Kafka Streams.
- Storm - A stream-processing framework.
- Samza - A YARN-based stream processing framework.
- Storm Spout - Consume messages from Kafka and emit as Storm tuples
- Kafka-Storm - Kafka 0.8, Storm 0.9, Avro integration
- SparkStreaming - Kafka receiver supports Kafka 0.8 and above
- Flink - Apache Flink has an integration with Kafka
- IBM Streams - A stream processing framework with Kafka source and sink to consume and produce Kafka messages
- Spring Cloud Stream - a framework for building event-driven microservices, Spring Cloud Data Flow - a cloud-native orchestration service for Spring Cloud Stream applications
- Apache Apex - Stream processing framework with connectors for Kafka as source and sink.
- Logstash - Input and Output plugins to enrich events and optionally store in Elasticsearch
- Logagent - Kafka Input and Kafka Output plugins
...
- Elasticsearch - This project, Kafka Standalone Consumer will read the messages from Kafka, processes and index them in Elasticsearch. There are also several Kafka Connect connectors for Elasticsearch.
- Presto - The Presto Kafka connector allows you to query Kafka in SQL using Presto.
- Hive - Hive SerDe that allows querying Kafka (Avro only for now) using Hive SQL
- OpenMLDB Kafka Connector - This project allows you to define and extract features from data streams using SQL for ML applications.
Management Consoles
- Kafka Manager - A tool for managing Apache Kafka.
- kafkat - Simplified command-line administration for Kafka brokers.
- Kafka Web Console - Displays information about your Kafka cluster including which nodes are up and what topics they host data for.
- Kafka Offset Monitor - Displays the state of all consumers and how far behind the head of the stream they are.
- Capillary – Displays the state and deltas of Kafka-based Apache Storm topologies. Supports Kafka >= 0.8. It also provides an API for fetching this information for monitoring purposes.
- Doctor Kafka - Service for cluster auto healing and workload balancing.
- Cruise Control - Fully automate the dynamic workload rebalance and self-healing of a Kafka cluster.
- Burrow - Monitoring companion that provides consumer lag checking as a service without the need for specifying thresholds.
- Chaperone - An audit system that monitors the completeness and latency of data stream.
- Sematext integration for Kafka monitoring that collects and charts 200+ Kafka metrics
- Xinfra Monitor - A framework that monitors and exposes metrics showing availability and performance of Kafka clusters and mirrored pipelines.
AWS Integration
- Automated AWS deployment
- Kafka -> S3 Mirroring tool from Pinterest.
- Alternative Kafka->S3 Mirroring tool
...