Page History

...

• Automated AWS deployment
• Kafka -> S3 Mirroring tool from Pinterest.
• Alternative Kafka->S3 Mirroring tool

Logging

• syslog (1M)
  
  • syslog producer : A syslog producer that support supports both raw data and protobuf with meta data for deep analytics usage. 
  • syslog-ng (https://syslog-ng.org/) is one of the most widely used open source log collection tools, capable of filtering, classifying, parsing log data and forwarding it to a wide variety of destinations. Since its most recent release (3.7.1 Kafka is a first-class destination in the syslog-ng tool; details on the integration can be found at https://githubczanik.blogs.balabit.com/balabit2015/syslog-ng/releases/tag/syslog-ng-3.7.1) support for delivering messages to kafka is added using the official Java client libraries and syslog-ng also provides a way to set the native kafka producer configuration file11/kafka-and-syslog-ng/ .
• klogd - A python syslog publisher
• klogd2 - A java syslog publisher
• Tail2Kafka - A simple log tailing utility
• Fluentd plugin - Integration with Fluentd
• Remote log viewer
• LogStash integration - Integration with LogStash and Fluentd
• Syslog Collector written in Go
• Klogger - A simple proxy service for Kafka.
• fuse-kafka: A file system logging agent based on Kafka
• omkafka: Another syslog integration, this one in C and uses librdkafka library
• logkafka - Collect logs and send lines to Apache Kafka

...