Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: added hedwig, kafka, norbert, s4

...

  • HBase - HBase is the Hadoop database. Its an open-source, distributed, column-oriented store modeled after the Google paper, Bigtable: A Distributed Storage System for Structured Data by Chang et al.
    We use ZooKeeper for master election, server lease management, bootstrapping, and coordination between servers.
  • Hedwig Hedwig is a publish-subscribe system designed to carry large amounts of data across the internet in a guaranteed-delivery fashion from those who produce it (publishers) to those who are interested in it (subscribers).
  • Kafka Kafka is a distributed publish/subscribe messaging system
  • Norbert Partitioned routing and cluster management.
  • Mesos Mesos is a cluster management platform that provides resource sharing and isolation across distributed applications, or frameworks. For example, Hadoop MapReduce, MPI, Hypertable, and Spark
  • Neo4j - Neo4j is a Graph Database. It's a disk based, ACID compliant transactional storage engine for big graphs and fast graph traversals, using external indicies like Lucene/Solr for global searches.
    We use ZooKeeper in the Neo4j High Availability components for write-master election, read slave coordination and other cool stuff. ZooKeeper is a great and focused project - we like!
  • Rackspace - The Email & Apps team uses ZooKeeper to coordinate sharding and responsibility changes in a distributed e-mail client that pulls and indexes data for search. ZooKeeper also provides distributed locking for connections to prevent a cluster from overwhelming servers.
  • S4 S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data.
  • Yahoo! - ZooKeeper is used for a myriad of services inside Yahoo! for doing leader election, configuration management, sharding, locking, group membership etc.

...