• Created by Unknown User (starchmd), last updated on Nov 25, 2014 2 minute read

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

What is Streaming OODT?

Streaming OODT (S-OODT) is an attempt to bring technologies that support big-cluster processing and stream processing into the OODT processing system. This will enable OODT to tackle bigger data sets and new data paradigms moving into the future. The core technologies handle: stream management, cluster management, processing, file system efficiency and backward-compatibility (running standard OODT within a managed cluster).   These technologies include:

  • Apache Kafka (Stream management)
  • Apache Mesos (Cluster management)
  • Apache Spark and Spark Streaming (Processing)
  • Tachyon and Hadoop HDFS (File system efficiency)

For detailed design documentation please refer to: Streaming OODT

What is required for Streaming OODT?

Streaming OODT is designed for use on clusters of machines.  It requires at least one head node and at least one slave node.  The head node runs the main servers, and the slaves run the client services including processing slaves and HDFS slaves.  In general the cluster should have moderate local disk storage and above normal RAM on each slave node. Currently S-OODT has only been tested on linux hosts.

In order to install and run S-OODT the following items are reuired to be configured on your cluster before you install.

 

Required software:

  • Apache Maven
  • Java
  • GNU Compiler Tools (gcc, g++)
  • SVN

Required environment variables:

  • M2_HOME
  • JAVA_HOME

Using Streaming OODT

The following sections describe how to install, setup and use S-OODT.

Cluster Installation

In order to install S-OODT, first make sure you have the required software.  Then login to your head-node and perform the following steps.

Note: Root user access should not be required.

Setup Bootstrap Script
#Make a temp directory
mkdir tmp
cd tmp
#Get install script (minimal bootstrap script)
svn export https://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/
#Setup environment variables
cd setup
mv env-vars.sh.tmpl env-vars.sh

 

```

mkdir tmp
cd tmp
svn export https://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/
cd setup/
vi hosts
mv env-vars.sh.tmpl env-vars.sh
chmod +x install.sh
mkdir /data/cluster-local/software (and all)
./install.sh

```

  • No labels