Versions Compared

Old Version 7

changes.mady.by.user jay vyas

Saved on

compared with

New Version Current

changes.mady.by.user Evans Ye

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

As part of completion of BIGTOP-996, we have created a list of HOW-TOs for bigtop, which will be maintained over time.  Please notify and open a JIRA if any of these become obsolete.
Create a Bare Metal Hadoop Cluster from the latest bigtop repositories

Follow the directions on How to install Bigtop 0.8.0 Hadoop on CentOS 6 with Puppet

Create a BigTop Hadoop VM cluster: hadoop cluster  cluster reproducible, easily destroyed/recreated, configurable hadoop sandbox cluster with vagrant/docker

See bigtop-deploy/provisioner, and go into the README files underneath vagrant or docker. 

 

Create a BigTop Hadoop VM : Static one node hadoop cluster by loading a VM directly

See bigtop-deploy/, and go into the README files underneath vagrant-puppet.

Use the BigPetStore application to smoke test or demonstrate a hadoop deployment

See bigtop-bigpetstore, and follow the README file.  

Run the smoke tests + API tests for Apache BigTop's hadoop distribution on HDFS

...

which aims to improve these.

 

Run BigTop Ecosystem interoperability smoke tests (proper subset of the full bigtop smoke tests).

...

Add a package for my new hadoop ecosystem tool "XYZ" to bigtop.

This takes experience and understanding of RPM / deb packaging.   Sean Macrory is a commiter for bigtop and can help get started down this road.  Best place to start is to ping the mailing list.  For introductions to bigtop packaging you can also read some of the posts at https://blogs.apache.org/bigtop/ See Bigtop Packaging .

Learn about Apache Hadoop using BigTop

This is the most exciting thing about bigtop - it is a great sandbox for learning how hadoop really works.  For newcomers, here are 3 places where you can "dive into" bigtop to start learning about how the hadoop ecosystem really works.

1) You can start by playing with the bigpetstore app on your local machine, which can run a whole hadoop workflow for you, creating data that you can visualize in the bigpetstore web app  app.

2)   Once you "get it", you can move on to spinning up your own hadoop cluster using the vagrant recipes, or else, by directly grabbing a bigtop image from the nightly builds on the jenkins server. 

3) Finally, as you get more advanced, you can see how bigtop pulls down source from various hadoop projects, packages it into rpms, installs those rpms, and coordinates their configuration/startup with puppet, by looking in bigtop-deploy/.