As part of completion of BIGTOP-996, we have created a list of HOW-TOs for bigtop, which will be maintained over time.  Please notify and open a JIRA if any of these become obsolete.
Create a Bare Metal Hadoop Cluster from the latest bigtop repositories

Follow the directions on How to install Bigtop 0.8.0 Hadoop on CentOS 6 with Puppet

Create a BigTop Hadoop VM cluster: hadoop cluster reproducible, easily destroyed/recreated, configurable hadoop sandbox cluster with vagrant/docker

See provisioner, and go into the README files underneath vagrant or docker. 

Use the BigPetStore application to smoke test or demonstrate a hadoop deployment

See bigtop-bigpetstore, and follow the README file. 

Run the smoke tests + API tests for Apache BigTop's hadoop distribution on HDFS

See the README.md file in the root of bigtop/.  It has basic directions for maven based smoke tests.  Also track the BIGTOP-1222 JIRA,

which aims to improve these.

Run BigTop Ecosystem interoperability smoke tests (proper subset of the full bigtop smoke tests).

This is the BIGTOP-1222 JIRA which is underway.  For now, you should run a subset of the bigtop maven tests, and

customize them by commenting out submodules as necessary from test-execution/pom.xml.

Add a package for my new hadoop ecosystem tool "XYZ" to bigtop.

This takes experience and understanding of RPM / deb packaging.  See Bigtop Packaging .

Learn about Apache Hadoop using BigTop

This is the most exciting thing about bigtop - it is a great sandbox for learning how hadoop really works.  For newcomers, here are 3 places where you can "dive into" bigtop to start learning about how the hadoop ecosystem really works.

1) You can start by playing with the bigpetstore app on your local machine, which can run a whole hadoop workflow for you, creating data that you can visualize in the bigpetstore web app.

2) Once you "get it", you can move on to spinning up your own hadoop cluster using the vagrant recipes, or else, by directly grabbing a bigtop image from the nightly builds on the jenkins server. 

3) Finally, as you get more advanced, you can see how bigtop pulls down source from various hadoop projects, packages it into rpms, installs those rpms, and coordinates their configuration/startup with puppet, by looking in bigtop-deploy/.

  • No labels