Page History

NOTE: This Wiki is obsolete as of November 2016 and is retained for reference only.

Spark QA Infrastructure Overview

...

The following list of software packages are installed on each worker node, and required to build the myriad variations of spark:

• Java JDK 7u71 (default)
• Java JDK 6u45
• System packages (installed via yum):

• sssd 1.11.6

• htop 1.0.3

• git 1.7.1

• tree 1.5.3

• policycoreutils-python 2.0.83

• gcc 4.4.7

• bzip2 1.0.5

• docker-io 1.5.0

• glibc.i686 2.12

• glibc-devel.i686 2.12

• libstdc++.i686 4.4.7

• zlib-devel.i686 1.2.3

• ncurses-devel.i686 5.7

• Python 2.6 and python packages:
  • installed via yum:

    • python

(

• • • 2.6

default, 2

• • • .

7)

• • • 6

    • python-pip 1.3.1

• pypy

• • • numpy 1.4.1

    • scipy 0.7.2

    • python-matplotlib 0.99.1.2

    • sympy 0.6.3

    • python-nose 0.10.4

  • installed via pip:
    • unittest2 0.8.0
    • numpy: 1.9.1
• Python 2.7 and 3.4.3 packages (installed via Anaconda 2.1.0):
  • docker-compose 1.2.0
  • numpy 1.9.2
  • scipy 0.15.1
  • matplotlib 1.4.3
  • ipython 3.1.0
  • pandas 0.16.0
  • nose 1.3.6
  • sympy 0.7.6
  • pymongo 2.8

  • CherryPy 3.6.0

  • google-api-python-client 1.3.2

  • jsonpickle 0.9.2

  • pygeocoder 1.2.5

  • python-

pandas

• ipython

• unittest2

numpy

• • crontab 1.9.3

  • utm 0.4.0

  • wsgiref 0.1.2

  • xmltodict 0.9.2

• R and friends:

  • R

  • rJava
  • testthat
  • digest

  • Version : 3.1.3 (installed via yum)

  • testthat 0.9.1
  • digest 0.6.8
• pypy 2.5.1 (default, /usr/bin/pypy-2.0.2 also available)
• Maven 3.1.1 (default)
• Maven 3.0.5SBT

Hardware Configuration

The Jenkins cluster has the following hardware configurations:

...