Spark QA Infrastructure Overview

Spark's Jenkins CI and QA infrastructure is hosted by UC Berkeley's AMPLab.  The machines live in the IS&T datacenter.  We have one Jenkins master, and currently eight Jenkins worker nodes.

Jenkins Account Requests

Please see the AMPLab Jenkins wiki for instructions.

Jenkins Worker Configuration

The following list of software packages are installed on each worker node, and required to build the myriad variations of spark:

• Java JDK 7u71 (default)
• Java JDK 6u45
• System packages (installed via yum):

• sssd 1.11.6

• htop 1.0.3

• git 1.7.1

• tree 1.5.3

• policycoreutils-python 2.0.83

• gcc 4.4.7

• bzip2 1.0.5

• docker-io 1.5.0

• glibc.i686 2.12

• glibc-devel.i686 2.12

• libstdc++.i686 4.4.7

• zlib-devel.i686 1.2.3

• ncurses-devel.i686 5.7

• Python 2.6 and packages:
  • installed via yum:

    • python 2.6.6

    • python-pip 1.3.1

    • numpy 1.4.1

    • scipy 0.7.2

    • python-matplotlib 0.99.1.2

    • sympy 0.6.3

    • python-nose 0.10.4

  • installed via pip:
    • unittest2 0.8.0
    • numpy: 1.9.1
• Python 2.7 and 3.4.3 packages (installed via Anaconda 2.1.0):
  • docker-compose 1.2.0
  • numpy 1.9.2
  • scipy 0.15.1
  • matplotlib 1.4.3
  • ipython 3.1.0
  • pandas 0.16.0
  • nose 1.3.6
  • sympy 0.7.6
  • pymongo 2.8

  • CherryPy 3.6.0

  • google-api-python-client 1.3.2

  • jsonpickle 0.9.2

  • pygeocoder 1.2.5

  • python-crontab 1.9.3

  • utm 0.4.0

  • wsgiref 0.1.2

  • xmltodict 0.9.2

• R and friends:

  • R Version : 3.1.3 (installed via yum)

  • testthat 0.9.1
  • digest 0.6.8
• pypy 2.5.1 (default, /usr/bin/pypy-2.0.2 also available)
• Maven 3.1.1 (default)
• Maven 3.0.5

Hardware Configuration

The Jenkins cluster has the following hardware configurations:

Master:

• 1 Intel® R2308GZ4GC Server System
  • 1 Intel® S2600GZ Server Board
  • 4x Intel® i350 Gigabit Ethernet
  • 2x Intel® 82599ES 10Gigabit Ethernet
  • 1x Intel® Intel Sandy Bridge E5-2670 2.6Ghz CPU (16 cores w/HT)
  • 1x Intel® Integrated RAID Module RMS25CB040
• 4x Seagate Constellation ES.3 3TB SAS HDD as a 2×2 RAID 10
• 128GB - 8x Samsung M393B2G70QH0-CK0 1600Mhz DDR3 16GB SDRAM DIMMs
• CentOS 6.6

Workers:

• 2 Intel® H2216JFQKR Server Systems (essentially 4-node blade systems)
• Each contains 4 independent Intel® HNS2600JFQ Compute Modules
  • 1 Intel® S2600JFQ Server Board
  • 2x Intel® i350 Gigabit Ethernet
  • 1x Mellanox® ConnectX-3 40Gb/s Infiniband (can double as 10Gb/s Ethernet)
  • 2x Intel® Sandy Bridge E5-2670 2.6Ghz CPUs (32 cores w/HT)
• 128GB - 8x Samsung M393B2G70BH0-CK0 1600Mhz DDR3 16GB SDRAM DIMMs
• 1x Innodisk SATADOM-ML 3SE 16GB SSD - for boot/OS
• 4x Seagate Constellation.2 ST91000640NS 1TB SATA HDD - for Data
• CentOS 6.6