...

To set up PostgreSQL and MADlib with Anaconda Python on OSX, follow the super quick start.  Otherwise, follow the regular guides for installing from binaries or compiling from source.

For developers, you may want to use the Docker image described in the Developer Guide.

Sometimes there are release specific variations of the installation procedures.  These exceptions are listed at the bottom of this page in the section called 82903224 Release Specific Installations.

MADlib works with Python 2.6 and requires python version 2.7. Currently, Python 3.x is not supported.

Currently supported database version: Postgres 9 and 10, Greenplum 4.3 and 5.X.versions: Please see this page for supported databases and OS

The following python libraries are required for their associated modules:

Deep Learning: dill, grpcio==1.39.0, protobuf==3.17.3, hyperopt==0.2.5, tensorflow == 1.14, scikit-learn==0.19

XGBoost: pandas, xgboost==0.82

KNN: scipy==1.2.1

Unit tests: pgsanity

Anchor
Anchor
Super Quick Start Super Quick Start

Super Quick Start

To set up PostgreSQL + MADlib with Anaconda Python on OSX: 

•  PYTHON=/Users/janedoe/anaconda/bin/python

•  brew install postgresql --with-python

•  brew services start postgresql

• - Set up database and roles

• python 

•  Install Postgres with the Python extension specified (i.e., --with-python), as described here in the PostgreSQL documentation. Note that previously you could install postgres with python support using brew by running 'brew install postgresql --with-python' but passing the '--with-python' flag is not supported anymore.
  

•  Set up database and roles

•  Install the .dmg - Install the .dmg of latest madlib downloaded from MADlib website httphttps://madlib.apache.org/download.html

•  /usr/local/madlib/bin/madpack -s madlib -p postgres install

...

MADlib requires the GNU M4 Unix macro processor which must be present for installation to succeed.

If the environment variables listed below are defined, it can save you some typing.

...

1. Download the MADlib binary
   • For Postgres: OS X and Linux binaries can be found on the MADlib download page
   • For Greenplum: Linux .gppkg binaries can be found on Pivotal Network in the "Greenplum Advanced Analytics Group"
     • NOTE: the above .gppkg binaries work for both open and closed source Greenplum and can be downloaded by anybody (after creating a Pivotal Network account)
2. Install the package.
   
   1. Postgres:
      • on OSX double click the installer package

      • on Redhat / CentOS run the following as root:

        Code Block
        yum install <madlib_package> --nogpgcheck

        or

        Code Block
        language bash
        rpm -i <madlib_package>

        
        

   2. Greenplum:

      • on Redhat / CentOS run the following as gpadmin:

        Code Block
        language bash
        gppkg -i <madlib_package>

        
        

   3. NOTE: if you are using an rpm package on a CentOS 5 system, please add --no-deps flag to the command.
3. Ensure that the environment is setup for your database deployment and that the database is up and running.

   • Ensure that psql, postgres, and pg_config are in your path

     Code Block
     language bash
     which psql postgres pg_config

     
     

   • Ensure that the database is started and running

     Code Block
     language bash
     psql -c 'select version()'

     The above may need user/port/password setting depending on how the database has been configured.

4. Run the MADlib deployment utility to deploy MADlib into each database that you want to use it:

   • Postgres:

     Code Block
     language bash
     /usr/local/madlib/bin/madpack -s madlib –p postgres install

     if environment variables are defined. Otherwise use a fully defined connection string:

     Code Block
     language bash
     /usr/local/madlib/bin/madpack -s madlib -p postgres -c [user[/password]@][host][:port][/database] install

     
     

   • Greenplum Database:

     Code Block
     language bash
     /usr/local/madlib/bin/madpack –p greenplum install

     The above may need user/port/password setting depending on how the database has been configured.

     Run the MADlib madpack deployment utility to install MADlib into each database that you want to use it in:

5. After installation gpadmin should grant all privileges on schema madlib to users who will be accessing MADlib functions. Otherwise, users will get "ERROR: permission denied for schema MADlib."  Also, install checks (see next step below) will fail if CREATE TEMP TABLE privileges are not granted on the schema where MADlib is installed. See the PostgreSQL docs for information on schemas and privileges.

6. Test your installation

...

pgxn install madlib
pgxn load madlib 

[ 86%] Performing build step for 'EP_boost'
Ignored: make
[ 86%] Performing install step for 'EP_boost'
Ignored: make
[ 86%] Completed 'EP_boost'
[ 86%] Built target EP_boost
make[1]: *** [all] Error 2
make: *** [all] Error 2
ERROR: command returned 2: make PG_CONFIG=/usr/local/pg10/bin/pg_config all

MAKEFLAGS='-j1' pgxn install madlib
pgxn load madlib 

brew install boost@1.60
export BOOST_INCLUDEDIR=/usr/local/opt/boost@1.60/include/

Anchor
Anchor
Compile From Source Compile From Source

Compiling From Source

...

• gcc and g++
  • For OS X, Clang will work for compiling the source, but not for the documentation.
  • NOTE: on Ubuntu 16.04, you will need to use GCC 4 and not the default compiler (GCC 5) to run MADlib. Refer to MADLIB-1068 for details. The outstanding GCC 5 issue is being tracked under MADLIB-1145.
    
  •  To compile on newer versions of XCode we need to enable the CXX11 flag. Setting -DCXX11=1 during cmake, will auto-download Boost 1.75.0 if Boost > 1.65.0 is not found on the system.
    Note: Setting -DCXX11=1 will enable C++11, which is not fully supported, i.e, MADlib compiles but some install-check/dev-check tests may fail.
• python 2.6 or 2.7
  • python 3.x is not currently supported
  python 2.6 or 2.7
  • python 3.x is not currently supported by MADlib.
• cmake
  • NOTE: the latest version of cmake might cause issues. Please try cmake 3.5.2 in case you get an error or a segmentation fault.
  • NOTE: On Centos 6 (possibly other Linux variants), we have seen occasions where cmake will have issues running (seg fault) if the greenplum_path.sh file has been sourced prior to the cmake execution. If you encounter issues, you can use ldd on the cmake executable to confirm dynamic libraries are picked up from the Greenplum installation directories. If this is the case, start a new shell in which the greenplum_path.sh file is not sourced in your current running shell session. You can reference MADLIB-1093 for additional details.
• An installed version of Greenplum Database or PostgreSQL (64-bit) 9.2+ with plpython support enabled.  
  • NOTE: plpython may not be enabled in Postgres by default.

...

• Postgres:

  Code Block
  language bash
  $BUILD_ROOT/src/bin/madpack -s madlib –p postgres install

  if environment variables are defined. Otherwise use a fully defined connection string:

  Code Block
  language bash
  $BUILD_ROOT/src/bin/madpack -s madlib -p postgres -c [user[/password]@][host][:port][/database] install

  
  

• Greenplum Database:

  Code Block
  language bash
  $BUILD_ROOT/src/bin/madpack –p greenplum install

  The above may need user/port/password setting depending on how the database has been configured.

• To install:

  Code Block
  language bash
  $BUILD_ROOT/src/bin/madpack -p postgres -c [user[/password]@][host][:port][/database] install

  
  

• To make sure that the installation is successful:

  Code Block
  language bash
  $BUILD_ROOT/src/bin/madpack -p postgres -c [user[/password]@][host][:port][/database] install-check

  For more information on the usage of madpack

  ][host][:port][/database] install-check

  
  

• For more information on the usage of madpack:
  

  Code Block
  language bash
  $BUILD_ROOT/src/bin/madpack --help

  
  

Anchor
Env variables Env variables

Defining environment variables

The variables below will be automatically used by the madpack installer if no connection string is provided:

1. User: PGUSER or USER (defaults to OS username)
2. Password: PGPASSWORD (defaults to empty)
3. Host: PGHOST (defaults to 'localhost')
4. Database: PGDATABASE (defaults to OS username)
5. Port: PGPORT (defaults to 5432)

An example of deploying MADlib using the environment variables:

export PGPORT=5430
export PGHOST=127.0.0.1
export PGDATABASE=madlibtest
$BUILD_ROOT/src/bin/madpack -

...

p postgres install

Anchor

...

GPDB Variables GPDB Variables

...

Defining

...

GPDB variables

The variables below will be automatically used by the madpack installer if no connection string is provided:

1. User: PGUSER or USER (defaults to OS username)
2. Password: PGPASSWORD (defaults to empty)
3. Host: PGHOST (defaults to 'localhost')
4. Database: PGDATABASE (defaults to OS username)
5. Port: PGPORT (defaults to 5432)

An example of deploying MADlib using the environment variables:

can be set in GPDB in case memory-related issues show up. Feel free to adjust them based on the specifics of the installed system.

set max_statement_mem='50GB';
set statement_mem='50GB';
set memory_spill_ratio=80;
set gp_resqueue_memory_policy=auto;
set work_mem='4GB';
set gp_vmem_protect_limit=20000

export PGPORT=5430
export PGHOST=127.0.0.1
export PGDATABASE=madlibtest
$BUILD_ROOT/src/bin/madpack -p postgres install

Anchor
Upgrading MADlib gppkg Upgrading MADlib gppkg

Upgrading MADlib gppkg

...