Versions Compared

Old Version 45

changes.mady.by.user Frank McQuillan

Saved on

compared with

New Version Current

changes.mady.by.user Orhan Kislal

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

To set up PostgreSQL and MADlib with Anaconda Python on OSX, follow the super quick start.  Otherwise, follow the regular guides for installing from binaries or compiling from source.

For developers, you may want to use the Docker image described in the Developer Guide.

Sometimes there are release specific variations of the installation procedureprocedures.  These exceptions are listed at the bottom of this page in the section called Release Specific Installations.

Please note that VM's with MADlib pre-installed are available from the Pivotal download site:  Greenplum database sandbox VM and Pivotal HDB sandbox VM.  Using the VMs may be an alternative to following the installation steps described on this page for some folks.

MADlib requires python version 2.7. Currently, Python 3.x is not supported.

Currently supported database versions: Please see this page for supported databases and OS

The following python libraries are required for their associated modules:

Deep Learning: dill, grpcio==1.39.0, protobuf==3.17.3, hyperopt==0.2.5, tensorflow == 1.14, scikit-learn==0.19

XGBoost: pandas, xgboost==0.82

KNN: scipy==1.2.1

Unit tests: pgsanityMADlib works with Python 2.6 and 2.7.  Currently, Python 3.x is not supported.

Anchor
Super Quick Start
Super Quick Start
Super Quick Start

To set up PostgreSQL + MADLib MADlib with Anaconda Python on OSX: 

...

  • PYTHON=/Users/janedoe/anaconda/bin/

...

  • python 
  •  Install Postgres with the Python extension specified (i.e., --with-python), as described here in the PostgreSQL documentation. Note that previously you could install postgres with python support using brew by running 'brew install postgresql --with-python' but passing the '--with-python' flag is not supported anymore.

  •  Set up database and roles

  •  Install the .dmg of latest madlib downloaded from MADlib website https://madlib.

...

...

  •  /usr/local/madlib/bin/madpack

...

  • -s

...

  • madlib

...

  • -p

...

  • postgres

...

  • install

Anchor
Quick Start With Binaries
Quick Start With Binaries
Quick Start With Binaries

...

  • PostgreSQL
  • Greenplum databasePivotal HDB/Apache HAWQ (incubating)

MADlib requires the GNU M4 Unix macro processor which must be present for installation to succeed.

If the environment variables listed below are defined, it can save you some typing.

Postgres platform notes:

Code Block
 /usr/local/madlib/bin/madpack -s madlib -p postgres install
madpack.py : INFO : Detected PostgreSQL version 9.5.
madpack.py : INFO : *** Installing MADlib ***
madpack.py : INFO : MADlib tools version = 1.9.1 (//usr/local/madlib/Versions/1.9.1/bin/../madpack/madpack.py)
madpack.py : INFO : MADlib database version = None (host=localhost:5432, db=postgres, schema=madlib)
madpack.py : INFO : Testing PL/Python environment...
madpack.py : INFO : > Creating language PL/Python...
madpack.py : ERROR : SQL command failed:
SQL: CREATE LANGUAGE plpythonu;
ERROR: could not access file "$libdir/plpython2": No such file or directory
madpack.py : ERROR : Cannot create language plpythonu. Please check Stopping installation...
madpack.py : ERROR : MADlib installation failed.

Installing MADlib

  1. Download the MADlib binary

Install the package at the OS level.

...

  • on OSX double click the installer package

  • on Redhat / CentOS run the following as root:

    Code Block
    yum install <madlib_package> --nogpgcheck

...

  • on Redhat / CentOS run the following as gpadmin:

    Code Block
    languagebash
    gppkg -i <madlib_package>

    Please note:  For installation of MADlib on GPDB 4.3.10, an additional step is required after running gppkg and before running the madpack utility.  See here for details or refer to the readme file with the Pivotal Network MADlib package download.

if you
                have configured and installed portid (your platform) with
                `--with-python` option. Stopping installation...
madpack.py : ERROR : MADlib installation failed

Installing MADlib

...

on Redhat / CentOS run the following as gpadmin:

...

languagebash
  1. Download the MADlib binary
    • For Postgres: OS X and Linux binaries can be found on the MADlib download page
    • For Greenplum: Linux .gppkg binaries can be found on Pivotal Network in the "Greenplum Advanced Analytics Group"
      • NOTE: the above .gppkg binaries work for both open and closed source Greenplum and can be downloaded by anybody (after creating a Pivotal Network account)
  2. Install the package.
    1. Postgres:
      • on OSX double click the installer package

      • on Redhat / CentOS run the following as root:

        Code Block
        yum install <madlib_package> --nogpgcheck

        or

        Code Block
        languagebash
        rpm -i <madlib_package>


    2. Greenplum:

      • on Redhat / CentOS run the following as gpadmin:

        Code Block
        languagebash
      • gppkg -i <madlib_package>

...


    2. NOTE: if you are using an rpm package on a CentOS 5 system, please add --no-deps flag to the command.
  2. Ensure that the environment is setup for your database deployment and that the database is up and running.

    • Ensure that psql, postgres, and pg_config are in your path

      Code Block
      languagebash
      which psql
which postgres 
which pg_config


    • Ensure that the database is started and running

      Code Block
      languagebash
      psql -c 'select version()'

      The above may need user/port/password setting depending on how the database has been configured.

  3. Run the MADlib deployment utility to install deploy MADlib into each database that you want to use it:

    • Postgres:

      Code Block
      languagebash
      /usr/local/madlib/bin/madpack -s madlib –p postgres install

      if environment variables are defined. Otherwise use a fully defined connection string:

      Code Block
      languagebash
      /usr/local/madlib/bin/madpack -s madlib -p postgres -c [user[/password]@][host][:port][/database] install


    • Greenplum Database:

      Code Block
      languagebash
      /usr/local/madlib/bin/madpack –p greenplum install

      The above may need user/port/password setting depending on how the database has been configured.

    • HDB/HAWQ:

      Code Block
      languagebash
      /usr/local/madlib/bin/madpack –p hawq install

      The above may need user/port/password setting depending on how the database has been configured.

    For more information on madpack:

    Code Block
    languagebash
    /usr/local/madlib/bin/madpack --help

    Help output for madpack is also attached to this wiki page for your reference.

    • Run the MADlib madpack deployment utility to install MADlib into each database that you want to use it in:

  4. After installation gpadmin should grant all privileges on schema madlib to users who will be accessing MADlib After installation gpadmin should grant all privileges on schema madlib to users who will be accessing MADlib functions. Otherwise, users will get "ERROR: permission denied for schema MADlib."  See the PostgreSQL docs for  Also, install checks (see next step below) will fail if CREATE TEMP TABLE privileges are not granted on the schema where MADlib is installed. See the PostgreSQL docs for information on schemas and privileges.

  5. Test your installation

    • Postgres:

      Code Block
      languagebash
      /usr/local/madlib/bin/madpack -s madlib –p postgres install-check


    • Greenplum Database:

      Code Block
      languagebash
      /usr/local/madlib/bin/madpack –p greenplum install-check

      The above may need user/port/password setting depending on how the database has been configured.

      HDB/HAWQ:

      Code Block
      languagebash
      /usr/local/madlib/bin/madpack –p hawq install-check

      The above may need user/port/password setting depending on how the database has been configured 

      Please note that if the optimizer_control GUC is set to off in Greenplum, the following install checks will fail, and these MADlib functions will not work:  decision tree, random forest, LDA , k-Means, PMML export for decision tree, PMML export for random forest.  This will be fixed in a future release (MADLIB-1109).  The parameter optimizer_control controls whether the server configuration parameter optimizer can be changed. The parameter optimizer controls whether the GPORCA optimizer is enabled when running SQL queries.

Anchor
PGXN
PGXN
Installing from PGXN (PostgreSQL)

...

Requirements for installing MADlib:

  • gcc and g++ (For OSX, Clang will work for compiling the source, but not for documentation.). Note: C++11 is not fully supported yet.  
  • m4
  • patch
  • cmake
  • pgxn installed
  • PostgreSQL (64-bit) 9.2+ with plpython support enabled. Note: plpython may not be enabled in Postgres by default.
 


Use below commnd command to install and load the latest MADlib package uploaded on PGXN.  

Code Block
languagebash
pgxn install madlib
pgxn load madlib 

 

...

 If you see the following error, it's likely that you are using Parallel Execution flags for make. 

Code Block
languagebash
[ 86%] Performing build step for 'EP_boost'
Ignored: make
[ 86%] Performing install step for 'EP_boost'
Ignored: make
[ 86%] Completed 'EP_boost'
[ 86%] Built target EP_boost
make[1]: *** [all] Error 2
make: *** [all] Error 2
ERROR: command returned 2: make PG_CONFIG=/usr/local/pg10/bin/pg_config all

You can run this as a workaround

Prerequisites

Requirements for installing MADlib:

  • gcc (For OSX, Clang will work for compiling the source, but not for documentation.)
  • An installed version of HDB/HAWQ, Greenplum Database 4.2+ or PostgreSQL (64-bit) 9.2+ with plpython support enabled.  Note: plpython may not be enabled in Postgres by default.  
  • MADlib works with Python 2.6 and 2.7.  Currently, Python 3.x is not supported.

Installing MADlib

In the $MADLIB_ROOT directory (location of MADlib source) run the following commands:

Code Block
languagebash
mkdir build 
cd build 
cmake .. 
make

Above, we built the executables in the build folder. This can, however, be any user-named folder (henceforth called $BUILD_ROOT).

Deploying MADlib

Deploy MADlib into the database with MADlib package manager madpack located under $BUILD_ROOT/src/bin.

...

:

Code Block
languagebash

...

$BUILD_ROOT/src/bin/madpack -p postgres -c [user[/password]@][host][:port][/database] install
MAKEFLAGS='-j1' pgxn install madlib
pgxn load madlib 

Or, if you want to use parallel execution, you can also install Boost 1.60 yourself, and tell cmake where to find it.

For example, on OSX that looks like this

...

:


Code Block
languagebash

...

$BUILD_ROOT/src/bin/madpack -p postgres -c [user[/password]@][host][:port][/database] install-check

...

For more information on the usage of madpack:

Code Block
languagebash
$BUILD_ROOT/src/bin/madpack --help
brew install boost@1.60
export BOOST_INCLUDEDIR=/usr/local/opt/boost@1.60/include/

Anchor
Compile From Source
Compile From Source
Compiling From Source

Prerequisites

Requirements for installing MADlib:

  • gcc and g++
    • For OS X, Clang will work for compiling the source, but not for the documentation. To compile on newer versions of XCode we need to enable the CXX11 flag. Setting -DCXX11=1 during cmake, will auto-download Boost 1.75.0 if Boost > 1.65.0 is not found on the system.
      Note: Setting -DCXX11=1 will enable C++11, which is not fully supported, i.e, MADlib compiles but some install-check/dev-check tests may fail.
  • python 2.6 or 2.7
    • python 3.x is not currently supported by MADlib.
  • cmake
    • NOTE: the latest version of cmake might cause issues. Please try cmake 3.5.2 in case you get an error or a segmentation fault.
    • NOTE: On Centos 6 (possibly other Linux variants), we have seen occasions where cmake will have issues running (seg fault) if the greenplum_path.sh file has been sourced prior to the cmake execution. If you encounter issues, you can use ldd on the cmake executable to confirm dynamic libraries are picked up from the Greenplum installation directories. If this is the case, start a new shell in which the greenplum_path.sh file is not sourced in your current running shell session. You can reference MADLIB-1093 for additional details.
  • An installed version of Greenplum Database or PostgreSQL (64-bit) 9.2+ with plpython support enabled.  
    • NOTE: plpython may not be enabled in Postgres by default.

Installing MADlib

In the $MADLIB_ROOT directory (location of the MADlib source) run the following commands:

Code Block
languagebash
mkdir build 
cd build 
cmake .. 
make -j8 # if this causes issues, switch back to a plan `make`

Above, we built the executables in the build folder. This can, however, be any user-named folder (henceforth called $BUILD_ROOT).

Deploying MADlib

Deploy MADlib into the database with MADlib package manager madpack located under $BUILD_ROOT/src/bin.

Run the MADlib deployment utility to install MADlib into each database that you want to use it:

  • Postgres:

    Code Block
    languagebash
    $BUILD_ROOT/src/bin/madpack -s madlib –p postgres install

    if environment variables are defined. Otherwise use a fully defined connection string:

    Code Block
    languagebash
    $BUILD_ROOT/src/bin/madpack -s madlib -p postgres -c [user[/password]@][host][:port][/database] install


  • Greenplum Database:

    Code Block
    languagebash
    $BUILD_ROOT/src/bin/madpack –p greenplum install

    The above may need user/port/password setting depending on how the database has been configured.

  • To install:

    Code Block
    languagebash
    $BUILD_ROOT/src/bin/madpack -p postgres -c [user[/password]@][host][:port][/database] install


  • To make sure that the installation is successful:

    Code Block
    languagebash
    $BUILD_ROOT/src/bin/madpack -p postgres -c [user[/password]@][host][:port][/database] install-check


  • For more information on the usage of madpack:

    Code Block
    languagebash
    $BUILD_ROOT/src/bin/madpack --help


Anchor
Env variables
Env variables
Defining environment variables

The variables below will be automatically used by the madpack installer if no connection string is provided:

  1. User: PGUSER or USER (defaults to OS username)
  2. Password: PGPASSWORD (defaults to empty)
  3. Host: PGHOST (defaults to 'localhost')
  4. Database: PGDATABASE (defaults to OS username)
  5. Port: PGPORT (defaults to 5432)

An example of deploying MADlib using the environment variables:

Code Block
languagebash
export PGPORT=5430
export PGHOST=127.0.0.1
export PGDATABASE=madlibtest
$BUILD_ROOT/src/bin/madpack -p postgres install

Anchor
GPDB Variables
GPDB Variables
Defining GPDB variables

The variables below can be set in GPDB in case memory-related issues show up. Feel free to adjust them based on the specifics of the installed system.

Code Block
languagesql
set max_statement_mem='50GB';
set statement_mem='50GB';
set memory_spill_ratio=80;
set gp_resqueue_memory_policy=auto;
set work_mem='4GB';
set gp_vmem_protect_limit=20000

Anchor
Upgrading MADlib gppkg
Upgrading MADlib gppkg
Upgrading MADlib gppkg

  1. Download the MADlib binary

Upgrade MADlib gppkg.

  • Greenplum Database:

    • Upgrading gppkg to a higher version of MADlib:
      For example, upgrading from 1.15.1 to 1.16

      on Redhat / CentOS run the following as gpadmin:

      Code Block
      languagebash
      gppkg -u <madlib_package_upgrading_to>

      Upgrade the MADlib deployment in the database

      Code Block
      languagebash
      madpack -p <platform> -c <connection> upgrade


    • Upgrading gppkg for the same version of MADlib:
      For example, upgrading from madlib_gppkg_1.16+1 to madlib_gppkg_1.16+2

      on Redhat / CentOS run the following as gpadmin:

      Code Block
      languagebash
      gppkg -u <madlib_package_upgrading_to>

      MADlib deployment in the database does not need to be upgraded as the MADlib version has not changed.

Anchor
Release Specific Installations
Release Specific Installations
Release Specific Installations

Sometimes there are release specific variations of the installation procedures.  These exceptions are listed in this section.

06/27/19 - Upgrading MADlib from 1.15

Currently, upgrading the rpm from 1.15 using rpm -U does not work due to a change in the rpm post uninstall script in MADlib version 1.15.1.  Below are the steps to follow to upgrade from MADlib version 1.15:

  1. Remove existing MADlib rpm (this does not affect the database in any way)
    rpm -e <madlib 1.15 package name>
  2. Remove old MADlib files
    rm -rf /usr/local/madlib/Versions
  3. Install the MADlib 1.15.1 or 1.16 rpm
    rpm -i <madlib 1.16 package name>
  4. Upgrade the MADlib deployment in the database
    madpack -p <platform> -c <connection> upgrade

01/11/18 - Upgrading MADlib to 1.13

The upgrade to v1.13 has a minor problem with some leftover functions. The issue can be fixed with the following commands before running the regular madpack upgrade command.

Code Block
languagebash
psql <<DB_NAME>> -c "DROP FUNCTION IF EXISTS <<SCHEMA>>.knn(VARCHAR);"
psql <<DB_NAME>> -c "DROP FUNCTION IF EXISTS <<SCHEMA>>.knn();"

<<DB_NAME>> denotes the name of the database.

<<SCHEMA>> denotes the name of the madlib schema.

We have also attached a script to this wiki page  called 'fix_upgrade.sh' that you can use.

11/30/16 - Installation of MADlib 1.9.1 on GPDB 4.3.11

The procedure exactly the same as described below for installation of MADlib on GPDB 4.3.10 .

10/19/16 - Installation of MADlib 1.9.1 on GPDB 4.3.10

...

The variables below will be automatically used by the madpack installer if no connection string is provided:

  1. User: PGUSER or USER (defaults to OS username)
  2. Password: PGPASSWORD (defaults to empty)
  3. Host: PGHOST (defaults to 'localhost')
  4. Database: PGDATABASE (defaults to OS username)
  5. Port: PGPORT (defaults to 5432)

An example of deploying MADlib using the environment variables:

Code Block
languagebash
export PGPORT=5430
export PGHOST=127.0.0.1
export PGDATABASE=madlibtest
$BUILD_ROOT/src/bin/madpack -p postgres install

 

...

10/19/16 - Installation of MADlib on GPDB 4.3.10

This is an important note for installation of MADlib on GPDB 4.3.10.  It does not apply to any other releases.

1) Fix madpack install utility
* issue: After gppkg installation MADlib, you must run the script 
fix_madpack.sh BEFORE running the madpack utility (see below)

2) install checks
* issue: some failures may happen on MADlib install checks,  however the MADlib install actually completed OK.

This is a poor customer experience that will be fixed in the next release. On the positive side, once the installation is done, MADlib should work OK.

------------------------------

More on fixing madpack from #1 above:

After gppkg installation MADlib, you must run the script 
fix_madpack.sh BEFORE running the madpack utility.
The syntax for fix_madpack.sh is below.

This can be somewhat confusing because after gppkg
installation, you will get a message on the console
that says:

“Please run the following command to deploy MADlib
usage: madpack install [-s schema_name] -p hawq -c user@host:port/database
etc...”

So the correct order of operations is:

1. gppkg install of MADlib
2. run fix_madpack.sh
3. run madpack utility

*****************************************************
COMMAND NAME: fix_madpack.sh
*****************************************************

Script to fix a MADlib installation issue on GPDB 4.3.10.

This script patches a line in madpack.py, the MADlib installation
script. A backup of the original file is created in the same folder as
madpack.py called 'madpack.py.orig'.

*****************************************************
SYNOPSIS
*****************************************************

fix_madpack.sh [--prefix <MADLIB_INSTALL_PATH>]

fix_madpack.sh -h

...

The following tasks should be performed prior to executing this script:

* Set $GPHOME to the correct GPDB installation directory containing MADlib
OR
* Set MADlib installation path using the --prefix option

...

--prefix <MADLIB_INSTALL_PATH>
Optional. Expected MADlib installation path. If not set, the default value
${GPHOME}/madlib is used.

-h | -? | --help
Displays the online help.

...

/home/gpadmin/madlib/fix_madpack.sh --prefix /usr/local/gpdb/madlib

...

This is an important note for installation of MADlib on
HDB/HAWQ 2.0.1.

1) gppkg
* issue: does not end cleanly after installing MADlib so you need to manually exit via ctl-z, however the MADlib install actually completed OK.

GPDB 4.3.10.  It does not apply to any other releases.

1) Fix madpack install utility2) QuickLZ compression
* issue: After gppkg installation MADlib, you must run the script 
removefix_compressionmadpack.sh BEFORE running the madpack utility (see below).  The script is downloadable from the Pivotal Network.

23) install checks
* issue: some failures may happen on MADlib install checks,  however the MADlib install actually completed OK.

...

------------------------------

More on removing compression fixing madpack from #2 #1 above:

After gppkg installation MADlib, you must run the script 
removefix_compressionmadpack.sh BEFORE sh BEFORE running the madpack utility.
The syntax for removefix_compressionmadpack.sh is sh is below.

This can be somewhat confusing because after gppkg
installation, you will get a message on the console
that says:

...

1. gppkg install of MADlib
2. run removerun fix_compressionmadpack.sh
3. run madpack utility

*****************************************************
COMMAND NAME: removefix_compressionmadpack.sh
*****************************************************

Script to fix a MADlib installation issue on GPDB 4.3.10.

This script patches a line in madpack.py, the MADlib installation
script. A backup of the original file is created in the same folder as
madpack.py called 'madpack.py.orig'.  The script is downloadable from the Pivotal NetworkMADlib install script for HDB/HAWQ 2.0.1+ to remove 'QUICKLZ' 
compression. Works on the current MADlib installation 
(but not all versions of MADlib in the case that multiple 
versions are installed).

*****************************************************
SYNOPSIS
*****************************************************

removefix_compressionmadpack.sh [--prefix <MADLIB_INSTALL_PATH>]

removefix_compressionmadpack.sh -h


*****************************************************
PREREQUISITES
*****************************************************

...

* Set $GPHOME to the correct HAWQ GPDB installation directory containing MADlib
OR
* Set MADlib installation path using the --prefix option

...


*****************************************************
EXAMPLE
*****************************************************

/home/gpadmin/madlib/removefix_compressionmadpack.sh --prefix /usr/local/hdbgpdb/madlib

...