Tutorials
- Quick Start (from gz)
- Quick Start (from source)
- Install WebServer Demo
- Check out 2.1.0 & build
Tutotial: Quick Start (from gz)
Audience: DUCC Users/Administrators
Instructions on how to install DUCC 2.0 for single host system (my laptop!)
See DUCC Project Documentation for further details.
The following steps were performed on Red Hat Enterprise Linux Workstation release 6.4 (Santiago)
Prerequisites
Install jdk (1.7 or better)
Java Home & Version
[degenaro@oc4534203605 ~]$ env | grep JAVA_HOME JAVA_HOME=/home/degenaro/apps/java/java-home [degenaro@oc4534203605 ~]$ /home/degenaro/apps/java/java-home/bin/java -version java version "1.7.0" Java(TM) SE Runtime Environment (build pxa6470sr9-20150417_01(SR9)) IBM J9 VM (build 2.6, JRE 1.7.0 Linux amd64-64 Compressed References 20150406_242981 (JIT enabled, AOT enabled) J9VM - R26_Java726_SR9_20150406_1443_B242981 JIT - tr.r11_20150401_88894 GC - R26_Java726_SR9_20150406_1443_B242981_CMPRSS J9CL - 20150406_242981) JCL - 20150414_02 based on Oracle 7u79-b14 [degenaro@oc4534203605 ~]$
Start sshd
[degenaro@oc4534203605 ~]$ sudo su root [sudo] password for degenaro: [root@oc4534203605 degenaro]# chkconfig sshd on [root@oc4534203605 degenaro]# /sbin/service sshd restart Stopping sshd: [FAILED] Starting sshd: [ OK ] [root@oc4534203605 degenaro]# exit exit [degenaro@oc4534203605 ~]$
Create user ducc
[degenaro@oc4534203605 ~]$ sudo adduser ducc
Enable passwordless ssh for user ducc
[degenaro@oc4534203605 ~]$ sudo su - ducc [ducc@oc4534203605 ~]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/ducc/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/ducc/.ssh/id_rsa. Your public key has been saved in /home/ducc/.ssh/id_rsa.pub. The key fingerprint is: 4f:4a:81:60:c9:a5:d8:88:80:f3:6c:a1:ef:31:8b:8e ducc@hal9000.apache.com The key's randomart image is: +--[ RSA 2048]----+ |o .oo. | |+..=oo . | |.=o.o . . | |. + . | | o S . | | + . + | | o + . . | |o o | |E. | +-----------------+ [ducc@oc4534203605 ~]$ cd .ssh [ducc@oc4534203605 .ssh]$ cat id_rsa.pub >> authorized_keys2 [ducc@oc4534203605 .ssh]$ chmod 600 *
Install
Visit http://uima.apache.org/downloads.cgi#Latest%20Official%20Releases and download Apache UIMA DUCC Version 2.0.0 tar.gz file to /home/ducc.
Download DUCC .gz file
[ducc@oc4534203605 .ssh]$ cd [ducc@oc4534203605 ~]$ pwd /home/ducc [ducc@oc4534203605 ~]$ ls Desktop Downloads uima-ducc-2.0.0-bin.tar.gz
Unzip
Important: insure proper umask before untarring.
[ducc@oc4534203605 ~]$ umask 022 [ducc@oc4534203605 ~]$ tar -zxf uima-ducc-2.0.0-bin.tar.gz -C ~ducc
Configure
[ducc@oc4534203605 ~]$ cd apache-uima-ducc-2.0.0/admin/ [[ducc@oc4534203605 admin]$ ./ducc_post_install Using DUCC HOME: /home/ducc/apache-uima-ducc-2.0.0 Existing /home/ducc/apache-uima-ducc-2.0.0/resources/site.ducc.properties moved to /home/ducc/apache-uima-ducc-2.0.0/resources/site.ducc.properties.bak Existing /home/ducc/apache-uima-ducc-2.0.0/resources/ducc.properties moved to /home/ducc/apache-uima-ducc-2.0.0/resources/ducc.properties.bak Python version: 2.6.6 (r266:84292, Nov 21 2013, 10:50:32) [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] Enter hostname of DUCC head[oc4534203605.ibm.com] Ducc head is configured as oc4534203605.ibm.com Existing /home/ducc/apache-uima-ducc-2.0.0/resources/ducc.nodes moved to /home/ducc/apache-uima-ducc-2.0.0/resources/ducc.nodes.bak Initial /home/ducc/apache-uima-ducc-2.0.0/resources/ducc.nodes created. Existing /home/ducc/apache-uima-ducc-2.0.0/resources/jobdriver.nodes moved to /home/ducc/apache-uima-ducc-2.0.0/resources/jobdriver.nodes.bak Initial nodepool /home/ducc/apache-uima-ducc-2.0.0/resources/jobdriver.nodes created. ActiveMQ is automanaged on node oc4534203605.ibm.com Enter full path to the Java executable: [/usr/bin/java] Initialized property "ducc.jvm" to /usr/bin/java JAVA: java version "1.7.0" JAVA: Java(TM) SE Runtime Environment (build pxa6470sr9-20150417_01(SR9)) JAVA: IBM J9 VM (build 2.6, JRE 1.7.0 Linux amd64-64 Compressed References 20150406_242981 (JIT enabled, AOT enabled) JAVA: J9VM - R26_Java726_SR9_20150406_1443_B242981 JAVA: JIT - tr.r11_20150401_88894 JAVA: GC - R26_Java726_SR9_20150406_1443_B242981_CMPRSS JAVA: J9CL - 20150406_242981) JAVA: JCL - 20150414_02 based on Oracle 7u79-b14 Java version: "1.7.0" Java is verified. Enter password for webserver keystore:whyaducc keystore created as whyaducc Web server keystore generated from ducc.properties DUCC book installed into webserver root Merging /home/ducc/apache-uima-ducc-2.0.0/resources/default.ducc.properties with /home/ducc/apache-uima-ducc-2.0.0/resources/site.ducc.properties into /home/ducc/apache-uima-ducc-2.0.0/resources/ducc.properties Using /usr/bin/java /usr/bin/java -cp ../lib/uima-ducc/uima-ducc-common-2.0.0.jar org.apache.uima.ducc.common.utils.OsArch Os Architecture: amd64 cc -g ducc_ling.c -o ducc_ling 050 ducc_ling Version 2.0.0 compiled Jul 21 2015 at 08:44:47 ducc_ling is installed for architecture amd64. See the installation guide for multi-user setup. Initial DUCC setup complete.
Check
[ducc@oc4534203605 admin]$ ./check_ducc Set ducc_ling version from oc4534203605.ibm.com : /home/ducc/apache-uima-ducc-2.0.0/admin/amd64/ducc_ling -v >/home/ducc/apache-uima-ducc-2.0.0/state/duccling.version 050 ducc_ling Version 2.0.0 compiled Jul 21 2015 at 08:44:47 JVM: /usr/bin/java ENV: Java is configured as: /usr/bin/java ENV java full version JRE 1.7.0 IBM Linux build pxa6470sr9-20150417_01(SR9) ENV: Threading enabled: True MEM: memory is 15 gB ENV: system is Linux ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com /bin/netstat -an ActiveMQ broker is not running on tcp://oc4534203605.ibm.com:61617 Checking 1 nodes ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com ps -eo user:14,pid,comm,args Waiting for Completion ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com /home/ducc/apache-uima-ducc-2.0.0/admin/check_ducc -x 1437566853 Checking oc4534203605.ibm.com ... no processes found. WARN: Soft limit RLIMIT_NOFILES is too small at 4096 (< 8192 ). DUCC may be unable to open sufficient files or sockets. ducc_ling module /home/ducc/apache-uima-ducc-2.0.0/admin/amd64/ducc_ling : setuid bit is not set. Processes will run as user ducc Node health checks return. All threads returned [ducc@oc4534203605 admin]$
Privileged ducc_ling
[degenaro@oc4534203605 ~]$ sudo su root [sudo] password for degenaro: [root@oc4534203605 degenaro]# cd ~ducc/apache-uima-ducc-2.0.0/admin/ [root@oc4534203605 admin]# ls amd64 ducc_boot.py ducc_props_manager ducc_reap_rogues.sh ducc_util.pyc local_hooks.pyc rm_qload sign_jconsole_jar vary_off build_duccling ducc_post_install ducc.py ducc_statedump jconsole.xml pam_test rm_qoccupancy start_ducc vary_on check_ducc ducc_props_diff ducc.pyc ducc_util.py local_hooks.py read_nodes.py rm_reconfigure stop_ducc [root@oc4534203605 admin]# chown ducc.ducc amd64/ [root@oc4534203605 admin]# chmod 700 amd64/ [root@oc4534203605 admin]# chown root.ducc amd64/ducc_ling [root@oc4534203605 admin]# chmod 4750 amd64/ducc_ling [root@oc4534203605 admin]# exit exit [degenaro@oc4534203605 ~]$
Start
[ducc@oc4534203605 admin]$ ./start_ducc Merging /home/ducc/apache-uima-ducc-2.0.0/resources/default.ducc.properties with /home/ducc/apache-uima-ducc-2.0.0/resources/site.ducc.properties into /home/ducc/apache-uima-ducc-2.0.0/resources/ducc.properties Set ducc_ling version from oc4534203605.ibm.com : /home/ducc/apache-uima-ducc-2.0.0/admin/amd64/ducc_ling -v >/home/ducc/apache-uima-ducc-2.0.0/state/duccling.version JVM: /usr/bin/java ENV: Java is configured as: /usr/bin/java ENV java full version JRE 1.7.0 IBM Linux build pxa6470sr9-20150417_01(SR9) ENV: Threading enabled: True MEM: memory is 15 gB ENV: system is Linux allnodes /home/ducc/apache-uima-ducc-2.0.0/resources/ducc.nodes Class definition file is ducc.classes OK: Class and node definitions validated. OK: Class configuration checked WARN: Soft limit RLIMIT_NOFILES is too small at 4096 (< 8192 ). DUCC may be unable to open sufficient files or sockets. ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com /bin/netstat -an Starting broker on oc4534203605.ibm.com ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com ' /home/ducc/apache-uima-ducc-2.0.0/admin/ducc.py -c broker ' ----------- INFO: Using default configuration [] INFO: Using default configuration ----------- (you can configure options in one of these file: /etc/default/activemq /home/ducc/.activemqrc) [] (you can configure options in one of these file: /etc/default/activemq /home/ducc/.activemqrc) ----------- ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com /bin/netstat -an Waiting for broker ..... 0 ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com /bin/netstat -an Waiting for broker ..... 1 ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com /bin/netstat -an ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com /bin/netstat -an ActiveMQ broker is found on configured host and port: oc4534203605.ibm.com:61617 Starting 1 agents ********** Starting agents from file /home/ducc/apache-uima-ducc-2.0.0/resources/ducc.nodes Starting warmssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com ' /home/ducc/apache-uima-ducc-2.0.0/admin/ducc.py -cagent -b -d 1437567159.08 --nodup ' ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.comssh -o BatchMode=yes -o ConnectTimeout=10 ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com ' /home/ducc/apache-uima-ducc-2.0.0/admin/ducc.py -c sm -b --or_parms=warm -d 1437567159.08 --nodup ' ssh -o BatchMode=yes -o ConnectTimeout=10 ' /home/ducc/apache-uima-ducc-2.0.0/admin/ducc.py -c rm -b --or_parms=warm -d 1437567159.08 --nodup 'Waiting for Completion oc4534203605.ibm.com ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com ' /home/ducc/apache-uima-ducc-2.0.0/admin/ducc.py -c ws -b --or_parms=warm -d 1437567159.09 --nodup '' /home/ducc/apache-uima-ducc-2.0.0/admin/ducc.py -c pm -b --or_parms=warm -d 1437567159.08 --nodup ' oc4534203605.ibm.com ' /home/ducc/apache-uima-ducc-2.0.0/admin/ducc.py -c or -b --or_parms=warm -d 1437567159.09 --nodup ' oc4534203605.ibm.com Starting pm WARN: Soft limit RLIMIT_NOFILES is too small at 4096 (< 8192 ). DUCC may be unable to open sufficient files or sockets. PID 12028 oc4534203605.ibm.com Starting rm WARN: Soft limit RLIMIT_NOFILES is too small at 4096 (< 8192 ). DUCC may be unable to open sufficient files or sockets. PID 12029 oc4534203605.ibm.com Starting ws WARN: Soft limit RLIMIT_NOFILES is too small at 4096 (< 8192 ). DUCC may be unable to open sufficient files or sockets. PID 12030 oc4534203605.ibm.com Starting sm WARN: Soft limit RLIMIT_NOFILES is too small at 4096 (< 8192 ). DUCC may be unable to open sufficient files or sockets. PID 12034 oc4534203605.ibm.com Starting or WARN: Soft limit RLIMIT_NOFILES is too small at 4096 (< 8192 ). DUCC may be unable to open sufficient files or sockets. PID 12035 oc4534203605.ibm.com WARN: Soft limit RLIMIT_NOFILES is too small at 4096 (< 8192 ). DUCC may be unable to open sufficient files or sockets. ducc_ling module /home/ducc/apache-uima-ducc-2.0.0/admin/amd64/ducc_ling : setuid bit is not set. Processes will run as user ducc ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com ps -eo user:14,pid,comm,args WARN: Soft limit RLIMIT_NOFILES is too small at 4096 (< 8192 ). DUCC may be unable to open sufficient files or sockets. DUCC Agent started PID 12132 All threads returned [ducc@oc4534203605 admin]$
Visit DUCC Webserver: display daemons
Submit Job
[degenaro@oc4534203605 ~]$ [degenaro@oc4534203605 bin]$ export PATH=$PATH:$PWD [degenaro@oc4534203605 bin]$ cd ~ [degenaro@oc4534203605 ~]$ mkdir ducc-test [degenaro@oc4534203605 ~]$ cd ducc-test [degenaro@oc4534203605 ducc-test]$ cp -pr ~ducc/apache-uima-ducc-2.0.0/examples/ . [degenaro@oc4534203605 ducc-test]$ cd examples/simple/ [degenaro@oc4534203605 simple]$ ducc_submit -f 1.job Job 1 submitted [degenaro@oc4534203605 simple]$
Visit DUCC Webserver: display jobs
End.
Tutorial: Quick Start (from source)
Audience: DUCC Users/Administrators
Instructions on how to fetch, build and install DUCC 1.0 for single host system (my laptop!)
See DUCC Project Documentation for further details.
The following steps were performed on Red Hat Enterprise Linux Workstation release 6.4 (Santiago)
Prerequisites
- Install svn
- Install mvn
- Install jdk (1.6 or better)
export JAVA_HOME=~/apps/java/java-home - Install latex
- Install ant
Start sshd
[degenaro]$ sudo su root [root]# chkconfig sshd on [root]# /sbin/service sshd restart [root]# exit [degenaro]$
Create user ducc
[degenaro]$ sudo adduser ducc
Permit universal read+execute access to home directory for users degenaro and ducc
[degenaro]$ sudo su root [root]# cd /home [root]# chmod 755 degenaro [root]# chmod 755 ducc [root]# exit [degenaro]$
Enable passwordless ssh for user ducc
[degenaro]$ sudo su ducc [ducc]$ cd [ducc]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/ducc/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/ducc/.ssh/id_rsa. Your public key has been saved in /home/ducc/.ssh/id_rsa.pub. The key fingerprint is: 4f:4a:81:60:c9:a5:d8:88:80:f3:6c:a1:ef:31:8b:8e ducc@hal9000.apache.com The key's randomart image is: +--[ RSA 2048]----+ |o .oo. | |+..=oo . | |.=o.o . . | |. + . | | o S . | | + . + | | o + . . | |o o | |E. | +-----------------+ [ducc]$ cd .ssh [ducc]$ cat id_rsa.pub >> authorized_keys2 [ducc]$ chmod 600 * [ducc]$ exit [degenaro]$
Insure JAVA_HOME is set, and PATH contains LaTeX
Sample code to set JAVA_HOME
# .bashrc # Source global definitions if [ -f /etc/bashrc ]; then . /etc/bashrc fi # User specific aliases and functions export JAVA_HOME=~/apps/java/java-home export PATH=$PATH:/usr/local/tex
Fetch DUCC from svn
[degenaro]$ svn co https://svn.apache.org/repos/asf/uima/sandbox/uima-ducc/trunk ~/svn/ducc/trunk
Build DUCC jars
[degenaro]$ cd ~/svn/ducc/trunk [degenaro]$ mvn clean install -Pbuild-duccdocs
Install
[degenaro]$ cd target [degenaro]$ sudo su ducc [ducc]$ umask 022 [ducc]$ tar -zxf uima-ducc-1.0.0-SNAPSHOT-bin.tar.gz -C ~ducc [ducc]$ cd ~ [ducc]$ ln -s apache-uima-ducc-1.0.0-SNAPSHOT ducc_install [ducc]$ cd ~ducc/ducc_install/admin [ducc]$ ./ducc_post_install
Check
[ducc]$ cd ~ducc/ducc_install/admin [ducc]$ ./check_ducc -s [ducc]$ exit [degenaro]$
Optional: enable ducc_ling and login to Web-server via Linux authentication
[degenaro]$ sudo su root [root]$ mkdir /local [root]$ mkdir /local/ducc [root]$ mkdir /local/ducc/bin [root]$ chown ducc.ducc /local/ducc [root]$ chown ducc.ducc /local/ducc/bin [root]$ chmod 700 /local/ducc [root]$ chmod 700 /local/ducc/bin [root]$ cp ~ducc/ducc_install/duccling/src/ducc_ling /local/ducc/bin [root]$ chown root.ducc /local/ducc/bin/ducc_ling [root]$ chmod 4750 /local/ducc/bin/ducc_ling [root]$ exit [degenaro]$
[degenaro]$ sudo su ducc [ducc]$ cd ~ducc/ducc_install/resources 1. edit ducc.properties 2. locate: ducc.ws.login.enabled = false 3. modify: ducc.ws.login.enabled = true 4. locate: ducc.agent.launcher.ducc_spawn_path=${DUCC_HOME}/admin/ducc_ling 5. modify: ducc.agent.launcher.ducc_spawn_path=/local/ducc/bin/ducc_ling 6. save
Start
[ducc]$ cd ~ducc/ducc_install/admin [ducc]$ ./start_ducc -s [ducc]$ exit [degenaro]$
ducc-mon System Daemons
Start your favorite browser (firefox, chrome, opera) and visit the ducc-mon system daemons page.
http://localhost:42133/system.daemons.jsp
Submit a test Job
[degenaro]$ cd ~ducc/ducc_install/bin [degenaro]$ export PATH=$PATH:$PWD [degenaro]$ cd ~ [degenaro]$ mkdir ducc [degenaro]$ cd ducc [degenaro]$ cp -pR ~ducc/ducc_install/examples . [degenaro]$ cd examples/simple [degenaro]$ ducc_submit -f 1.job Job 1 submitted [degenaro]$
ducc-mon Jobs
Visit the ducc-mon jobs page.
http://localhost:42133/jobs.jsp
End.
Tutorial: Install WebServer Demo
Audience: DUCC Developers
From local workstation, copy DUCC binary tar-ball to installation host
[degenaro]$ scp uima-ducc-2.1.0-SNAPSHOT-bin.tar.gz degenaro@uima-ducc-vm.apache.org:~degenaro
Login to installation host
Note: Developers who are presently unable to login should request access via uima-dev mailing list
[degenaro@oc0883280806 Downloads]$ ssh degenaro@uima-ducc-vm.apache.org ... degenaro@uima-ducc-vm:~$
Switch to user duccy (note: use https://reference.apache.org/committer/otp-md5 for password challenge)
degenaro@uima-ducc-vm:~$ sudo su - duccy otp-md5 451 ui3416 Password: duccy@uima-ducc-vm:~$
Stop demo (if running)
degenaro@uima-ducc-vm:~$ killall -9 python
Stop ducc (if running)
degenaro@uima-ducc-vm:~$ killall -9 java
Stop custom service and/or sleepers (if running)
degenaro@uima-ducc-vm:~$ killall -9 service
Install new code
degenaro@uima-ducc-vm:~$ cd versions/2.1.0-beta/ degenaro@uima-ducc-vm:~/ducc/versions$ rm -fr trash/* degenaro@uima-ducc-vm:~/ducc/versions$ mv u* trash degenaro@uima-ducc-vm:~/ducc/versions$ cp -p ~degenaro/uima-ducc-2.1.0-SNAPSHOT-bin.tar.gz . degenaro@uima-ducc-vm:~/ducc/versions$ umask 022 degenaro@uima-ducc-vm:~/ducc/versions$ tar -zxf *gz degenaro@uima-ducc-vm:~/ducc/versions$ ls apache-uima-ducc-2.1.0-SNAPSHOT trash uima-ducc-2.1.0-SNAPSHOT-bin.tar.gz
...remove old logs
duccy@uima-ducc-vm:~/versions/2.1.0-beta$ cd duccy@uima-ducc-vm:~$ cd ducc/logs duccy@uima-ducc-vm:~/ducc/logs$ rm -fr * duccy@uima-ducc-vm:~/ducc/logs$
Run ducc_post_install
duccy@uima-ducc-vm:~/ducc/logs$ cd duccy@uima-ducc-vm:~$ cd ducc_runtime/admin duccy@uima-ducc-vm:~/ducc_runtime/admin$ ./ducc_post_install Using DUCC HOME: /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT Python version: 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2] Enter hostname of DUCC head[uima-ducc-vm] Ducc head is configured as uima-ducc-vm Initial /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/resources/ducc.nodes created. Enter full path to the Java executable: [/usr/bin/java]/usr/local/java-home/bin/java Initialized property "ducc.jvm" to /usr/local/java-home/bin/java JAVA: java version "1.7.0_79" JAVA: Java(TM) SE Runtime Environment (build 1.7.0_79-b15) JAVA: Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode) java_home /usr/local/java-home ActiveMQ is automanaged on node uima-ducc-vm broker configuration edit ducc.broker.admin.username=admin ducc.broker.admin.password=xxxxxxxxxxxxxxxx Database host is configured at listen_address: DUCCHOST Must reconfigure listen_address to uima-ducc-vm Starting the database. This might take a few moments if it is the first time. Database is started. Waiting for initialization :/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/lib/cassandra/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/lib/guava/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/lib/apache-log4j/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/lib/uima-ducc/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/apache-activemq/lib/* /usr/local/java-home/bin/java -DDUCC_HOME=/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT org.apache.uima.ducc.database.DbCreate uima-ducc-vm ducc xFpMnJ3aQ log4j:WARN No appenders could be found for logger (com.datastax.driver.core.SystemProperties). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Waiting for database to boot ... Waiting for database to boot ... Waiting for database to boot ... Waiting for database to boot ... Waiting for database to boot ... Waiting for database to boot ... connect Created database super user ducc connect Changed default database super user's password and revoked its superuser authority. connect From this point, this DB can only be accessed in super user mode by user 'ducc' connect Connected to cluster: DUCC connect Datatacenter: datacenter1 Host: uima-ducc-vm/140.211.11.134 Rack: rack1 createSchema Created database user 'guest' with SELECT priveleges on DUCC tables. createSchema EXECUTE STATEMENT: CREATE TABLE IF NOT EXISTS rmnodes (name varchar,memory int,nodepool varchar,shares_left int,responsive boolean,online boolean,reservable boolean,ip varchar,quantum int,classes varchar,share_order int,blacklisted boolean,heartbeats int,assignments int,np_assignments int, PRIMARY KEY(name)) createSchema EXECUTE STATEMENT: CREATE INDEX IF NOT EXISTS rmnodes_responsive_idx ON rmnodes(responsive) createSchema EXECUTE STATEMENT: CREATE INDEX IF NOT EXISTS rmnodes_online_idx ON rmnodes(online) createSchema EXECUTE STATEMENT: CREATE INDEX IF NOT EXISTS rmnodes_reservable_idx ON rmnodes(reservable) createSchema EXECUTE STATEMENT: CREATE TABLE IF NOT EXISTS rmshares (node varchar,ducc_dbid bigint,job_id bigint,uuid uuid,share_order int,quantum int,init_time bigint,evicted boolean,purged boolean,fixed boolean,blacklisted boolean,state varchar,pid bigint,jobtype varchar,investment bigint, PRIMARY KEY(node,ducc_dbid,job_id)) createSchema EXECUTE STATEMENT: CREATE TABLE IF NOT EXISTS rmload (class varchar,job_id bigint,user varchar,memory int,state varchar,demand int,occupancy int,jobtype varchar, PRIMARY KEY(job_id)) createSchema EXECUTE STATEMENT: CREATE TABLE IF NOT EXISTS ducc.smreg (numeric_id varchar,uuid varchar,is_archived boolean,description varchar,administrators varchar,scheduling_class varchar,log_directory varchar,working_directory varchar,jvm varchar,process_jvm_args varchar,classpath varchar,environment varchar,process_memory_size varchar,process_descriptor_DD varchar,process_debug varchar,process_executable varchar,process_executable_args varchar,process_initialization_time_max varchar,service_dependency varchar,service_request_endpoint varchar,service_linger varchar,service_ping_arguments varchar,service_ping_class varchar,service_ping_classpath varchar,service_ping_jvm_args varchar,service_ping_dolog varchar,service_ping_timeout varchar,instance_failures_window varchar,instance_failures_limit varchar,instance_init_failures_limit varchar, PRIMARY KEY(numeric_id)) createSchema EXECUTE STATEMENT: CREATE INDEX IF NOT EXISTS ON ducc.smreg(is_archived) createSchema EXECUTE STATEMENT: CREATE TABLE IF NOT EXISTS ducc.smmeta (numeric_id varchar,is_archived boolean,uuid varchar,reference varchar,enabled varchar,autostart varchar,instances varchar,endpoint varchar,disable_reason varchar,implementors varchar,ping_active varchar,ping_only varchar,service_alive varchar,service_class varchar,service_dependency varchar,service_healthy varchar,service_state varchar,last_use varchar,last_use_readable varchar,service_statistics varchar,service_type varchar,submit_error varchar,user varchar,references varchar,stopped varchar,last_ping varchar,last_ping_readable varchar,last_runnable varchar,last_runnable_readable varchar,work_instances varchar,registration_date varchar,registration_date_millis varchar, PRIMARY KEY(numeric_id)) createSchema EXECUTE STATEMENT: CREATE INDEX IF NOT EXISTS ON ducc.smmeta(is_archived) createSchema EXECUTE STATEMENT: CREATE TABLE IF NOT EXISTS job_history (type varchar,ducc_id bigint,history boolean,work blob, PRIMARY KEY(type,ducc_id))WITH CLUSTERING ORDER BY (ducc_id desc) createSchema EXECUTE STATEMENT: CREATE INDEX IF NOT EXISTS job_history_history_idx ON job_history(history) createSchema EXECUTE STATEMENT: CREATE TABLE IF NOT EXISTS res_history (type varchar,ducc_id bigint,history boolean,work blob, PRIMARY KEY(type,ducc_id))WITH CLUSTERING ORDER BY (ducc_id desc) createSchema EXECUTE STATEMENT: CREATE INDEX IF NOT EXISTS res_history_history_idx ON res_history(history) createSchema EXECUTE STATEMENT: CREATE TABLE IF NOT EXISTS svc_history (type varchar,ducc_id bigint,history boolean,work blob, PRIMARY KEY(type,ducc_id))WITH CLUSTERING ORDER BY (ducc_id desc) createSchema EXECUTE STATEMENT: CREATE INDEX IF NOT EXISTS svc_history_history_idx ON svc_history(history) createSchema EXECUTE STATEMENT: CREATE TABLE IF NOT EXISTS orckpt (id int,work blob,p2jmap blob, PRIMARY KEY(id)) Database is initialized. Stopping the database. kill -TERM `cat /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/state/cassandra.pid` Configuring DUCC to use the database. Java version: "1.7.0_79" Java is verified. keystore = /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/webserver/etc/keystore Web server keystore generated from ducc.properties DUCC book installed into webserver root Merging /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/resources/default.ducc.properties with /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/resources/site.ducc.properties into /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/resources/ducc.properties Using /usr/local/java-home/bin/java /usr/local/java-home/bin/java -cp ../lib/uima-ducc/uima-ducc-common-2.1.0-SNAPSHOT.jar org.apache.uima.ducc.common.utils.OsArch Os Architecture: amd64 cc -g ducc_ling.c -o ducc_ling 050 ducc_ling Version 2.1.0 compiled Jul 6 2016 at 18:12:55 ducc_ling is installed for architecture amd64. See the installation guide for additional modifications needed to run user processes with their OS user credentials. Initial DUCC setup complete. duccy@uima-ducc-vm:~/ducc_runtime/admin$
Install local modifications
duccy@uima-ducc-vm:~/ducc_runtime/admin$ cd ~/ducc_runtime/examples/uima-ducc-vm/driver/ duccy@uima-ducc-vm:~/ducc_runtime/examples/uima-ducc-vm/driver$ ./installer installed: /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/apache-activemq/conf/activemq-ducc.xml installed: /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/resources/site.ducc.properties installed: /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/resources/ducc.administrators installed: /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/systemtest/ducc.nodes duccy@uima-ducc-vm:~/ducc_runtime/examples/uima-ducc-vm/driver$
Launch DUCC (with "sim" nodes)
duccy@uima-ducc-vm:~/ducc_runtime/examples/uima-ducc-vm/driver$ cd ~/ducc_runtime/examples/systemtest duccy@uima-ducc-vm:~/ducc_runtime/examples/systemtest$ ./start_sim --nothreading -n ducc.nodes -c all Merging /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/resources/default.ducc.properties with /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/resources/site.ducc.properties into /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/resources/ducc.properties Running as 22281 Set ducc_ling version from uima-ducc-vm : /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/admin/amd64/ducc_ling -v >/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/state/duccling.version -------- start broker broker host localhost Waiting for broker 0 Waiting for broker 1 Waiting for broker 2 Waiting for broker 3 -------- start database Starting database Database host is configured at listen_address: uima-ducc-vm Must reconfigure listen_address to localhost waiting for database to start Connected to cluster: DUCC Tables found: smmeta smreg job_history svc_history rmnodes rmshares rmload res_history orckpt Datatacenter: datacenter1; Host: localhost/127.0.0.1; Rack: rack1 OK -------- specials_only False -------- start agents Starting from nodes in ducc.nodes Starting agent on uima-ducc-demo instance 1 as pseudo-node uima-ducc-demo-1 IP 192.168.4.1 memory 15 ... Started, PID 22620 Starting agent on uima-ducc-demo instance 2 as pseudo-node uima-ducc-demo-2 IP 192.168.4.2 memory 15 ... Started, PID 22675 Starting agent on uima-ducc-demo instance 3 as pseudo-node uima-ducc-demo-3 IP 192.168.4.3 memory 15 ... Started, PID 22728 Starting agent on uima-ducc-demo instance 4 as pseudo-node uima-ducc-demo-4 IP 192.168.4.4 memory 15 ... Started, PID 22779 Starting agent on uima-ducc-demo instance 5 as pseudo-node uima-ducc-demo-5 IP 192.168.4.5 memory 30 ... Started, PID 22836 Starting agent on uima-ducc-demo instance 6 as pseudo-node uima-ducc-demo-6 IP 192.168.4.6 memory 30 ... Started, PID 22895 Starting agent on uima-ducc-demo instance 7 as pseudo-node uima-ducc-demo-7 IP 192.168.4.7 memory 30 ... Started, PID 22976 Starting agent on uima-ducc-demo instance 8 as pseudo-node uima-ducc-demo-8 IP 192.168.4.8 memory 45 ... Started, PID 23043 Starting agent on uima-ducc-demo instance 9 as pseudo-node uima-ducc-demo-9 IP 192.168.4.9 memory 45 ... Started, PID 23128 Starting agent on uima-ducc-demo instance 10 as pseudo-node uima-ducc-demo-10 IP 192.168.4.10 memory 60 ... Started, PID 23212 Start ws on uima-ducc-vm PID 23234 Start sm on uima-ducc-vm PID 23258 Start rm on uima-ducc-vm PID 23286 Start or on uima-ducc-vm PID 23322 Start pm on uima-ducc-vm PID 23343 All Work completed duccy@uima-ducc-vm:~/ducc_runtime/examples/systemtest$
Wait for all DUCC daemons to be running
Visit uima-ducc-vm.apache.org:42133/system.daemons.jsp
Create services
duccy@uima-ducc-vm:~/ducc_runtime/examples/systemtest$ cd ~/ducc_runtime/examples/uima-ducc-vm/driver/ duccy@uima-ducc-vm:~/ducc_runtime/examples/uima-ducc-vm/driver$ ./prep yak cmd: /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/systemtest/reg_service -i 0 --description "Test Service 0 Type on-demand" /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/bin/ducc_services --register --instances 1 --description 'Test Service 0 Type on-demand' --process_jvm_args '-DdefaultBrokerURL=tcp://localhost:61617' --classpath '/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/lib/uima-ducc/examples/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/lib/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/apache-activemq/lib/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/apache-activemq/lib/optional/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/simple/resources/service' --service_ping_arguments 'broker-jmx-port=9999' --environment 'AE_INIT_TIME=5000 AE_INIT_RANGE=1000 INIT_ERROR=0' --process_memory_size '15' --working_directory '/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/uima-ducc-vm/driver' --process_descriptor_DD '/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/simple/resources/service/Service_FixedSleep_0.xml' --scheduling_class 'fixed' Service Register succeeded - Registered - UIMA-AS:FixedSleepAE_0:tcp://localhost:61617 ID[0] registered service: 0 chinchilla cmd: /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/systemtest/reg_service -i 1 --description "Test Service 1 Type auto-start" /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/bin/ducc_services --register --instances 1 --description 'Test Service 1 Type auto-start' --process_jvm_args '-DdefaultBrokerURL=tcp://localhost:61617' --classpath '/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/lib/uima-ducc/examples/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/lib/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/apache-activemq/lib/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/apache-activemq/lib/optional/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/simple/resources/service' --service_ping_arguments 'broker-jmx-port=9999' --environment 'AE_INIT_TIME=5000 AE_INIT_RANGE=1000 INIT_ERROR=0' --process_memory_size '15' --working_directory '/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/uima-ducc-vm/driver' --process_descriptor_DD '/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/simple/resources/service/Service_FixedSleep_1.xml' --scheduling_class 'fixed' Service Register succeeded - Registered - UIMA-AS:FixedSleepAE_1:tcp://localhost:61617 ID[1] registered service: 1 cmd: /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/bin/ducc_services --autostart true --modify 1 Service Modify succeeded - Modify accepted: - UIMA-AS:FixedSleepAE_1:tcp://localhost:61617 ID[1] autostart service: 1 roadrunner cmd: /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/systemtest/reg_service -i 2 --description "Test Service 2 Type on-demand" /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/bin/ducc_services --register --instances 1 --description 'Test Service 2 Type on-demand' --process_jvm_args '-DdefaultBrokerURL=tcp://localhost:61617' --classpath '/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/lib/uima-ducc/examples/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/lib/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/apache-activemq/lib/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/apache-activemq/lib/optional/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/simple/resources/service' --service_ping_arguments 'broker-jmx-port=9999' --environment 'AE_INIT_TIME=5000 AE_INIT_RANGE=1000 INIT_ERROR=0' --process_memory_size '15' --working_directory '/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/uima-ducc-vm/driver' --process_descriptor_DD '/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/simple/resources/service/Service_FixedSleep_2.xml' --scheduling_class 'fixed' Service Register succeeded - Registered - UIMA-AS:FixedSleepAE_2:tcp://localhost:61617 ID[2] registered service: 2 guppy cmd: /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/systemtest/reg_service -i 3 --description "Test Service 3 Type on-demand" /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/bin/ducc_services --register --instances 1 --description 'Test Service 3 Type on-demand' --process_jvm_args '-DdefaultBrokerURL=tcp://localhost:61617' --classpath '/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/lib/uima-ducc/examples/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/lib/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/apache-activemq/lib/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/apache-activemq/lib/optional/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/simple/resources/service' --service_ping_arguments 'broker-jmx-port=9999' --environment 'AE_INIT_TIME=5000 AE_INIT_RANGE=1000 INIT_ERROR=0' --process_memory_size '15' --working_directory '/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/uima-ducc-vm/driver' --process_descriptor_DD '/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/simple/resources/service/Service_FixedSleep_3.xml' --scheduling_class 'fixed' Service Register succeeded - Registered - UIMA-AS:FixedSleepAE_3:tcp://localhost:61617 ID[3] registered service: 3 kiwi cmd: /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/systemtest/reg_service -i 4 --description "Test Service 4 Type on-demand" /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/bin/ducc_services --register --instances 1 --description 'Test Service 4 Type on-demand' --process_jvm_args '-DdefaultBrokerURL=tcp://localhost:61617' --classpath '/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/lib/uima-ducc/examples/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/lib/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/apache-activemq/lib/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/apache-activemq/lib/optional/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/simple/resources/service' --service_ping_arguments 'broker-jmx-port=9999' --environment 'AE_INIT_TIME=5000 AE_INIT_RANGE=1000 INIT_ERROR=0' --process_memory_size '15' --working_directory '/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/uima-ducc-vm/driver' --process_descriptor_DD '/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/simple/resources/service/Service_FixedSleep_4.xml' --scheduling_class 'fixed' Service Register succeeded - Registered - UIMA-AS:FixedSleepAE_4:tcp://localhost:61617 ID[4] registered service: 4 duccy@uima-ducc-vm:~/ducc_runtime/examples/uima-ducc-vm/driver$
Create Custom services
duccy@uima-ducc-vm:~/ducc_runtime/examples/uima-ducc-vm/driver$ cd ~/ducc_runtime/examples/uima-ducc-vm/custom degenaro@uima-ducc-vm:~/ducc/install/examples/uima-ducc-vm/custom$ make cc -g service.c -o service javac -classpath ~/ducc/install/lib/uima-ducc-cli.jar CustomPing.java degenaro@uima-ducc-vm:~/ducc/install/examples/uima-ducc-vm/custom$ ./customsvc Service setup using project_home : /home/degenaro/ducc/versions/apache-uima-ducc-1.0.1-SNAPSHOT/examples/uima-ducc-vm DUCC_HOME : /home/degenaro/ducc/install autostart : True CLASSPATH: /home/degenaro/ducc/versions/apache-uima-ducc-1.0.1-SNAPSHOT/examples/uima-ducc-vm/custom Register new service /home/degenaro/ducc/install/bin/ducc_services --register --process_executable 'service' --service_request_endpoint 'CUSTOM:localhost:7175' --service_ping_class 'CustomPing' --description 'Custom Pinger' --autostart 'True' --service_ping_timeout '10000' --service_ping_classpath '/home/degenaro/ducc/versions/apache-uima-ducc-1.0.1-SNAPSHOT/examples/uima-ducc-vm/custom' --service_ping_dolog 'false' --process_executable_args '7175' Service Register succeeded - Registered. - CUSTOM:localhost:7175 ID[5]
Start driver
duccy@uima-ducc-vm:~/ducc_runtime/examples/uima-ducc-vm/custom$ cd ~/ducc_runtime/examples/uima-ducc-vm/driver duccy@uima-ducc-vm:~/ducc_runtime/examples/uima-ducc-vm/driver$ ./driver_start duccy@uima-ducc-vm:~/ducc_runtime/examples/uima-ducc-vm/driver$ nohup: appending output to ‘nohup.out’ ...
Check for Jobs to be running
Visit uima-ducc-vm.apache.org:42133/jobs.jsp
End.
Tutorial: Check out 2.1.0 & build
[degenaro]$ svn co https://svn.apache.org/repos/asf/uima/uima-ducc/trunk ... [degenaro]$ mvn clean install -Pbuild-duccdocs
End.
cd