Versions Compared

Old Version 142

changes.mady.by.user Lou Degenaro

Saved on

compared with

New Version Current

changes.mady.by.user Lou Degenaro

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

DUCC Project Documentation

Tutorials

  1. Quick Start (from gz)
  2. Quick Start (from source)
  3. Install WebServer Demo
  4. Check out 2.1.0 .0& build

...

Anchor
t1t0t1
t0

Tutotial: Quick Start (from gz)

Audience: DUCC Users/Administrators

Instructions on how to fetch, build and install DUCC 12.0 for single host system (my laptop!)
See DUCC Project Documentation for further details.

The following steps were performed on Red Hat Enterprise Linux Workstation release 6.4 (Santiago)

Image Removed

Prerequisites

...

Install jdk (1.

...

7 or better)

...

Java Home & Version

Code Block
[degenaro@oc4534203605 ~]$ env | grep JAVA_HOME
JAVA_HOME=

...

/home/degenaro/apps/java/java-home

...


[degenaro@oc4534203605 ~]$ /home/degenaro/apps/java/java-home/bin/java -version
java version "1.7.0"
Java(TM) SE Runtime Environment (build pxa6470sr9-20150417_01(SR9))
IBM J9 VM (build 2.6, JRE 1.7.0 Linux amd64-64 Compressed References 20150406_242981 (JIT enabled, AOT enabled)
J9VM - R26_Java726_SR9_20150406_1443_B242981
JIT  - tr.r11_20150401_88894
GC   - R26_Java726_SR9_20150406_1443_B242981_CMPRSS
J9CL - 20150406_242981)
JCL - 20150414_02 based on Oracle 7u79-b14
[degenaro@oc4534203605 ~]$

Start sshd

Code Block
[degenaro@oc4534203605 ~]$ sudo su root
[sudo] password for degenaro: 
[root@oc4534203605 degenaro]# chkconfig sshd on
[root@oc4534203605 degenaro]# /sbin/service sshd restart
Stopping sshd:

Start sshd

Code Block
[degenaro]$ sudo su root
[root]# chkconfig sshd on
[root]# /sbin/service sshd restart
[root]# exit
[degenaro]$

Create user ducc

Code Block
[degenaro]$ sudo adduser ducc

Permit universal read+execute access to home directory for users degenaro and ducc

Code Block
[degenaro]$ sudo su root
[root]# cd /home
[root]# chmod 755 degenaro
[root]# chmod 755 ducc
[root]# exit
[degenaro]$

Enable passwordless ssh for user ducc

Code Block
 
[degenaro]$ sudo su ducc 
[ducc]$ cd
[ducc]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ducc/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/ducc/.ssh/id_rsa.
Your public key has been saved in /home/ducc/.ssh/id_rsa.pub.
The key fingerprint is:
4f:4a:81:60:c9:a5:d8:88:80:f3:6c:a1:ef:31:8b:8e ducc@hal9000.apache.com
The key's randomart image is:
+--[ RSA 2048]----+
|o  .oo.          |
|+..=oo .         |
|.=o.o . .[FAILED]
Starting sshd:         |
|. +      .       |
| o      S .      |
|  +    . + [  OK  ]
[root@oc4534203605  |
| o +    . .      |
|o o              |
|E.               |
+-----------------+
[ducc]$ cd .ssh
[ducc]$ cat id_rsa.pub >> authorized_keys2
[ducc]$ chmod 600 *
[ducc]$ exit
[degenaro]$

Insure JAVA_HOME is set, and PATH contains LaTeX

Sample code to set JAVA_HOME

degenaro]# exit
exit
[degenaro@oc4534203605 ~]$

Create user ducc

Code Block
[degenaro@oc4534203605 ~]$ sudo adduser ducc

Enable passwordless ssh for user ducc

Code Block
[degenaro@oc4534203605 ~]$ sudo su - ducc
[ducc@oc4534203605 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ducc/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/ducc/.ssh/id_rsa.
Your public key has been saved in /home/ducc/.ssh/id_rsa.pub.
The key fingerprint is:
4f:4a:81:60:c9:a5:d8:88:80:f3:6c:a1:ef:31:8b:8e ducc@hal9000.apache.com
The key's randomart image is:
+--[ RSA 2048]----+
|o  .oo.
Code Block
borderStylesolid
title.bashrc
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

# User specific aliases and functions

export JAVA_HOME=~/apps/java/java-home
export PATH=$PATH:/usr/local/tex

Fetch DUCC from svn

Code Block
[degenaro]$ svn co https://svn.apache.org/repos/asf/uima/sandbox/uima-ducc/trunk ~/svn/ducc/trunk

Build DUCC jars

Code Block
[degenaro]$ cd ~/svn/ducc/trunk
[degenaro]$ mvn clean install -Pbuild-duccdocs

Install

Code Block
[degenaro]$ cd target
[degenaro]$ sudo su ducc
[ducc]$ umask 022
[ducc]$ tar -zxf uima-ducc-1.0.0-SNAPSHOT-bin.tar.gz -C ~ducc
[ducc]$ cd ~
[ducc]$ ln -s apache-uima-ducc-1.0.0-SNAPSHOT ducc_install 
[ducc]$ cd ~ducc/ducc_install/admin
[ducc]$ ./ducc_post_install

Check

Code Block
[ducc]$ cd ~ducc/ducc_install/admin
[ducc]$ ./check_ducc -s
[ducc]$ exit
[degenaro]$

Optional: enable ducc_ling and login to Web-server via Linux authentication

Code Block
[degenaro]$ sudo su root
[root]$ mkdir /local
[root]$ mkdir /local/ducc
[root]$ mkdir /local/ducc/bin
[root]$ chown ducc.ducc /local/ducc
[root]$ chown ducc.ducc /local/ducc/bin
[root]$ chmod 700 /local/ducc
[root]$ chmod 700 /local/ducc/bin
[root]$ cp ~ducc/ducc_install/duccling/src/ducc_ling /local/ducc/bin
[root]$ chown root.ducc /local/ducc/bin/ducc_ling
[root]$ chmod 4750 /local/ducc/bin/ducc_ling
[root]$ exit
[degenaro]$
Code Block
[degenaro]$ sudo su ducc
[ducc]$ cd ~ducc/ducc_install/resources
1. edit ducc.properties
2. locate: ducc.ws.login.enabled = false
3. modify: ducc.ws.login.enabled = true
4. locate: ducc.agent.launcher.ducc_spawn_path=${DUCC_HOME}/admin/ducc_ling
5. modify: ducc.agent.launcher.ducc_spawn_path=/local/ducc/bin/ducc_ling
6. save

Start

Code Block
[ducc]$ cd ~ducc/ducc_install/admin
[ducc]$ ./start_ducc -s
[ducc]$ exit
[degenaro]$

ducc-mon System Daemons

Start your favorite browser (firefox, chrome, opera) and visit the ducc-mon system daemons page.

http://localhost:42133/system.daemons.jsp

Image Removed

Submit a test Job

Code Block
[degenaro]$ cd ~ducc/ducc_install/bin
[degenaro]$ export PATH=$PATH:$PWD
[degenaro]$ cd ~
[degenaro]$ mkdir ducc
[degenaro]$ cd ducc
[degenaro]$ cp -pR ~ducc/ducc_install/examples .
[degenaro]$ cd examples/simple
[degenaro]$ ducc_submit -f 1.job
Job 1 submitted
[degenaro]$

ducc-mon Jobs

Visit the ducc-mon jobs page.

http://localhost:42133/jobs.jsp

Image Removed

End.

...

Install WebServer Demo

 

Audience: DUCC Developers

From local workstation, copy DUCC binary tar-ball to installation host

|
|+..=oo .         |
|.=o.o . .        |
|. +      .       |
| o      S .      |
|  +    . +       |
| o +    . .      |
|o o              |
|E.               |
+-----------------+
[ducc@oc4534203605 ~]$ cd .ssh
[ducc@oc4534203605 .ssh]$ cat id_rsa.pub >> authorized_keys2
[ducc@oc4534203605 .ssh]$ chmod 600 *

Install

Visit http://uima.apache.org/downloads.cgi#Latest%20Official%20Releases and download Apache UIMA DUCC Version 2.0.0 tar.gz file to /home/ducc.

Download DUCC .gz file

Code Block
[ducc@oc4534203605 .ssh]$ cd
[ducc@oc4534203605 ~]$ pwd
/home/ducc
[ducc@oc4534203605 ~]$ ls
Desktop  Downloads  uima-ducc-2.0.0-bin.tar.gz

 Unzip

Important: insure proper umask before untarring.

Code Block
[ducc@oc4534203605 ~]$ umask 022
[ducc@oc4534203605 ~]$ tar -zxf uima-ducc-2.0.0-bin.tar.gz -C ~ducc

Configure

Code Block
[ducc@oc4534203605 ~]$ cd apache-uima-ducc-2.0.0/admin/
[[ducc@oc4534203605 admin]$ ./ducc_post_install 
Using DUCC HOME: /home/ducc/apache-uima-ducc-2.0.0 

Existing /home/ducc/apache-uima-ducc-2.0.0/resources/site.ducc.properties moved to /home/ducc/apache-uima-ducc-2.0.0/resources/site.ducc.properties.bak
Existing /home/ducc/apache-uima-ducc-2.0.0/resources/ducc.properties moved to /home/ducc/apache-uima-ducc-2.0.0/resources/ducc.properties.bak
Python version:
2.6.6 (r266:84292, Nov 21 2013, 10:50:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)]
Enter hostname of DUCC head[oc4534203605.ibm.com]
Ducc head is configured as oc4534203605.ibm.com 

Existing /home/ducc/apache-uima-ducc-2.0.0/resources/ducc.nodes moved to /home/ducc/apache-
Code Block
[degenaro]$ scp uima-ducc-2.0.0-bin.tar.gz degenaro@uima-ducc-vm.apache.org:~degenaro

Login to installation host

Note: Developers who are presently unable to login should request access via uima-dev mailing list

Code Block
[degenaro@oc0883280806 Downloads]$ ssh degenaro@uima-ducc-vm.apache.org
...
degenaro@uima-ducc-vm:~$

Stop demo (if running)

Code Block
degenaro@uima-ducc-vm:~$ killall -9 python

Stop ducc (if running)

Code Block
degenaro@uima-ducc-vm:~$ killall -9 java

Stop custom service and/or sleepers (if running)

Code Block
degenaro@uima-ducc-vm:~$ killall -9 service

Install new code

Code Block
degenaro@uima-ducc-vm:~$ cd ~/ducc/versions
degenaro@uima-ducc-vm:~/ducc/versions$ rm -fr trash/*
degenaro@uima-ducc-vm:~/ducc/versions$ mv u* trash
degenaro@uima-ducc-vm:~/ducc/versions$ mv ~/uima-ducc-2.0.0-bin.tar.gz .
degenaro@uima-ducc-vm:~/ducc/versions$ umask 022
degenaro@uima-ducc-vm:~/ducc/versions$ tar -zxf *gz
degenaro@uima-ducc-vm:~/ducc/versions$ ls
apache-uima-ducc-2.0.0  trash  uima-ducc-2.0.0-bin.tar.gz
Code Block
degenaro@uima-ducc-vm:~$ cd ~/ducc
degenaro@uima-ducc-vm:~/ducc$ rm release
degenaro@uima-ducc-vm:~/ducc$ rm 1.1.0
degenaro@uima-ducc-vm:~/ducc$ ln -s versions/apache-uima-ducc-2.0.0 2.0.0
degenaro@uima-ducc-vm:~/ducc$ ln -s 2.0.0 release
degenaro@uima-ducc-vm:~/ducc$ ls -atl
drwxr-xr-x  3 degenaro degenaro 4096 Oct  7 13:06 .
lrwxrwxrwx  1 degenaro degenaro    5 Oct  7 13:00 release -> 2.0.0
lrwxrwxrwx  1 degenaro degenaro   31 Oct  7 12:59 2.0.0 -> versions/resources/ducc.nodes.bak
Initial /home/ducc/apache-uima-ducc-2.0.0/resources/ducc.nodes created.
Existing /home/ducc/apache-uima-ducc-2.0.0/resources/jobdriver.nodes moved to /home/ducc/apache-uima-ducc-2.0.0/resources/jobdriver.nodes.bak
Initial nodepool /home/ducc/apache-uima-ducc-2.0.0/resources/jobdriver.nodes created.

ActiveMQ is automanaged on node  oc4534203605.ibm.com
Enter full path to the Java executable: [/usr/bin/java]
Initialized property "ducc.jvm" to /usr/bin/java
JAVA: java version "1.7.0"
JAVA: Java(TM) SE Runtime Environment (build pxa6470sr9-20150417_01(SR9))
JAVA: IBM J9 VM (build 2.6, JRE 1.7.0 Linux amd64-64 Compressed References 20150406_242981 (JIT enabled, AOT enabled)
JAVA: J9VM - R26_Java726_SR9_20150406_1443_B242981
JAVA: JIT  - tr.r11_20150401_88894
JAVA: GC   - R26_Java726_SR9_20150406_1443_B242981_CMPRSS
JAVA: J9CL - 20150406_242981)
JAVA: JCL - 20150414_02 based on Oracle 7u79-b14
Java version: "1.7.0"
Java is verified.
Enter password for webserver keystore:whyaducc
keystore created as whyaducc

Web server keystore generated from ducc.properties

DUCC book installed into webserver root

Merging /home/ducc/apache-uima-ducc-2.0.0/resources/default.ducc.properties with /home/ducc/apache-uima-ducc-2.0.0/resources/site.ducc.properties into /home/ducc/apache-uima-ducc-2.0.0
drwxr-xr-x  4 degenaro degenaro 4096 Oct  7 12:55 versions
drwxr-xr-x 19 degenaro degenaro 4096 Oct  7 12:54 ..
lrwxrwxrwx  1 degenaro degenaro   28 Aug 27 14:44 logs -> /home/degenaro/tmp/ducc/logs
lrwxrwxrwx  1 degenaro degenaro    7 Mar 29  2014 install -> release
degenaro@uima-ducc-vm:~/ducc$ rm -fr logs/*

Run ducc_post_install

/resources/ducc.properties
Using /usr/bin/java
/usr/bin/java -cp ../lib/uima-ducc/uima-ducc-common-2.0.0.jar org.apache.uima.ducc.common.utils.OsArch
Os Architecture: amd64
cc -g    ducc_ling.c   -o ducc_ling
050 ducc_ling Version 2.0.0 compiled Jul 21 2015 at 08:44:47
ducc_ling is installed for architecture amd64. See the installation guide for multi-user setup.
Initial DUCC setup complete.

 Check 

Code Block
[ducc@oc4534203605 admin]$ ./check_ducc
Set ducc_ling version from oc4534203605.ibm.com : /home/ducc/apache-uima-ducc-2.0.0/admin/amd64/ducc_ling -v >/home/ducc
Code Block
degenaro@uima-ducc-vm:~/ducc/install/admin$ ./ducc_post_install 
Using DUCC HOME: /home/degenaro/ducc/versions/apache-uima-ducc-2.0.0-SNAPSHOT 

Python version:
2.7.3 (default, Feb 27 2014, 19:58:35) 
[GCC 4.6.3]
Enter hostname of DUCC head[uima-ducc-vm]
Ducc head is configured as uima-ducc-vm 

Initial /home/degenaro/ducc/versions/apache-uima-ducc-2.0.0-SNAPSHOT/resources/ducc.nodes created.
Initial nodepool /home/degenaro/ducc/versions/apache-uima-ducc-2.0.0-SNAPSHOT/resources/jobdriver.nodes created.

ActiveMQ is automanaged on node  uima-ducc-vm
Enter full path to the Java binary directory: [/home/degenaro/local/sun/jdk1.7.0_45/bin/java]
Initialized property "ducc.jvm" to /home/degenaro/local/sun/jdk1.7.0_45/bin/java
JAVA: java version "1.7.0_45"
JAVA: Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
JAVA: Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
Java version: "1.7.0_45"
Java is verified.
Enter password for webserver keystore:<xxxxxxxx>
keystore created as xxxxxxxx

Web server keystore generated from ducc.properties

DUCC book installed into webserver root

cc -g    ducc_ling.c   -o ducc_ling
Initial ducc_ling is installed.  See the installation guide for multi-user setup.
Merging /home/degenaro/ducc/versions/state/duccling.version
050 ducc_ling Version 2.0.0 compiled Jul 21 2015 at 08:44:47
JVM: /usr/bin/java
ENV: Java is configured as: /usr/bin/java
ENV java full version JRE 1.7.0 IBM Linux build pxa6470sr9-20150417_01(SR9)
ENV: Threading enabled: True
MEM: memory is 15 gB
ENV: system is Linux
ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com /bin/netstat -an
ActiveMQ broker is not running on tcp://oc4534203605.ibm.com:61617
Checking 1 nodes
ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com ps -eo user:14,pid,comm,args
Waiting for Completion
ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com /home/ducc/apache-uima-ducc-2.0.0/admin/check_ducc -x 1437566853
 
Checking oc4534203605.ibm.com ...
    no processes found.
    WARN: Soft limit RLIMIT_NOFILES is too small at 4096 (< 8192 ).  DUCC may be unable to open sufficient files or sockets.
    ducc_ling module /home/ducc/apache-uima-ducc-2.0.0-SNAPSHOT/admin/resourcesamd64/default.ducc.properties with /home/degenaro/ducc/versions/apache-uima-ducc-2.0.0-SNAPSHOT/resources/site.ducc.properties into /home/degenaro/ducc/versions/ducc_ling : setuid bit is not set. Processes will run as user ducc
    Node health checks return.
All threads returned
[ducc@oc4534203605 admin]$

Privileged ducc_ling

Code Block
[degenaro@oc4534203605 ~]$ sudo su root
[sudo] password for degenaro: 
[root@oc4534203605 degenaro]# cd ~ducc/apache-uima-ducc-2.0.0-SNAPSHOT/resources/ducc.properties

Install local modifications

Code Block
degenaro@uima-ducc-vm:~/ducc/install/examples/uima-ducc-vm/driver$ ./installer
installed: /home/degenaro/ducc/versions/apache-uima-ducc-2.0.0/apache-uima/apache-activemq/conf/activemq-ducc.xml
installed: /home/degenaro/ducc/versions/apache-uima-ducc-2.0.0/resources/site.ducc.properties
installed: /home/degenaro/ducc/versions/apache-uima-ducc-2.0.0/resources/ducc.administrators
installed: /home/degenaro/ducc/versions/apache-uima-ducc-2.0.0/examples/systemtest/ducc.nodes

Launch DUCC (with "sim" nodes)

Code Block
degenaro@uima-ducc-vm:~$ cd ~/ducc/install/examples/systemtest
degenaro@uima-ducc-vm:~/ducc/install/examples/systemtest$ ./start_sim --nothreading -n ducc.nodes -c all
Running as 574
Set ducc_ling version from uima-ducc-vm : /home/degenaro/ducc/versions/2.0.0/apache-uima-ducc-2.0.0/admin/ducc_ling -v >/home/degenaro/ducc/versions/2.0.0/apache-uima-ducc-2.0.0/state/duccling.version
ducc_ling OK
broker host localhost
Waiting for broker 0
Starting from nodes in ducc.nodes
Starting agent on uima-ducc-vm instance 1 as pseudo-node uima-ducc-vm-1 IP 192.168.4.1 memory 15
  ... Started,  PID 1344
Starting agent on uima-ducc-vm instance 2 as pseudo-node uima-ducc-vm-2 IP 192.168.4.2 memory 15
  ... Started,  PID 1549
Starting agent on uima-ducc-vm instance 3 as pseudo-node uima-ducc-vm-3 IP 192.168.4.3 memory 15
  ... Started,  PID 1771
Starting agent on uima-ducc-vm instance 4 as pseudo-node uima-ducc-vm-4 IP 192.168.4.4 memory 15
  ... Started,  PID 1990
Starting agent on uima-ducc-vm instance 5 as pseudo-node uima-ducc-vm-5 IP 192.168.4.5 memory 30
  ... Started,  PID 2204
Starting agent on uima-ducc-vm instance 6 as pseudo-node uima-ducc-vm-6 IP 192.168.4.6 memory 30
  ... Started,  PID 2409
Starting agent on uima-ducc-vm instance 7 as pseudo-node uima-ducc-vm-7 IP 192.168.4.7 memory 30
  ... Started,  PID 2629
Starting agent on uima-ducc-vm instance 8 as pseudo-node uima-ducc-vm-8 IP 192.168.4.8 memory 45
  ... Started,  PID 2841
Starting agent on uima-ducc-vm instance 9 as pseudo-node uima-ducc-vm-9 IP 192.168.4.9 memory 45
  ... Started,  PID 3053
Start db on uima-ducc-vm
    PID 3233
Start ws on uima-ducc-vm
    PID 3407
Start sm on uima-ducc-vm
    PID 3591
Start rm on uima-ducc-vm
    PID 3766
Start or on uima-ducc-vm
    PID 3943
Start pm on uima-ducc-vm
    PID 4123
All Work completed

Create user logs directory

Code Block
degenaro@uima-ducc-vm:~$ cd /tmp
degenaro@uima-ducc-vm:/tmp$ mkdir degenaro
degenaro@uima-ducc-vm:/tmp$ mkdir degenaro/ducc
degenaro@uima-ducc-vm:/tmp$ mkdir degenaro/ducc/logs

degenaro@uima-ducc-vm:~$ cd ~/ducc
degenaro@uima-ducc-vm:~/ducc$ ln -s /tmp/degenaro/ducc/logs/ logs
degenaro@uima-ducc-vm:~/ducc$

Wait for all DUCC daemons to be running

Visit uima-ducc-vm.apache.org:42133/system.daemons.jsp

Image Removed

Create services

/admin/
[root@oc4534203605 admin]# ls
amd64           ducc_boot.py       ducc_props_manager  ducc_reap_rogues.sh  ducc_util.pyc   local_hooks.pyc  rm_qload        sign_jconsole_jar  vary_off
build_duccling  ducc_post_install  ducc.py             ducc_statedump       jconsole.xml    pam_test         rm_qoccupancy   start_ducc         vary_on
check_ducc      ducc_props_diff    ducc.pyc            ducc_util.py         local_hooks.py  read_nodes.py    rm_reconfigure  stop_ducc
[root@oc4534203605 admin]# chown ducc.ducc amd64/
[root@oc4534203605 admin]# chmod 700 amd64/
[root@oc4534203605 admin]# chown root.ducc amd64/ducc_ling 
[root@oc4534203605 admin]# chmod 4750 amd64/ducc_ling 
[root@oc4534203605 admin]# exit
exit
[degenaro@oc4534203605 ~]$

Start 

Code Block
[ducc@oc4534203605 admin]$ ./start_ducc
Merging /home/ducc/apache-uima-ducc-2.0.0/resources/default.ducc.properties with /home/ducc/apache-uima-ducc-2.0.0/resources/site.ducc.properties into /home/ducc/apache-uima-ducc-2.0.0/resources/ducc.properties
Set ducc_ling version from oc4534203605.ibm.com : /home/ducc/apache-uima-ducc-2.0.0/admin/amd64/ducc_ling -v >/home/ducc/apache-uima-ducc-2.0.0/state/duccling.version
JVM: /usr/bin/java
ENV: Java is configured as: /usr/bin/java
ENV java full version JRE 1.7.0 IBM Linux build pxa6470sr9-20150417_01(SR9)
ENV: Threading enabled: True
MEM: memory is 15 gB
ENV: system is Linux
allnodes /home/ducc/apache-uima-ducc-2.0.0/resources/ducc.nodes
Class definition file is ducc.classes
OK: Class and node definitions validated.
OK: Class configuration checked
WARN: Soft limit RLIMIT_NOFILES is too small at 4096 (< 8192 ).  DUCC may be unable to open sufficient files or sockets.
ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com /bin/netstat -an
Starting broker on oc4534203605.ibm.com
ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com ' /home/ducc/apache-uima-ducc-2.0.0/admin/ducc.py -c broker '
----------- INFO: Using default configuration
[] INFO: Using default configuration
----------- (you can configure options in one of these file: /etc/default/activemq /home/ducc/.activemqrc)
[] (you can configure options in one of these file: /etc/default/activemq /home/ducc/.activemqrc)
----------- 
ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com /bin/netstat -an
Waiting for broker ..... 0
ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com /bin/netstat -an
Waiting for broker ..... 1
ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com /bin/netstat -an
ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com /bin/netstat -an
ActiveMQ broker is found on configured host and port: oc4534203605.ibm.com:61617
Starting 1 agents
********** Starting agents from file /home/ducc/apache-uima-ducc-2.0.0/resources/ducc.nodes
Starting warmssh -o BatchMode=yes -o ConnectTimeout=10 
oc4534203605.ibm.com ' /home/ducc/apache-uima-ducc-2.0.0/admin/ducc.py -cagent -b -d 1437567159.08 --nodup '
ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.comssh -o BatchMode=yes -o ConnectTimeout=10 ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com ' /home/ducc/apache-uima-ducc-2.0.0/admin/ducc.py -c sm -b --or_parms=warm -d 1437567159.08 --nodup '
 ssh -o BatchMode=yes -o ConnectTimeout=10 ' /home/ducc/apache-uima-ducc-2.0.0/admin/ducc.py -c rm -b --or_parms=warm -d 1437567159.08 --nodup 'Waiting for Completion
oc4534203605.ibm.com
 ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com  ' /home/ducc/apache-uima-ducc-2.0.0/admin/ducc.py -c ws -b --or_parms=warm -d 1437567159.09 --nodup '' /home/ducc/apache-uima-ducc-2.0.0/admin/ducc.py -c pm -b --or_parms=warm -d 1437567159.08 --nodup '

 oc4534203605.ibm.com ' /home/ducc/apache-uima-ducc-2.0.0/admin/ducc.py -c or -b --or_parms=warm -d 1437567159.09 --nodup '
oc4534203605.ibm.com Starting pm
     WARN: Soft limit RLIMIT_NOFILES is too small at 4096 (< 8192 ).  DUCC may be unable to open sufficient files or sockets.
     PID 12028
oc4534203605.ibm.com Starting rm
     WARN: Soft limit RLIMIT_NOFILES is too small at 4096 (< 8192 ).  DUCC may be unable to open sufficient files or sockets.
     PID 12029
oc4534203605.ibm.com Starting ws
     WARN: Soft limit RLIMIT_NOFILES is too small at 4096 (< 8192 ).  DUCC may be unable to open sufficient files or sockets.
     PID 12030
oc4534203605.ibm.com Starting sm
     WARN: Soft limit RLIMIT_NOFILES is too small at 4096 (< 8192 ).  DUCC may be unable to open sufficient files or sockets.
     PID 12034
oc4534203605.ibm.com Starting or
     WARN: Soft limit RLIMIT_NOFILES is too small at 4096 (< 8192 ).  DUCC may be unable to open sufficient files or sockets.
     PID 12035
oc4534203605.ibm.com 
    WARN: Soft limit RLIMIT_NOFILES is too small at 4096 (< 8192 ).  DUCC may be unable to open sufficient files or sockets.
    ducc_ling module /home/ducc/apache-uima-ducc-2.0.0/admin/amd64/ducc_ling : setuid bit is not set. Processes will run as user ducc
    ssh -o BatchMode=yes -o ConnectTimeout=10 oc4534203605.ibm.com ps -eo user:14,pid,comm,args
    WARN: Soft limit RLIMIT_NOFILES is too small at 4096 (< 8192 ).  DUCC may be unable to open sufficient files or sockets.
    DUCC Agent started PID 12132
All threads returned
[ducc@oc4534203605 admin]$

 Visit DUCC Webserver: display daemons

Image Added

Submit Job

Code Block
[degenaro@oc4534203605 ~]$ 
[degenaro@oc4534203605 bin]$ export PATH=$PATH:$PWD
[degenaro@oc4534203605 bin]$ cd ~
[degenaro@oc4534203605 ~]$ mkdir ducc-test
[degenaro@oc4534203605 ~]$ cd ducc-test
[degenaro@oc4534203605 ducc-test]$ cp -pr ~ducc/apache-uima-ducc-2.0.0/examples/ .
[degenaro@oc4534203605 ducc-test]$ cd examples/simple/
[degenaro@oc4534203605 simple]$ ducc_submit -f 1.job
Job 1 submitted
[degenaro@oc4534203605 simple]$

Visit DUCC Webserver: display jobs

 Image Added

 End.

...

Anchor
t1
t1

Tutorial: Quick Start (from source)

Audience: DUCC Users/Administrators

Instructions on how to fetch, build and install DUCC 1.0 for single host system (my laptop!)
See DUCC Project Documentation for further details.

The following steps were performed on Red Hat Enterprise Linux Workstation release 6.4 (Santiago)

Image Added

Prerequisites

  • Install svn
  • Install mvn
  • Install jdk (1.6 or better)
    export JAVA_HOME=~/apps/java/java-home
  • Install latex
  • Install ant

Start sshd

Code Block
[degenaro]$ sudo su root
[root]# chkconfig sshd on
[root]# /sbin/service sshd restart
[root]# exit
[degenaro]$

Create user ducc

Code Block
[degenaro]$ sudo adduser ducc

Permit universal read+execute access to home directory for users degenaro and ducc

Code Block
[degenaro]$ sudo su root
[root]# cd /home
[root]# chmod 755 degenaro
[root]# chmod 755 ducc
[root]# exit
[degenaro]$

Enable passwordless ssh for user ducc

Code Block
 
[degenaro]$ sudo su ducc 
[ducc]$ cd
[ducc]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ducc/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/ducc/.ssh/id_rsa.
Your public key has been saved in /home/ducc/.ssh/id_rsa.pub.
The key fingerprint is:
4f:4a:81:60:c9:a5:d8:88:80:f3:6c:a1:ef:31:8b:8e ducc@hal9000.apache.com
The key's randomart image is:
+--[ RSA 2048]----+
|o  .oo.          |
|+..=oo .         |
|.=o.o . .        |
|. +      .       |
| o      S .      |
|  +    . +       |
| o +    . .      |
|o o              |
|E.               |
+-----------------+
[ducc]$ cd .ssh
[ducc]$ cat id_rsa.pub >> authorized_keys2
[ducc]$ chmod 600 *
[ducc]$ exit
[degenaro]$

Insure JAVA_HOME is set, and PATH contains LaTeX

Sample code to set JAVA_HOME

Code Block
borderStylesolid
title.bashrc
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

# User specific aliases and functions

export JAVA_HOME=~/apps/java/java-home
export PATH=$PATH:/usr/local/tex

Fetch DUCC from svn

Code Block
[degenaro]$ svn co https://svn.apache.org/repos/asf/uima/sandbox/uima-ducc/trunk ~/svn/ducc/trunk

Build DUCC jars

Code Block
[degenaro]$ cd ~/svn/ducc/trunk
[degenaro]$ mvn clean install -Pbuild-duccdocs

Install

Code Block
[degenaro]$ cd target
[degenaro]$ sudo su ducc
[ducc]$ umask 022
[ducc]$ tar -zxf uima-ducc-1.0.0-SNAPSHOT-bin.tar.gz -C ~ducc
[ducc]$ cd ~
[ducc]$ ln -s apache-uima-ducc-1.0.0-SNAPSHOT ducc_install 
[ducc]$ cd ~ducc/ducc_install/admin
[ducc]$ ./ducc_post_install

Check

Code Block
[ducc]$ cd ~ducc/ducc_install/admin
[ducc]$ ./check_ducc -s
[ducc]$ exit
[degenaro]$

Optional: enable ducc_ling and login to Web-server via Linux authentication

Code Block
[degenaro]$ sudo su root
[root]$ mkdir /local
[root]$ mkdir /local/ducc
[root]$ mkdir /local/ducc/bin
[root]$ chown ducc.ducc /local/ducc
[root]$ chown ducc.ducc /local/ducc/bin
[root]$ chmod 700 /local/ducc
[root]$ chmod 700 /local/ducc/bin
[root]$ cp ~ducc/ducc_install/duccling/src/ducc_ling /local/ducc/bin
[root]$ chown root.ducc /local/ducc/bin/ducc_ling
[root]$ chmod 4750 /local/ducc/bin/ducc_ling
[root]$ exit
[degenaro]$
Code Block
[degenaro]$ sudo su ducc
[ducc]$ cd ~ducc/ducc_install/resources
1. edit ducc.properties
2. locate: ducc.ws.login.enabled = false
3. modify: ducc.ws.login.enabled = true
4. locate: ducc.agent.launcher.ducc_spawn_path=${DUCC_HOME}/admin/ducc_ling
5. modify: ducc.agent.launcher.ducc_spawn_path=/local/ducc/bin/ducc_ling
6. save

Start

Code Block
[ducc]$ cd ~ducc/ducc_install/admin
[ducc]$ ./start_ducc -s
[ducc]$ exit
[degenaro]$

ducc-mon System Daemons

Start your favorite browser (firefox, chrome, opera) and visit the ducc-mon system daemons page.

http://localhost:42133/system.daemons.jsp

Image Added

Submit a test Job

Code Block
[degenaro]$ cd ~ducc/ducc_install/bin
[degenaro]$ export PATH=$PATH:$PWD
[degenaro]$ cd ~
[degenaro]$ mkdir ducc
[degenaro]$ cd ducc
[degenaro]$ cp -pR ~ducc/ducc_install/examples .
[degenaro]$ cd examples/simple
[degenaro]$ ducc_submit -f 1.job
Job 1 submitted
[degenaro]$

ducc-mon Jobs

Visit the ducc-mon jobs page.

http://localhost:42133/jobs.jsp

Image Added

End.

...

Anchor
t2
t2

Tutorial: Install WebServer Demo

 

Audience: DUCC Developers


From local workstation, copy DUCC binary tar-ball to installation host

Code Block
[degenaro]$ scp uima-ducc-2.1.0-bin.tar.gz degenaro@uima-ducc-vm2.apache.org:~degenaro

Login to installation host

Note: Developers who are presently unable to login should request access via uima-dev mailing list

Code Block
[degenaro@oc0883280806 Downloads]$ ssh degenaro@uima-ducc-vm2.apache.org
...
degenaro@uima-ducc-vm2:~$

Switch to user duccy (note: use https://reference.apache.org/committer/otp-md5 for password challenge)

Code Block
degenaro@uima-ducc-vm2:~$ sudo su - duccy
otp-md5 451 ui3416
Password:
duccy@uima-ducc-vm2:~$

Remove crontab

Code Block
duccy@uima-ducc-vm2:~$ crontab -r

Stop demo (if running)

Code Block
duccy@uima-ducc-vm2:~$ killall -9 python

Stop ducc (if running)

Code Block
duccy@uima-ducc-vm2:~$ killall -9 java

Stop custom service and/or sleepers (if running)

Code Block
duccy@uima-ducc-vm2:~$ killall -9 service

Install new code

Code Block
duccy@uima-ducc-vm2:~$ cd install
duccy@uima-ducc-vm2:~/install$ rm -fr *
duccy@uima-ducc-vm2:~/install$ cp -p ~degenaro/uima-ducc-2.1.0-bin.tar.gz .
duccy@uima-ducc-vm2:~/install$ umask 022
duccy@uima-ducc-vm2:~/install$ tar -zxf *
duccy@uima-ducc-vm2:~/install$ cd
duccy@uima-ducc-vm2:~$

...remove old logs and dumps

Code Block
duccy@uima-ducc-vm2:~$ cd ~/ducc/logs
duccy@uima-ducc-vm2:~/ducc/logs$ rm -fr *
duccy@uima-ducc-vm2:~$ cd ~/spool
duccy@uima-ducc-vm2:~/spool$ rm -fr * 
duccy@uima-ducc-vm2:~$ cd ~/dump
duccy@uima-ducc-vm2:~/dump$ rm -fr * 
duccy@uima-ducc-vm2:~/dump$ cd
duccy@uima-ducc-vm2:~$

Run ducc_post_install

Code Block
duccy@uima-ducc-vm2:~$ cd ~/ducc_runtime/admin
duccy@uima-ducc-vm2:~/ducc_runtime/admin$ ./ducc_post_install
Using DUCC HOME: /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT 

Python version:
2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2]
Enter hostname of DUCC head[uima-ducc-vm]localhost
Ducc head is configured as localhost

Initial /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/resources/ducc.nodes created.
Enter full path to the Java executable: [/opt/java/bin/java]/opt/oracle/jdk1.8.0_102/bin/java
Initialized property "ducc.jvm" to /opt/oracle/jdk1.8.0_102/bin/java
JAVA: java version "1.8.0_102"
JAVA: Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
JAVA: Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)
java_home /opt/oracle/jdk1.8.0_102
ActiveMQ is automanaged on node  uima-ducc-vm
broker configuration edit
ducc.broker.admin.username=admin

ducc.broker.admin.password=xxxxxxxxxxxxxxxx

Database host is configured at listen_address: DUCCHOST
Must reconfigure listen_address to uima-ducc-vm
Starting the database.  This might take a few moments if it is the first time.
Database is started.  Waiting for initialization
:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/lib/cassandra/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/lib/guava/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/lib/apache-log4j/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/lib/uima-ducc/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/apache-activemq/lib/*
/usr/local/java-home/bin/java -DDUCC_HOME=/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT org.apache.uima.ducc.database.DbCreate uima-ducc-vm ducc xFpMnJ3aQ
log4j:WARN No appenders could be found for logger (com.datastax.driver.core.SystemProperties).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Waiting for database to boot ...
Waiting for database to boot ...
Waiting for database to boot ...
Waiting for database to boot ...
Waiting for database to boot ...
Waiting for database to boot ...
connect Created database super user ducc
connect Changed default database super user's password and revoked its superuser authority.
connect From this point, this DB can only be accessed in super user mode by user 'ducc'
connect Connected to cluster: DUCC
connect Datatacenter: datacenter1 Host: uima-ducc-vm/140.211.11.134 Rack: rack1
createSchema Created database user 'guest' with SELECT priveleges on DUCC tables.
createSchema EXECUTE STATEMENT: CREATE TABLE IF NOT EXISTS rmnodes (name varchar,memory int,nodepool varchar,shares_left int,responsive boolean,online boolean,reservable boolean,ip varchar,quantum int,classes varchar,share_order int,blacklisted boolean,heartbeats int,assignments int,np_assignments int, PRIMARY KEY(name))
createSchema EXECUTE STATEMENT: CREATE INDEX IF NOT EXISTS rmnodes_responsive_idx ON rmnodes(responsive)
createSchema EXECUTE STATEMENT: CREATE INDEX IF NOT EXISTS rmnodes_online_idx ON rmnodes(online)
createSchema EXECUTE STATEMENT: CREATE INDEX IF NOT EXISTS rmnodes_reservable_idx ON rmnodes(reservable)
createSchema EXECUTE STATEMENT: CREATE TABLE IF NOT EXISTS rmshares (node varchar,ducc_dbid bigint,job_id bigint,uuid uuid,share_order int,quantum int,init_time bigint,evicted boolean,purged boolean,fixed boolean,blacklisted boolean,state varchar,pid bigint,jobtype varchar,investment bigint, PRIMARY KEY(node,ducc_dbid,job_id))
createSchema EXECUTE STATEMENT: CREATE TABLE IF NOT EXISTS rmload (class varchar,job_id bigint,user varchar,memory int,state varchar,demand int,occupancy int,jobtype varchar, PRIMARY KEY(job_id))
createSchema EXECUTE STATEMENT: CREATE TABLE IF NOT EXISTS ducc.smreg (numeric_id varchar,uuid varchar,is_archived boolean,description varchar,administrators varchar,scheduling_class varchar,log_directory varchar,working_directory varchar,jvm varchar,process_jvm_args varchar,classpath varchar,environment varchar,process_memory_size varchar,process_descriptor_DD varchar,process_debug varchar,process_executable varchar,process_executable_args varchar,process_initialization_time_max varchar,service_dependency varchar,service_request_endpoint varchar,service_linger varchar,service_ping_arguments varchar,service_ping_class varchar,service_ping_classpath varchar,service_ping_jvm_args varchar,service_ping_dolog varchar,service_ping_timeout varchar,instance_failures_window varchar,instance_failures_limit varchar,instance_init_failures_limit varchar, PRIMARY KEY(numeric_id))
createSchema EXECUTE STATEMENT: CREATE INDEX IF NOT EXISTS ON ducc.smreg(is_archived)
createSchema EXECUTE STATEMENT: CREATE TABLE IF NOT EXISTS ducc.smmeta (numeric_id varchar,is_archived boolean,uuid varchar,reference varchar,enabled varchar,autostart varchar,instances varchar,endpoint varchar,disable_reason varchar,implementors varchar,ping_active varchar,ping_only varchar,service_alive varchar,service_class varchar,service_dependency varchar,service_healthy varchar,service_state varchar,last_use varchar,last_use_readable varchar,service_statistics varchar,service_type varchar,submit_error varchar,user varchar,references varchar,stopped varchar,last_ping varchar,last_ping_readable varchar,last_runnable varchar,last_runnable_readable varchar,work_instances varchar,registration_date varchar,registration_date_millis varchar, PRIMARY KEY(numeric_id))
createSchema EXECUTE STATEMENT: CREATE INDEX IF NOT EXISTS ON ducc.smmeta(is_archived)
createSchema EXECUTE STATEMENT: CREATE TABLE IF NOT EXISTS job_history (type varchar,ducc_id bigint,history boolean,work blob, PRIMARY KEY(type,ducc_id))WITH CLUSTERING ORDER BY (ducc_id desc)
createSchema EXECUTE STATEMENT: CREATE INDEX IF NOT EXISTS job_history_history_idx ON job_history(history)
createSchema EXECUTE STATEMENT: CREATE TABLE IF NOT EXISTS res_history (type varchar,ducc_id bigint,history boolean,work blob, PRIMARY KEY(type,ducc_id))WITH CLUSTERING ORDER BY (ducc_id desc)
createSchema EXECUTE STATEMENT: CREATE INDEX IF NOT EXISTS res_history_history_idx ON res_history(history)
createSchema EXECUTE STATEMENT: CREATE TABLE IF NOT EXISTS svc_history (type varchar,ducc_id bigint,history boolean,work blob, PRIMARY KEY(type,ducc_id))WITH CLUSTERING ORDER BY (ducc_id desc)
createSchema EXECUTE STATEMENT: CREATE INDEX IF NOT EXISTS svc_history_history_idx ON svc_history(history)
createSchema EXECUTE STATEMENT: CREATE TABLE IF NOT EXISTS orckpt (id int,work blob,p2jmap blob, PRIMARY KEY(id))
Database is initialized.
Stopping the database.
kill -TERM `cat /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/state/cassandra.pid`
Configuring DUCC to use the database.
Java version: "1.7.0_79"
Java is verified.
keystore =  /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/webserver/etc/keystore

Web server keystore generated from ducc.properties

DUCC book installed into webserver root

Merging /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/resources/default.ducc.properties with /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/resources/site.ducc.properties into /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/resources/ducc.properties
Using /usr/local/java-home/bin/java
/usr/local/java-home/bin/java -cp ../lib/uima-ducc/uima-ducc-common-2.1.0-SNAPSHOT.jar org.apache.uima.ducc.common.utils.OsArch
Os Architecture: amd64
cc -g    ducc_ling.c   -o ducc_ling
050 ducc_ling Version 2.1.0 compiled Jul  6 2016 at 18:12:55
ducc_ling is installed for architecture amd64. See the installation guide for additional modifications needed to run user processes with their OS user credentials.
Initial DUCC setup complete.
duccy@uima-ducc-vm:~/ducc_runtime/admin$

Install local modifications

Code Block
duccy@uima-ducc-vm2:~/ducc_runtime/admin$ cd ~/ducc_runtime/examples/uima-ducc-vm/driver
duccy@uima-ducc-vm2:~/ducc_runtime/examples/uima-ducc-vm/driver$ ./installer
installed: /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/apache-activemq/conf/activemq-ducc.xml
installed: /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/resources/site.ducc.properties
installed: /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/resources/ducc.administrators
installed: /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/systemtest/ducc.nodes
duccy@uima-ducc-vm2:~/ducc_runtime/examples/uima-ducc-vm/driver$

Launch DUCC (with "sim" nodes)

Code Block
duccy@uima-ducc-vm2:~/ducc_runtime/examples/uima-ducc-vm/driver$ cd ~/ducc_runtime/examples/systemtest
duccy@uima-ducc-vm2:~/ducc_runtime/examples/systemtest$ ./start_sim --nothreading -n ducc.nodes -c all
Merging /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/resources/default.ducc.properties with /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/resources/site.ducc.properties into /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/resources/ducc.properties
Running as 22281
Set ducc_ling version from uima-ducc-vm : /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/admin/amd64/ducc_ling -v >/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/state/duccling.version
-------- start broker
broker host localhost
Waiting for broker 0
Waiting for broker 1
Waiting for broker 2
Waiting for broker 3
-------- start database
Starting database
Database host is configured at listen_address: uima-ducc-vm
Must reconfigure listen_address to localhost
waiting for database to start
Connected to cluster: DUCC
Tables found: smmeta smreg job_history svc_history rmnodes rmshares rmload res_history orckpt 
Datatacenter: datacenter1; Host: localhost/127.0.0.1; Rack: rack1
OK
-------- specials_only False
-------- start agents
Starting from nodes in ducc.nodes
Starting agent on uima-ducc-demo instance 1 as pseudo-node uima-ducc-demo-1 IP 192.168.4.1 memory 15
  ... Started,  PID 22620
Starting agent on uima-ducc-demo instance 2 as pseudo-node uima-ducc-demo-2 IP 192.168.4.2 memory 15
  ... Started,  PID 22675
Starting agent on uima-ducc-demo instance 3 as pseudo-node uima-ducc-demo-3 IP 192.168.4.3 memory 15
  ... Started,  PID 22728
Starting agent on uima-ducc-demo instance 4 as pseudo-node uima-ducc-demo-4 IP 192.168.4.4 memory 15
  ... Started,  PID 22779
Starting agent on uima-ducc-demo instance 5 as pseudo-node uima-ducc-demo-5 IP 192.168.4.5 memory 30
  ... Started,  PID 22836
Starting agent on uima-ducc-demo instance 6 as pseudo-node uima-ducc-demo-6 IP 192.168.4.6 memory 30
  ... Started,  PID 22895
Starting agent on uima-ducc-demo instance 7 as pseudo-node uima-ducc-demo-7 IP 192.168.4.7 memory 30
  ... Started,  PID 22976
Starting agent on uima-ducc-demo instance 8 as pseudo-node uima-ducc-demo-8 IP 192.168.4.8 memory 45
  ... Started,  PID 23043
Starting agent on uima-ducc-demo instance 9 as pseudo-node uima-ducc-demo-9 IP 192.168.4.9 memory 45
  ... Started,  PID 23128
Starting agent on uima-ducc-demo instance 10 as pseudo-node uima-ducc-demo-10 IP 192.168.4.10 memory 60
  ... Started,  PID 23212
Start ws on uima-ducc-vm
    PID 23234
Start sm on uima-ducc-vm
    PID 23258
Start rm on uima-ducc-vm
    PID 23286
Start or on uima-ducc-vm
    PID 23322
Start pm on uima-ducc-vm
    PID 23343
All Work completed
duccy@uima-ducc-vm2:~/ducc_runtime/examples/systemtest$


Wait for all DUCC daemons to be running

Visit uima-ducc-demo.apache.org:42133/system.daemons.jsp

Image Added

Create services

Code Block
duccy@uima-ducc-vm2:~/ducc_runtime/examples/systemtest$ cd ~/ducc_runtime/examples/uima-ducc-vm/driver/
duccy@uima-ducc-vm2:~/ducc_runtime/examples/uima-ducc-vm/driver$ ./prep
yak
cmd: /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/systemtest/reg_service -i 0 --description "Test Service 0 Type on-demand"
/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/bin/ducc_services --register --instances 1 --description 'Test Service 0 Type on-demand' --process_jvm_args '-DdefaultBrokerURL=tcp://localhost:61617' --classpath '/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/lib/uima-ducc/examples/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/lib/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/apache-activemq/lib/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/apache-activemq/lib/optional/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/simple/resources/service' --service_ping_arguments 'broker-jmx-port=9999' --environment 'AE_INIT_TIME=5000 AE_INIT_RANGE=1000 INIT_ERROR=0' --process_memory_size '15' --working_directory '/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/uima-ducc-vm/driver' --process_descriptor_DD '/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/simple/resources/service/Service_FixedSleep_0.xml' --scheduling_class 'fixed'
Service Register succeeded - Registered - UIMA-AS:FixedSleepAE_0:tcp://localhost:61617 ID[0]
registered service: 0
chinchilla
cmd: /usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/systemtest/reg_service -i 1 --description "Test Service 1 Type auto-start"
/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0
Code Block
degenaro@uima-ducc-vm:~$ cd ~/ducc/install/examples/uima-ducc-vm/driver
degenaro@uima-ducc-vm:~/ducc/install/examples/uima-ducc-vm/driver$ ./prep
/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/bin/ducc_services --register --instances 1 --description 'Test Service 1 Type auto-start' --process_jvm_args '-DdefaultBrokerURL=tcp://localhost:61617' --classpath '/homeusr/degenarolocal/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/lib/uima-ducc/examples/*:/homeusr/degenarolocal/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/apache-uima/lib/uima/*:/homeusr/degenarolocal/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/examples/simple/resources/service' --service_ping_arguments 'broker-jmx-port=9999' --environment 'AE_INIT_TIME=5000 AE_INIT_RANGE=1000 INIT_ERROR=0' --process_memory_size '15' --working_directory '/home/degenaroapache-uima/apache-activemq/lib/*:/usr/local/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/examplesapache-uima/uima-ducc-vm/driver' --process_DD '/home/degenaroapache-activemq/lib/optional/*:/usr/local/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/examples/simple/resources/service' --service/Service_FixedSleep_1.xmlping_arguments 'broker-jmx-port=9999' --scheduling_class 'fixed'
Service Register succeeded - Registered. - UIMA-AS:FixedSleepAE_1:tcp://localhost:61617 ID[0]
registered service: 1
/home/degenaroenvironment 'AE_INIT_TIME=5000 AE_INIT_RANGE=1000 INIT_ERROR=0' --process_memory_size '15' --working_directory '/usr/local/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/binexamples/uima-ducc_services --register --instances 1 --description 'Test Service 2' -vm/driver' --process_jvmdescriptor_argsDD '-DdefaultBrokerURL=tcp://localhost:61617' --classpath '/home/degenaro/usr/local/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/lib/uima-ducc/examples/*:simple/homeresources/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/lib/uima/*:/home/degenaroservice/Service_FixedSleep_1.xml' --scheduling_class 'fixed'
Service Register succeeded - Registered - UIMA-AS:FixedSleepAE_1:tcp://localhost:61617 ID[1]
registered service: 1
cmd: /usr/local/ducc/versions/2.1.0.1-beta/apache-uima-ducc--2.1.0.1-SNAPSHOT/examples/simple/resources/service'bin/ducc_services --service_ping_arguments 'broker-jmx-port=9999'autostart true --environment 'AE_INIT_TIME=5000 AE_INIT_RANGE=1000 INIT_ERROR=0' --process_memory_size '15' --working_directory '/home/degenaromodify 1
Service Modify succeeded - Modify accepted: - UIMA-AS:FixedSleepAE_1:tcp://localhost:61617 ID[1]
autostart service: 1
roadrunner
cmd: /usr/local/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/examples/uima-ducc-vm/driver' --process_DD '/home/degenarosystemtest/reg_service -i 2 --description "Test Service 2 Type on-demand"
/usr/local/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/examples/simple/resources/service/Service_FixedSleep_2.xml' --scheduling_class 'fixed'
Service Register succeeded - Registered. - UIMA-AS:FixedSleepAE_2:bin/ducc_services --register --instances 1 --description 'Test Service 2 Type on-demand' --process_jvm_args '-DdefaultBrokerURL=tcp://localhost:61617 ID[1]
registered service: 2
/home/degenaro' --classpath '/usr/local/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/bin/ducc_services --register --instances 1 --description 'Test Service 3' --process_jvm_args '-DdefaultBrokerURL=tcp://localhost:61617' --classpath '/home/degenaro/lib/uima-ducc/examples/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/lib/*:/usr/local/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/libapache-uima/uimaapache-duccactivemq/exampleslib/*:/homeusr/degenarolocal/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/apache-uima/apache-activemq/lib/uimaoptional/*:/homeusr/degenarolocal/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/examples/simple/resources/service' --service_ping_arguments 'broker-jmx-port=9999' --environment 'AE_INIT_TIME=5000 AE_INIT_RANGE=1000 INIT_ERROR=0' --process_memory_size '15' --working_directory '/homeusr/degenarolocal/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/examples/uima-ducc-vm/driver' --process_descriptor_DD '/homeusr/degenarolocal/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/examples/simple/resources/service/Service_FixedSleep_32.xml' --scheduling_class 'fixed'
Service Register succeeded - Registered. - UIMA-AS:FixedSleepAE_3:tcp://localhost:61617 ID[2]
registered service: 3
/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/bin/ducc_services --register --instances 1 --description 'Test Service 4' --process_jvm_args '-DdefaultBrokerURL= 'fixed'
Service Register succeeded - Registered - UIMA-AS:FixedSleepAE_2:tcp://localhost:61617' --classpath '/home/degenaro ID[2]
registered service: 2
guppy
cmd: /usr/local/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/lib/uima-ducc/examples/*:/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/lib/uima/*:/home/degenaro/ducc/versions/1.0.1systemtest/reg_service -i 3 --description "Test Service 3 Type on-demand"
/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/examples/simple/resources/service'bin/ducc_services --service_ping_arguments 'broker-jmx-port=9999'register --instances 1 --environmentdescription 'AE_INIT_TIME=5000 AE_INIT_RANGE=1000 INIT_ERROR=0Test Service 3 Type on-demand' --process_memoryjvm_size '15args '-DdefaultBrokerURL=tcp://localhost:61617' --working_directoryclasspath '/homeusr/degenarolocal/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/lib/uima-ducc/examples/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/lib/*:/usr/local/ducc/versions/2.1.0-SNAPSHOTbeta/examples/apache-uima-ducc-vm/driver' --process_DD '/home/degenaro2.1.0-SNAPSHOT/apache-uima/apache-activemq/lib/*:/usr/local/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/examples/simple/resources/service/Service_FixedSleep_4.xml' --scheduling_class 'fixed'
Service Register succeeded - Registered. - UIMA-AS:FixedSleepAE_4:tcp://localhost:61617 ID[3]
registered service: 4
/home/degenaroapache-uima/apache-activemq/lib/optional/*:/usr/local/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/examples/simple/bin/ducc_servicesresources/service' --register --instances 1 --description 'Test Service 5service_ping_arguments 'broker-jmx-port=9999' --environment 'AE_INIT_TIME=5000 AE_INIT_RANGE=1000 INIT_ERROR=0' --process_jvmmemory_argssize '-DdefaultBrokerURL=tcp://localhost:6161715' --classpath '/home/degenaro/ducc/versions/1.0.1/apache-uima-ducc-1.0.1-SNAPSHOT/lib/uima-ducc/examples/*:/home/degenaroworking_directory '/usr/local/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/libexamples/uima/*:/home/degenaro-ducc-vm/driver' --process_descriptor_DD '/usr/local/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/examples/simple/resources/service' --service_ping_arguments 'broker-jmx-port=9999' --environment 'AE_INIT_TIME=5000 AE_INIT_RANGE=1000 INIT_ERROR=0' --process_memory_size '15' --working_directory '/home/degenaro/Service_FixedSleep_3.xml' --scheduling_class 'fixed'
Service Register succeeded - Registered - UIMA-AS:FixedSleepAE_3:tcp://localhost:61617 ID[3]
registered service: 3
kiwi
cmd: /usr/local/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/examples/uima-ducc-vm/driver' --process_DD '/home/degenarosystemtest/reg_service -i 4 --description "Test Service 4 Type on-demand"
/usr/local/ducc/versions/2.1.0.1-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/examples/simple/resources/service/Service_FixedSleep_5.xml'bin/ducc_services --scheduling_class 'fixed'
Service Register succeeded - Registered. - UIMA-AS:FixedSleepAE_5:tcp://localhost:61617 ID[4]
registered service: 5
Service Modify succeeded - Service 1 modify request accepted. - UIMA-AS:FixedSleepAE_2:register --instances 1 --description 'Test Service 4 Type on-demand' --process_jvm_args '-DdefaultBrokerURL=tcp://localhost:61617 ID[1]
autostart service: 1

Create Custom services

Code Block
degenaro@uima-ducc-vm:~$ cd ~/ducc/install/examples/uima-ducc-vm/custom
degenaro@uima-ducc-vm:~/ducc/install/examples/uima-ducc-vm/custom$ make
cc -g    service.c   -o service
javac -classpath ~/ducc/install/lib/uima-ducc-cli.jar CustomPing.java
degenaro@uima-ducc-vm:~/ducc/install/examples/uima-ducc-vm/custom$ ./customsvc
Service setup using
   project_home  : /home/degenaro/ducc/versions/' --classpath '/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/lib/uima-ducc/examples/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/apache-uima/lib/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/examplesapache-uima/uima-ducc-vm
   DUCC_HOME     : /home/degenaro/ducc/install
   autostart     : True
   CLASSPATH:
     /home/degenaro/ducc/versionsapache-activemq/lib/*:/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/examplesapache-uima/uima-ducc-vm/custom
Register new service
/home/degenaroapache-activemq/lib/optional/*:/usr/local/ducc/install/bin/ducc_services --register --process_executable 'versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/simple/resources/service' --service_requestping_endpointarguments 'CUSTOM:localhost:7175broker-jmx-port=9999' --service_ping_class 'CustomPing' --description 'Custom Pingerenvironment 'AE_INIT_TIME=5000 AE_INIT_RANGE=1000 INIT_ERROR=0' --process_memory_size '15' --autostart 'True' --service_ping_timeout '10000working_directory '/usr/local/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0-SNAPSHOT/examples/uima-ducc-vm/driver' --serviceprocess_pingdescriptor_classpathDD '/homeusr/degenarolocal/ducc/versions/2.1.0-beta/apache-uima-ducc-2.1.0.1-SNAPSHOT/examples/uima-ducc-vm/custom' --service_ping_dolog 'falsesimple/resources/service/Service_FixedSleep_4.xml' --processscheduling_executable_argsclass '7175fixed' 
Service Register succeeded - Registered. - CUSTOM:localhost:7175 ID[5]

Start driver

 - UIMA-AS:FixedSleepAE_4:tcp://localhost:61617 ID[4]
registered service: 4
duccy@uima-ducc-vm2:~/ducc_runtime/examples/uima-ducc-vm/driver$

Create Custom services

Code Block
duccy@uima-ducc-vm2:~/ducc_runtime/examples/uima-ducc-vm/driver$
Code Block
degenaro@uima-ducc-vm:~$ cd ~/ducc_runtime/install/examples/uima-ducc-vm/drivercustom
degenaro@uimaduccy@uima-ducc-vmvm2:~/ducc/install_runtime/examples/uima-ducc-vm/driver$ ./driver_start
degenaro@uima-ducc-vmcustom$ make
cc -g    service.c   -o service
javac -classpath ~/ducc_runtime/lib/uima-ducc-cli.jar CustomPing.java
duccy@uima-ducc-vm2:~/ducc/install_runtime/examples/uima-ducc-vm/driver$ nohup: appending output to `nohup.out'

degenaro@uima-ducc-vm:~/ducc/install//custom$ ./customsvc
Service setup using
   project_home  : /home/duccy/ducc_runtime/examples/uima-ducc-vm/driver$
 tail -n 20 -f nohup.out 
driver.start
init:1
init:2
run.start:1
Thread-1 seconds to sleep: 124
init:3
run.start:2
Thread-2 seconds to sleep: 90
init:4
run.start:3
Thread-3 seconds to sleep: 22
init:5
run.start:4
Thread-4 seconds to sleep: 243
run.start:5
Thread-5 seconds to sleep: 225
dodo
['/home/degenaro/ducc/versions/apache-uima-ducc-1.0.1-SNAPSHOT/bin/ducc_submit', '--wait_for_completion', '--scheduling_class', 'normal', '--process_memory_size', '30', '--process_thread_count', '6', '--log_directory', '/tmp/ducc/driver/dodo/ducc/logs', '--working_directory', '/tmp/ducc/driver/dodo/ducc/work/3', '-f', '/home/degenaro/ducc/versions/apache-uima-ducc-1.0.1-SNAPSHOT/examples/uima-ducc-vm/jobs/2.job']
...

Check for Jobs to be running

Visit uima-ducc-vm.apache.org:42133/jobs.jsp

...

DUCC_HOME     : /home/duccy/ducc_runtime
   autostart     : True
   CLASSPATH:
     /home/duccy/ducc_runtime/examples/uima-ducc-vm/custom
Register new service
/home/duccy/ducc_runtime/bin/ducc_services --register --process_executable 'service' --service_request_endpoint 'CUSTOM:localhost:7175' --service_ping_class 'CustomPing' --description 'Custom Pinger' --autostart 'True' --service_ping_timeout '10000' --service_ping_classpath '/home/duccy/ducc_runtime/examples/uima-ducc-vm/custom' --service_ping_dolog 'false' --process_executable_args '7175' 
Service Register succeeded - Registered. - CUSTOM:localhost:7175 ID[5]

Start driver

Code Block
duccy@uima-ducc-vm2:~/ducc_runtime/examples/uima-ducc-vm/custom$ cd ~/ducc_runtime/examples/uima-ducc-vm/driver
duccy@uima-ducc-vm2:~/ducc_runtime/examples/uima-ducc-vm/driver$ ./driver_start
duccy@uima-ducc-vm2:~/ducc_runtime/examples/uima-ducc-vm/driver$ nohup: appending output to ‘nohup.out’
...

Check for Jobs to be running

Visit uima-ducc-vm.apache.org:42133/jobs.jsp

Image Added

Install crontab (for auto-restart and auto-cleanup)

Code Block
duccy@uima-ducc-vm2:~$ cd ~/ducc_runtime/examples/uima-ducc-vm/cron
duccy@uima-ducc-vm2:~/ducc_runtime/examples/uima-ducc-vm/cron$ crontab crontab.txt

End.

...

Anchor
t3
t3

Tutorial: Check out 2.1.0

...

& build

Code Block
[degenaro]$ svn co https://svn.apache.org/repos/asf/uima/sandbox/uima-ducc/branches/uima-ducc-1.0.0

 

trunk
...
[degenaro]$ mvn clean install -Pbuild-duccdocs

End.

...

cd