You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

This document outlines how to manually build and setup a Sqoop2 installation. It is expected that this setup will eventually be replaced by the assembly setup of Sqoop2 which will automate these steps.

Building Sqoop2

Checkout sources:

$ svn co https://svn.apache.org/repos/asf/sqoop/branches/sqoop2

And build them by doing the following:

$ cd sqoop2
$ mvn install

For the rest of this document we will refer to the workspace directory as sqoop workspace where necessary.

Setup Install Structure

Create another directory outside of Sqoop2 workspace where you will install the system. 

$ cd ..
$ mkdir s2test
$ cd s2test

Within this directory, create subdirectories as noted below

$ mkdir bin conf logs repository 
$ ls
$ ls -l
total 0
drwxr-xr-x 2 arvind staff 68 May  9 00:26 bin
drwxr-xr-x 2 arvind staff 68 May  9 00:26 conf
drwxr-xr-x 2 arvind staff 68 May  9 00:26 logs
drwxr-xr-x 2 arvind staff 68 May  9 00:26 repository
$

Add Tomcat Server

Download Tomcat version 7.0.X to a convenient location, and untar it in the s2test directory and rename the expanded directory to server.

$ wget http://apache.mirrors.hoobly.com/tomcat/tomcat-7/v7.0.27/bin/apache-tomcat-7.0.27.tar.gz
...
$ tar zxvf apache-tomcat-7.0.27.tar.gz
...
$ rm apache-tomcat-7.0.27.tar.gz
$ mv apache-tomcat-7.0.27 server

Now configure the Tomcat server to setup the configuration directory for Sqoop2. This can be done by creating a file called setenv.sh under server/bin directory. This file should contain the setting for Sqoop2 configuratiohn directory as follows:

$ # Working directory is s2test created earlier
$ echo "export JAVA_OPTS=-Dsqoop.config.dir=/path/to/s2test/conf" >> server/bin/setenv.sh
$ chmod +x server/bin/setenv.sh

Note: You must specify the correct path to the s2test directory in the above command.

Setup Configuration

Under the s2test/conf directory, setup the bootstrap configuration sqoop_bootstrap.properties as follows:

# File s2test/conf/sqoop_bootstrap.properties
sqoop.config.provider=org.apache.sqoop.core.PropertiesConfigurationProvider

Now setup the main configuration sqoop.properties as follows:

# File s2test/conf/sqoop.properties
# Log4J system
org.apache.sqoop.log4j.appender.file=org.apache.log4j.RollingFileAppender
org.apache.sqoop.log4j.appender.file.File=/path/to/s2test/logs/sqoop.log
org.apache.sqoop.log4j.appender.file.MaxFileSize=25MB
org.apache.sqoop.log4j.appender.file.MaxBackupIndex=5
org.apache.sqoop.log4j.appender.file.layout=org.apache.log4j.PatternLayout
org.apache.sqoop.log4j.appender.file.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} [%l] %m%n

org.apache.sqoop.log4j.debug=true
org.apache.sqoop.log4j.rootCategory=WARN, file
org.apache.sqoop.log4j.category.org.apache.sqoop=DEBUG
org.apache.sqoop.log4j.category.org.apache.derby=INFO

# Repository
org.apache.sqoop.repository.provider=org.apache.sqoop.repository.JdbcRepositoryProvider
org.apache.sqoop.repository.jdbc.handler=org.apache.sqoop.repository.derby.DerbyRepositoryHandler
org.apache.sqoop.repository.jdbc.transaction.isolation=READ_COMMITTED
org.apache.sqoop.repository.jdbc.maximum.connections=10
org.apache.sqoop.repository.jdbc.url=jdbc:derby:/path/to/s2test/repository/db;create=true
org.apache.sqoop.repository.jdbc.create.schema=true
org.apache.sqoop.repository.jdbc.driver=org.apache.derby.jdbc.EmbeddedDriver
org.apache.sqoop.repository.jdbc.user=sa
org.apache.sqoop.repository.jdbc.password=
org.apache.sqoop.repository.sysprop.derby.stream.error.file=/path/to/s2test/logs/derbyrepo.log

Note: You must specify the correct path to the s2test directory in the above configuration entries where necessary.

Deploy Sqoop2

To deploy Sqoop2 to the Tomcat, simply copy over the sqoop.war file from sqoop2/server/target/ to the s2test/server/webapps directory. It is recommend that when you do this the Tomcat server should not be running and that you delete any exploded webapp directory for sqoop2 if it exists in the Tomcat server.

$ cp /path/to/sqoop2/server/target/sqoop.war /path/to/s2test/server/webapps/
$ rm -rf /path/to/s2test/server/webapps/sqoop2/

Start Sqoop2

To start Sqoop2 now, simply start the Tomcat server. 

$ /path/to/s2test/server/bin/catalina.sh start

If everything is fine, you will see logfiles by the name derbyrepo.log and sqoop.log created under /path/to/s2test/logs directory. The contents of the sqoop.log should indicate successful startup of the system.

By default the Tomcat server listens on port 8080 and you can access Sqoop2 by going to the URL http://localhost:8080/sqoop/.

  • No labels