This document outlines how to manually build and setup a Sqoop2 installation. It is expected that this setup will eventually be replaced by the assembly setup of Sqoop2 which will automate these steps.
Building Sqoop2
Checkout sources:
$ svn co https://svn.apache.org/repos/asf/sqoop/branches/sqoop2
And build them by doing the following:
$ cd sqoop2 $ mvn install
For the rest of this document we will refer to the workspace directory as sqoop workspace where necessary.
Setup Install Structure
Create another directory outside of Sqoop2 workspace where you will install the system.
$ cd .. $ mkdir s2test $ cd s2test
Within this directory, create subdirectories as noted below
$ mkdir bin conf logs repository $ ls $ ls -l total 0 drwxr-xr-x 2 arvind staff 68 May 9 00:26 bin drwxr-xr-x 2 arvind staff 68 May 9 00:26 conf drwxr-xr-x 2 arvind staff 68 May 9 00:26 logs drwxr-xr-x 2 arvind staff 68 May 9 00:26 repository $
Add Tomcat Server
Download Tomcat version 7.0.X to a convenient location, and untar it in the s2test
directory and rename the expanded directory to server.
$ wget http://apache.mirrors.hoobly.com/tomcat/tomcat-7/v7.0.27/bin/apache-tomcat-7.0.27.tar.gz ... $ tar zxvf apache-tomcat-7.0.27.tar.gz ... $ rm apache-tomcat-7.0.27.tar.gz $ mv apache-tomcat-7.0.27 server
Now configure the Tomcat server to setup the configuration directory for Sqoop2. This can be done by creating a file called setenv.sh
under server/bin
directory. This file should contain the setting for Sqoop2 configuratiohn directory as follows:
$ # Working directory is s2test created earlier $ echo "export JAVA_OPTS=-Dsqoop.config.dir=/path/to/s2test/conf" >> server/bin/setenv.sh $ chmod +x server/bin/setenv.sh
Note: You must specify the correct path to the s2test
directory in the above command.
Setup Configuration
Under the s2test/conf
directory, setup the bootstrap configuration sqoop_bootstrap.properties
as follows:
# File s2test/conf/sqoop_bootstrap.properties sqoop.config.provider=org.apache.sqoop.core.PropertiesConfigurationProvider
Now setup the main configuration sqoop.properties
as follows:
# File s2test/conf/sqoop.properties # Log4J system org.apache.sqoop.log4j.appender.file=org.apache.log4j.RollingFileAppender org.apache.sqoop.log4j.appender.file.File=/path/to/s2test/logs/sqoop.log org.apache.sqoop.log4j.appender.file.MaxFileSize=25MB org.apache.sqoop.log4j.appender.file.MaxBackupIndex=5 org.apache.sqoop.log4j.appender.file.layout=org.apache.log4j.PatternLayout org.apache.sqoop.log4j.appender.file.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} [%l] %m%n org.apache.sqoop.log4j.debug=true org.apache.sqoop.log4j.rootCategory=WARN, file org.apache.sqoop.log4j.category.org.apache.sqoop=DEBUG org.apache.sqoop.log4j.category.org.apache.derby=INFO # Repository org.apache.sqoop.repository.provider=org.apache.sqoop.repository.JdbcRepositoryProvider org.apache.sqoop.repository.jdbc.handler=org.apache.sqoop.repository.derby.DerbyRepositoryHandler org.apache.sqoop.repository.jdbc.transaction.isolation=READ_COMMITTED org.apache.sqoop.repository.jdbc.maximum.connections=10 org.apache.sqoop.repository.jdbc.url=jdbc:derby:/path/to/s2test/repository/db;create=true org.apache.sqoop.repository.jdbc.create.schema=true org.apache.sqoop.repository.jdbc.driver=org.apache.derby.jdbc.EmbeddedDriver org.apache.sqoop.repository.jdbc.user=sa org.apache.sqoop.repository.jdbc.password= org.apache.sqoop.repository.sysprop.derby.stream.error.file=/path/to/s2test/logs/derbyrepo.log
Note: You must specify the correct path to the s2test
directory in the above configuration entries where necessary.
Deploy Sqoop2
To deploy Sqoop2 to the Tomcat, simply copy over the sqoop.war
file from sqoop2/server/target/
to the s2test/server/webapps
directory. It is recommend that when you do this the Tomcat server should not be running and that you delete any exploded webapp directory for sqoop2 if it exists in the Tomcat server.
$ cp /path/to/sqoop2/server/target/sqoop.war /path/to/s2test/server/webapps/ $ rm -rf /path/to/s2test/server/webapps/sqoop2/
Start Sqoop2
To start Sqoop2 now, simply start the Tomcat server.
$ /path/to/s2test/server/bin/catalina.sh start
If everything is fine, you will see logfiles by the name derbyrepo.log
and sqoop.log
created under /path/to/s2test/logs
directory. The contents of the sqoop.log
should indicate successful startup of the system.
By default the Tomcat server listens on port 8080 and you can access Sqoop2 by going to the URL http://localhost:8080/sqoop/.
You can try accessing the version resource by entering the following address in your browser: http://localhost:8080/sqoop/version
The output may be something like the following:
{"revision":"1238736","protocols":["1"],"date":"Wed May 9 00:23:18 PDT 2012","user":"arvind","url":"https:\/\/svn.apache.org\/repos\/asf\/sqoop\/branches\/sqoop2\/common","version":"2.0.0-SNAPSHOT"}
To stop Sqoop2, simply stop the Tomcat server. To update Sqoop2 to test out the changes you are working on, you should stop the server and redeploy the freshly generated war file as noted above.