Page History

{color:red}NOTE THAT the information mentioned on this page is still in progress and may not be fully implemented yet.{color}

{toc:style=outline|maxLevel=5}

h3. Building from sources

Checkout sources:
{noformat}
$ svn co https://svn.apache.org/repos/asf/sqoop/branches/sqoop2
{noformat}

Then, change to Sqoop2 source directory and build them:
{noformat}
$ cd sqoop2
$ mvn install
{noformat}

h3. Creating binaries

Now build and package Sqoop2 as distribution:
{noformat}
$ mvn package -Pdist
{noformat}

This process will create a directory and a tarball under {{dist/target}} directory.  The directory (named {{sqoop-2.0.0-SNAPSHOT}} as of this writing) contains necessary binaries to run Sqoop2, and its structure looks something like
{noformat}
--+ bin --+ sqoop.sh
  |
  + client --+ lib --+ sqoop-common.jar
  |                  |
  |                  + sqoop-client.jar
  |                  |
  |                  + (3rd-party client dependency jars)
  |
  + server --+ bin --+ setenv.sh
  |          |
  |          + conf --+ sqoop_bootstrap.properties
  |          |        |
  |          |        + sqoop.properties
  |          |
  |          + webapps --+ ROOT
  |                      |
  |                      + sqoop.war
  |
  + ...
{noformat}

As part of this process, a copy of the Tomcat server is also downloaded and put under the {{server}} directory in the above structure.

h3. Starting/Stopping Sqoop2 server

To start Sqoop2 server, change to Sqoop2 distribution directory and invoke the {{sqoop}} shell script:
{noformat}
cd dist/target/sqoop-2.0.0-SNAPSHOT
bin/sqoop.sh server start
{noformat}

The Sqoop2 server is then running as a web application within the Tomcat server.

Similarly, to stop Sqoop2 server, do the following:
{noformat}
bin/sqoop.sh server stop
{noformat}

h3. Starting/Running Sqoop2 client

To start an interactive shell,
{noformat}
bin/sqoop.sh client
{noformat}

This will bring up an interactive client ready for input commands:
{noformat}
Sqoop Shell: Type 'help' or '\h' for help.

sqoop:000>
{noformat}

Alternatively, the shell client can be run in script mode.  For example, a command script can be created as
{noformat}
echo "set server --host localhost --port 8080 --webapp sqoop" > version.sqp
echo "show version --all" >> version.sqp
{noformat}

Then, the command script can be run with
{noformat}
bin/sqoop.sh client version.sqp
{noformat}

The command for the shell client looks something like *<command> <function> <options>*:

- set
-- set server
--- set server --host <host>
--- set server --port <port>
--- set server --webapp <webapp>

{hide-if:type=page}
- show
-- show version
--- show version --all
--- show version --server
--- show version --client
--- show version --protocol
-- show connector
--- show connector --all
--- show connector --cid <connector_id>
{hide-if:type=page}
-- show connection
-- show job

- create
-- create connection --cid <connector_id>  <more_connection_options>
-- create job --xid <connection_id>  <more_job_options>

- start
-- start job --jid <job_id>

- stop
-- stop job --jid <job_id>
{hide-if}

h3. Modifying configuration

Both the default bootstrap configuration {{sqoop_bootstrap.properties}} and the main configuration {{sqoop.properties}} are located under the {{conf}} directory in the Sqoop2 distribution directory.

The bootstrap configuration {{sqoop_bootstrap.properties}} controls what the mechanism is to provide configuration:
{noformat}
sqoop.config.provider=org.apache.sqoop.core.PropertiesConfigurationProvider
{noformat}

The main configuration {{sqoop.properties}} controls what the mechanism is for repository, where the log files are, what the logging levels are, etc.
{noformat}
# Log4J system
org.apache.sqoop.log4j.appender.file=org.apache.log4j.RollingFileAppender
org.apache.sqoop.log4j.appender.file.File=logs/sqoop.log
org.apache.sqoop.log4j.appender.file.MaxFileSize=25MB
org.apache.sqoop.log4j.appender.file.MaxBackupIndex=5
org.apache.sqoop.log4j.appender.file.layout=org.apache.log4j.PatternLayout
org.apache.sqoop.log4j.appender.file.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} [%l] %m%n
org.apache.sqoop.log4j.debug=true
org.apache.sqoop.log4j.rootCategory=WARN, file
org.apache.sqoop.log4j.category.org.apache.sqoop=DEBUG
org.apache.sqoop.log4j.category.org.apache.derby=INFO

# Repository
org.apache.sqoop.repository.provider=org.apache.sqoop.repository.JdbcRepositoryProvider
org.apache.sqoop.repository.jdbc.handler=org.apache.sqoop.repository.derby.DerbyRepositoryHandler
org.apache.sqoop.repository.jdbc.transaction.isolation=READ_COMMITTED
org.apache.sqoop.repository.jdbc.maximum.connections=10
org.apache.sqoop.repository.jdbc.url=jdbc:derby:repository/db;create=true
org.apache.sqoop.repository.jdbc.create.schema=true
org.apache.sqoop.repository.jdbc.driver=org.apache.derby.jdbc.EmbeddedDriver
org.apache.sqoop.repository.jdbc.user=sa
org.apache.sqoop.repository.jdbc.password=
org.apache.sqoop.repository.sysprop.derby.stream.error.file=logs/derbyrepo.log
{noformat}

h3. Debugging information

The logs of the Tomcat server is located under the {{server/logs}} directory in the Sqoop2 distribution directory.

The logs of the Sqoop2 server and the Derby repository are located as {{sqoop.log}} and {{derbyrepo.log}} (by default unless changed by the above configuration), respectively, under the {{logs}} directory in the Sqoop2 distribution directory.