Sqoop2 Quickstart

NOTE THAT the information mentioned on this page is still in progress and may not be fully implemented yet.

Building from sources

Checkout sources:

$ svn co https://svn.apache.org/repos/asf/sqoop/branches/sqoop2

Then, change to Sqoop2 source directory and build them:

$ cd sqoop2
$ mvn install

Creating distribution

Now build and package Sqoop2 as distribution:

$ mvn package -Pdist

This process will create a directory and a tarball under dist/target directory. The directory (named sqoop-2.0.0-SNAPSHOT as of this writing) contains necessary binaries to run Sqoop2, and its structure looks something like

--+ bin --+ sqoop
  | 
  + conf --+ sqoop_bootstrap.properties
  |        |
  |        +- sqoop.properties
  |
  + server --+ bin --+ setenv.sh
  |          |
  |          + webapps --+ sqoop.war
  |
  + sqoop-client.jar
  |
  + sqoop-common.jar
  |
  + ...

As part of this process, a copy of Tomcat server is also downloaded and put under the server directory in the above structure.

Starting/Stopping Sqoop2 server

To start Sqoop2 server, change to Sqoop2 distribution directory and invoke the sqoop shell script:

cd dist/target/sqoop-2.0.0-SNAPSHOT
bin/sqoop server start

Similarly, to stop Sqoop2 server, do the following:

bin/sqoop server stop

Starting/Running Sqoop2 client

To start an interactive shell,

bin/sqoop client

This will bring up an interactive client ready for input commands:

Sqoop Shell: Type 'help' or '\h' for help.

sqoop:000>

Alternatively, the shell client can be run in script mode. For example, a command script can be created as

echo "set server --host localhost --port 8080 --webapp sqoop" > sqoop.script
echo "show version --all" >> sqoop.script

Then, the command script can be run with

bin/sqoop client sqoop.script

The command for the shell client looks something like <command> <function> <options>:

• set
  • set server
    • set server --host <host>
    • set server --port <port>
    • set server --webapp <webapp>

• show
  • show version
    • show version --all
    • show version --server
    • show version --client
    • show version --protocol
      Unknown macro: {hide-if}

      – show connector
      – show connection
      – show job

      • create
        • create connection --id <connector id> <more connection options>
        • create job --cid <connection id> <more job options>

      • start
        • start job --jid <job id>

      • stop
        • stop job --jid <job id>

Modifying configuration

Both the default bootstrap configuration sqoop_bootstrap.properties and the main configuration sqoop.properties are located under the conf directory in the Sqoop2 distribution directory.

The bootstrap configuration sqoop_bootstrap.properties controls what the mechanism is to provide configuration:

sqoop.config.provider=org.apache.sqoop.core.PropertiesConfigurationProvider

The main configuration sqoop.properties controls what the mechanism is for repository, where the log files are, what the logging level is, etc.

# Log4J system
org.apache.sqoop.log4j.appender.file=org.apache.log4j.RollingFileAppender
org.apache.sqoop.log4j.appender.file.File=logs/sqoop.log
org.apache.sqoop.log4j.appender.file.MaxFileSize=25MB
org.apache.sqoop.log4j.appender.file.MaxBackupIndex=5
org.apache.sqoop.log4j.appender.file.layout=org.apache.log4j.PatternLayout
org.apache.sqoop.log4j.appender.file.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} [%l] %m%n
org.apache.sqoop.log4j.debug=true
org.apache.sqoop.log4j.rootCategory=WARN, file
org.apache.sqoop.log4j.category.org.apache.sqoop=DEBUG
org.apache.sqoop.log4j.category.org.apache.derby=INFO

# Repository
org.apache.sqoop.repository.provider=org.apache.sqoop.repository.JdbcRepositoryProvider
org.apache.sqoop.repository.jdbc.handler=org.apache.sqoop.repository.derby.DerbyRepositoryHandler
org.apache.sqoop.repository.jdbc.transaction.isolation=READ_COMMITTED
org.apache.sqoop.repository.jdbc.maximum.connections=10
org.apache.sqoop.repository.jdbc.url=jdbc:derby:repository/db;create=true
org.apache.sqoop.repository.jdbc.create.schema=true
org.apache.sqoop.repository.jdbc.driver=org.apache.derby.jdbc.EmbeddedDriver
org.apache.sqoop.repository.jdbc.user=sa
org.apache.sqoop.repository.jdbc.password=
org.apache.sqoop.repository.sysprop.derby.stream.error.file=logs/derbyrepo.log