THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
| No Format |
|---|
mvn clean test -pl test -Dtest=org.apache.sqoop.integration.connector.kafka.FromRDBMSToKafkaTest |
There are different profiles as well: slow and fast. The fast integration tests run by default, while the slow integration tests need to be explicitly ran:
| Code Block | ||
|---|---|---|
| ||
mvn clean integration-test -Dslow
mvn clean integration-test -Dfast |
How to investigate failure of the integration tests?
Sqoop integration tests are truly integration - we run separate tomcat process with Sqoop 2 server and Hadoop MiniClusters to emulate real cluster as much as possible while still running on one single node. This is however making troubleshooting a bit difficult as various logs are on different places (as would be the case for real cluster). List of various important logs:
| Location | Notes |
|---|---|
test/target/surefire-reports/${testClass}-output.txt | Main testng log. It will contain any code executed directly from the test Java Class - helper methods creating or inserting data to databases will log here In addition this log will contain logs from MiniClusters - e.g. logs from HDFS/Mapreduce/YARN daemons. |
test/target/sqoop-cargo-tests/${testClass}/${testName}/sqoop-mini-cluster/log/tomcat.log | Tomcat log's, on production system this might be called catalina.log |
test/target/sqoop-cargo-tests/${testClass}/${testName}/sqoop-mini-cluster/log/sqoop.log | Sqoop server logs (what the server is logging out) |
test/target/MiniMRCluster_${randomNumber}/ | Logs from Yarn containers (and hence mapreduce tasks). Usually it's useful to look for "syslog", "stdin", "stdout" and "stderr" files. |
I'm running tests on Mac computer and my focus is stolen several times during the test execution
If you see new Java processes created UI application and stealing the focus, then you should export this property to avoid that:
...
First check the tomcat.log under /path/to/sqoop//test/target/sqoop-cargo-tests/ org.apache.sqoop.integration.connector.jdbc.generic.FromRDBMSToHDFSTest/testBasic/log/tomcat.log
...
AM
...
org.apache.catalina.startup.Catalina
...
stopServerSEVERE:
...
Catalina.stop:
...
java.io.FileNotFoundException:
...
/var/folders/l8/hyl1hnqj3vq57gdf8f9nb0740000gp/T/cargo/conf/conf/server.xml
...
(No
...
such
...
file
...
or
...
directory)
...
at
...
java.io.FileInputStream.open(Native
...
Method)
...
at
...
java.io.FileInputStream.<init>(FileInputStream.java:146)
...
at
...
org.apache.catalina.startup.Catalina.stopServer(Catalina.java:395)
...
at
...
sun.reflect.NativeMethodAccessorImpl.invoke0(Native
...
Method)
...
at
...
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
Solution : Nuke the directory /var/folders/l8/hyl1hnqj3vq57gdf8f9nb0740000gp/T/cargo
...