...
Code Block |
---|
<!-- Listeners --> <listener> <listener-class>org.apache.sqoop.server.ServerInitializer</listener-class> </listener> |
Sqoop Server
- The sqoop server is represented by the java class
SqoopServer.java
- SqoopServer.initialize() is the called from the
ServerInitiaizer
- SqoopServer.destroy() is called when the tomcat server is shutdown
Sqoop Servlets
- We have servlets for each Sqoop Entity that is exposed via the REST API in web.xml. Refer to Sqoop 2 (1.99.4) Entity Nomenclature and Relationships for more details on the supported Sqoop Entities.
- They receive the requests from the web or the Sqoop Client and process the request.
- They delegate most of the business logic to their corresponding
RequestHandler
implementations. All the rest APIs supported via the sqoop-server are documented here: http://sqoop.apache.org/docs/1.99.4/RESTAPI.html#id1
Code Block |
---|
<!-- Version servlet --> <servlet> <servlet-name>VersionServlet</servlet-name> <servlet-class>org.apache.sqoop.server.VersionServlet</servlet-class> <load-on-startup>1</load-on-startup> </servlet> <servlet-mapping> <servlet-name>VersionServlet</servlet-name> <url-pattern>/version</url-pattern> </servlet-mapping> <!-- Generic Configurable servlet --> <servlet> <servlet-name>v1.ConfigurableServlet</servlet-name> <servlet-class>org.apache.sqoop.server.v1.ConfigurableServlet</servlet-class> <load-on-startup>1</load-on-startup> </servlet> <servlet-mapping> <servlet-name>v1.ConfigurableServlet</servlet-name> <url-pattern>/v1/configurable/*</url-pattern> </servlet-mapping> <!-- Connector servlet --> <servlet> <servlet-name>v1.ConnectorServlet</servlet-name> <servlet-class>org.apache.sqoop.server.v1.ConnectorServlet</servlet-class> <load-on-startup>1</load-on-startup> </servlet> <servlet-mapping> <servlet-name>v1.ConnectorServlet</servlet-name> <url-pattern>/v1/connector/*</url-pattern> </servlet-mapping> <!-- Connectors servlet --> <servlet> <servlet-name>v1.ConnectorsServlet</servlet-name> <servlet-class>org.apache.sqoop.server.v1.ConnectorServlet</servlet-class> <load-on-startup>1</load-on-startup> </servlet> <servlet-mapping> <servlet-name>v1.ConnectorsServlet</servlet-name> <url-pattern>/v1/connectors/*</url-pattern> </servlet-mapping> <!-- Driver servlet --> <servlet> <servlet-name>v1.DriverServlet</servlet-name> <servlet-class>org.apache.sqoop.server.v1.DriverServlet</servlet-class> <load-on-startup>1</load-on-startup> </servlet> <servlet-mapping> <servlet-name>v1.DriverServlet</servlet-name> <url-pattern>/v1/driver/*</url-pattern> </servlet-mapping>...... <!-- LinkJob servlet --> <servlet> <servlet-name>v1.LinkServlet<JobServlet</servlet-name> <servlet-class>org.apache.sqoop.server.v1.LinkServlet<JobServlet</servlet-class> <load-on-startup>1</load-on-startup> </servlet> <servlet-mapping> <servlet-name>v1.LinkServlet<JobServlet</servlet-name> <url-pattern>/v1/linkjob/*</url-pattern> </servlet-mapping> <!-- LinksJobs servlet --> <servlet> <servlet-name>v1.LinksServlet<JobsServlet</servlet-name> <servlet-class>org.apache.sqoop.server.v1.LinksServlet<JobsServlet</servlet-class> <load-on-startup>1</load-on-startup> </servlet> <servlet-mapping> <servlet-name>v1.LinksServlet<JobsServlet</servlet-name> <url-pattern>/v1/linksjobs/*</url-pattern> </servlet-mapping> <!-- JobSubmissions servlet --> <servlet> <servlet-name>v1.JobServlet<SubmissionsServlet</servlet-name> <servlet-class>org.apache.sqoop.server.v1.JobServlet<SubmissionsServlet</servlet-class> <load-on-startup>1</load-on-startup> </servlet> <servlet-mapping> <servlet-name>v1.JobServlet<SubmissionsServlet</servlet-name> <url-pattern>/v1/jobsubmissions/*</url-pattern> </servlet-mapping> </web-app> |
- There is an authentication filter to authenticate all request.
Code Block |
---|
<!-- JobsFilter servlet --> <servlet><filter> <servlet-name>v1.JobsServlet</servlet<filter-name>authFilter</filter-name> <servlet<filter-class>org.apache.sqoop.serverfilter.v1.JobsServlet<SqoopAuthenticationFilter</servletfilter-class> <load-on-startup>1</load-on-startup> </servlet> <servlet-mapping> <servlet-name>v1.JobsServlet</servlet-name> <url-pattern>/v1/jobs/*</url-pattern> </servlet-mapping> <!-- Submissions servlet --> <servlet> <servlet-name>v1.SubmissionsServlet</servlet-name> <servlet-class>org.apache.sqoop.server.v1.SubmissionsServlet</servlet-class> <load-on-startup>1</load-on-startup> </servlet> <servlet-mapping> <servlet-name>v1.SubmissionsServlet</servlet-name> <url-pattern>/v1/submissions/*</url-pattern> </servlet-mapping> </web-app></filter> |
- There are two authentication mode supported: simple and Kerberos, which could be set in the sqoop.properties.
Code Block |
---|
#
# Authentication configuration
#
org.apache.sqoop.security.authentication.type=SIMPLE
org.apache.sqoop.security.authentication.handler=org.apache.sqoop.security.Authentication.SimpleAuthenticationHandler
org.apache.sqoop.security.authentication.anonymous=true
#org.apache.sqoop.security.authentication.type=KERBEROS
#org.apache.sqoop.security.authentication.handler=org.apache.sqoop.security.Authentication.KerberosAuthenticationHandler
#org.apache.sqoop.security.authentication.kerberos.principal=sqoop/_HOST@NOVALOCAL
#org.apache.sqoop.security.authentication.kerberos.keytab=/home/kerberos/sqoop.keytab
#org.apache.sqoop.security.authentication.kerberos.http.principal=HTTP/_HOST@NOVALOCAL
#org.apache.sqoop.security.authentication.kerberos.http.keytab=/home/kerberos/sqoop.keytab
#org.apache.sqoop.security.authentication.enable.doAs=true
#org.apache.sqoop.security.authentication.proxyuser.#USER#.users=*
#org.apache.sqoop.security.authentication.proxyuser.#USER#.groups=*
#org.apache.sqoop.security.authentication.proxyuser.#USER#.hosts=* |
Sqoop Request Handlers
Each Sqoop Servlet has its corresponding handler class that handles the request for that servlet. It then internally calls the internal sqoop core/ common code.
Code Block |
---|
public interface RequestHandler { static final String CONNECTOR_NAME_QUERY_PARAM = "cname"; static final String JOB_NAME_QUERY_PARAM = "jname"; JsonBean handleEvent(RequestContext ctx); } public class ConnectorRequestHandler implements RequestHandler { ... } |
All the rest APIs supported via the sqoop-server are documented here: http://sqoop.apache.org/docs/1.99.4/RESTAPI.html#id1
Sqoop Client
- Sqoop Client is represented by the java class
SqoopClient.java
- It has wrapper
ResourceRequest
classes for each sqoop entity, they encapsulate the request/postBody parameters to be sent in the request. Refer to Sqoop 2 (1.99.4) Entity Nomenclature and Relationships for more details on the supported Sqoop Entities. It used the bare bones
HttpURLConnection
object to make requests to the Sqoop-server.Code Block HttpURLConnection conn = new DelegationTokenAuthenticatedURL().openConnection(url, authToken);
Note SqoopClient used to use jersey REST client for making tomcat requests. Recently it was switched to Hadoop-auth/SPENGO for adding Kerberos support that are documented here
https://cwiki.apache.org/confluence/display/SQOOP/Security+Guide+On+Sqoop+2
Run command to start Sqoop Client.
Code Block |
---|
/sqoop.sh client |
- In Kerberos Authentication mode. Kinit is required to set Kerberos environment.
Code Block |
---|
kinit sqoop/server-fqdn@HADOOP.COM |