...
- Sqoop-server uses tomcat web server, it is very bare bones.
The main entry point is the TomcatToolRunner, it bootstraps the tomcat and loads all the sqoop related classes into its class path. It is invoked from the bash script .
Code Block /sqoop.sh server start
- The main hook for the sqoop server to start is this entry in the web.xml. Tomcat invokes it callbacks as it bootstraps and we use the contextInitialized callback to initialize all the related code.
Code Block |
---|
...
<!-- Listeners --> <listener> <listener-class>org.apache.sqoop.server.ServerInitializer</listener-class> </listener> |
Sqoop Server
- The sqoop server is represented by the java class
SqoopServer.java
- SqoopServer.initialize() is the
- called from the
ServerInitiaizer
- SqoopServer.destroy() is called when the tomcat server is shutdown
Sqoop Servlets
There are a bunch of servlets We have servlets for each Sqoop Entity that is exposed via the REST API in web.xml : https://github.com/apache/sqoop/blob/sqoop2/server/src/main/webapp/WEB-INF/web.xml
They receive the requests and process it. Each Servlet has its corresponding handler class that handles the request for that servlet. It then internally calls the internal sqoop core/ common code.
Sqoop Request Handlers
. Refer to Sqoop 2 (1.99.4) Entity Nomenclature and Relationships for more details on the supported Sqoop Entities.
They receive the requests from the web or the Sqoop Client and process the request. The delegate most of the business logic to their corresponding RequestHandler
implementations.
Code Block |
---|
<!-- Version servlet -->
<servlet>
<servlet-name>VersionServlet</servlet-name>
<servlet-class>org.apache.sqoop.server.VersionServlet</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>VersionServlet</servlet-name>
<url-pattern>/version</url-pattern>
</servlet-mapping>
<!-- Generic Configurable servlet -->
<servlet>
<servlet-name>v1.ConfigurableServlet</servlet-name>
<servlet-class>org.apache.sqoop.server.v1.ConfigurableServlet</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>v1.ConfigurableServlet</servlet-name>
<url-pattern>/v1/configurable/*</url-pattern>
</servlet-mapping>
<!-- Connector servlet -->
<servlet>
<servlet-name>v1.ConnectorServlet</servlet-name>
<servlet-class>org.apache.sqoop.server.v1.ConnectorServlet</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>v1.ConnectorServlet</servlet-name>
<url-pattern>/v1/connector/*</url-pattern>
</servlet-mapping>
<!-- Connectors servlet -->
<servlet>
<servlet-name>v1.ConnectorsServlet</servlet-name>
<servlet-class>org.apache.sqoop.server.v1.ConnectorServlet</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>v1.ConnectorsServlet</servlet-name>
<url-pattern>/v1/connectors/*</url-pattern>
</servlet-mapping>
<!-- Driver servlet -->
<servlet>
<servlet-name>v1.DriverServlet</servlet-name>
<servlet-class>org.apache.sqoop.server.v1.DriverServlet</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>v1.DriverServlet</servlet-name>
<url-pattern>/v1/driver/*</url-pattern>
</servlet-mapping>
<!-- Link servlet -->
<servlet>
<servlet-name>v1.LinkServlet</servlet-name>
<servlet-class>org.apache.sqoop.server.v1.LinkServlet</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>v1.LinkServlet</servlet-name>
<url-pattern>/v1/link/*</url-pattern>
</servlet-mapping>
<!-- Links servlet -->
<servlet>
<servlet-name>v1.LinksServlet</servlet-name>
<servlet-class>org.apache.sqoop.server.v1.LinksServlet</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>v1.LinksServlet</servlet-name>
<url-pattern>/v1/links/*</url-pattern>
</servlet-mapping>
<!-- Job servlet -->
<servlet>
<servlet-name>v1.JobServlet</servlet-name>
<servlet-class>org.apache.sqoop.server.v1.JobServlet</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>v1.JobServlet</servlet-name>
<url-pattern>/v1/job/*</url-pattern>
</servlet-mapping>
<!-- Jobs servlet -->
<servlet>
<servlet-name>v1.JobsServlet</servlet-name>
<servlet-class>org.apache.sqoop.server.v1.JobsServlet</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>v1.JobsServlet</servlet-name>
<url-pattern>/v1/jobs/*</url-pattern>
</servlet-mapping>
<!-- Submissions servlet -->
<servlet>
<servlet-name>v1.SubmissionsServlet</servlet-name>
<servlet-class>org.apache.sqoop.server.v1.SubmissionsServlet</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>v1.SubmissionsServlet</servlet-name>
<url-pattern>/v1/submissions/*</url-pattern>
</servlet-mapping>
</web-app> |
Sqoop Request Handlers
Each Sqoop They receive the requests and process it. Each Servlet has its corresponding handler class that handles the request for that servlet. It then internally calls the internal sqoop core/ common code.
Code Block |
---|
public interface RequestHandler { static final String CONNECTOR_NAME_QUERY_PARAM = "cname"; static final String JOB_NAME_QUERY_PARAM = "jname"; JsonBean handleEvent(RequestContext ctx); } public class ConnectorRequestHandler implements RequestHandler { ... } |
All the rest APIs supported via the sqoop-server are documented here: http://sqoop.apache.org/docs/1.99.4/RESTAPI.html#id1
Sqoop Client
There is also the Sqoop-client that used to invoke the Sqoop-server methods via the jersey REST client,- Sqoop Client is represented by the java class
SqoopClient.java
- It has wrapper
ResourceRequest
classes for each sqoop entity, they encapsulate the request/postBody parameters to be sent in the request. Refer to Sqoop 2 (1.99.4) Entity Nomenclature and Relationships for more details on the supported Sqoop Entities. It used the bare bones
HttpURLConnection
object to make requests to the Sqoop-server.Code Block HttpURLConnection conn = new DelegationTokenAuthenticatedURL().openConnection(url, authToken);
Note SqoopClient used to use jersey REST client for making tomcat requests. Recently it was switched to Hadoop-auth/SPENGO for adding Kerberos support that are documented