THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
JIRA: SQOOP-1680
| Table of Contents |
|---|
This document serves as a guide for the public facing Sqoop Repository API as of 1.99.5 release
| Warning |
|---|
This API can evolve in future releases and hence it is relevant to the state of the API in 1.99.5 |
...
Summary
Sqoop2 supports a persistent store for the sqoop entities such as the Configurables ( Connector and Driver) , Configs exposed by the Connectors, Jobs and Jobruns JobRuns/Submissions etc. The persistent store is commonly referred to as the repository. We also expose Rest APIs and shell commands to perform CRUD operations on the sqoop entities such as connectors and drivers, connector configs related to link and job information, sqoop job and its configs. Thus the persistent store comes handy in keeping a history of the sqoop entity objects created and updated over time. In order to access the persistent store with ease, we also expose a simple java based repository API that different data stores can implement to store the sqoop entity objects.
...
Entity Related APIs
| Entity | APIs | Notes |
|---|---|---|
CONNECTOR | public abstract MConnector registerConnector(MConnector mConnector, boolean autoUpgrade); public abstract List<MConnector> findConnectors(); | READ ONLY APIs.
|
DRIVER | public abstract MDriver registerDriver(MDriver mDriverConfig, boolean autoUpgrade); public abstract MDriver findDriver(String shortName); | READ ONLY APIs. |
LINK | public abstract void createLink(MLink link); public abstract void updateLink(MLink link); public abstract void updateLink(final MLink link, RepositoryTransaction tx); public abstract void enableLink(long id, boolean enabled); public abstract void deleteLink(long id); public abstract MLink findLink(long id); public abstract List<MLink> findLinksForConnector(long connectorId); public abstract List<MLink> findLinks(); | CRUD APIs |
JOB | public abstract void createJob(MJob job); public abstract void updateJob(MJob job); public abstract void updateJob(MJob job, RepositoryTransaction tx); public abstract void enableJob(long id, boolean enabled); public abstract void deleteJob(long id); public abstract MJob findJob(long id); public abstract MJob findJob(String name); public abstract List<MJob> findJobs(); public abstract List<MJob> findJobsForConnector(long connectorId); | CRUD APIs |
CONFIG | None |
|
| INPUT ( VALUES) | public abstract void deleteJobInputs(long jobId, RepositoryTransaction tx); public abstract void deleteLinkInputs(long linkId, RepositoryTransaction tx); | No Public API for users yet See SQOP-1516 for more details - 1.99.5 changes got Input RU
Input deletion can happen as part of the connector/driver upgrade path |
| SUBMISSION | public abstract void createSubmission(MSubmission submission); public abstract void updateSubmission(MSubmission submission); public abstract void purgeSubmissions(Date threshold); public abstract List<MSubmission> findUnfinishedSubmissions(); public abstract List<MSubmission> findSubmissions(); public abstract List<MSubmission> findSubmissionsForJob(long jobId); public abstract MSubmission findLastSubmissionForJob(long jobId); | CUD for internal sqoop use only READ ONLY APIs for user |
...
| Code Block |
|---|
/**
* Upgrade the connector with the same {@linkplain MConnector#uniqueName}
* in the repository with values from <code>newConnector</code>.
* <p/>
* All links and jobs associated with this connector will be upgraded
* automatically.
*
* @param oldConnector The old connector that should be upgraded.
* @param newConnector New properties for the Connector that should be
* upgraded.
*/
public final void upgradeConnector(MConnector oldConnector, MConnector newConnector) {
..}
/**
* Update the connector with the new data supplied in the
* <tt>newConnector</tt>. Also Update all configs associated with this
* connector in the repository with the configs specified in
* <tt>mConnector</tt>. <tt>mConnector </tt> must
* minimally have the configurableID and all required configs (including ones
* which may not have changed). After this operation the repository is
* guaranteed to only have the new configs specified in this object.
*
* @param newConnector The new data to be inserted into the repository for
* this connector.
* @param tx The repository transaction to use to push the data to the
* repository. If this is null, a new transaction will be created.
* method will not call begin, commit,
* rollback or close on this transaction.
*/
protected public abstract void upgradeConnectorAndConfigs(MConnector newConnector, RepositoryTransaction tx);
|
...
| Code Block |
|---|
public final void upgradeDriver(MDriver driver) {
..}
/**
* Upgrade the driver with the new data supplied in the
* <tt>mDriver</tt>. Also Update all configs associated with the driver
* in the repository with the configs specified in
* <tt>mDriver</tt>. <tt>mDriver </tt> must
* minimally have the configurableID and all required configs (including ones
* which may not have changed). After this operation the repository is
* guaranteed to only have the new configs specified in this object.
*
* @param newDriver The new data to be inserted into the repository for
* the driverConfig.
* @param tx The repository transaction to use to push the data to the
* repository. If this is null, a new transaction will be created.
* method will not call begin, commit,
* rollback or close on this transaction.
*/
publicprotected abstract void upgradeDriverAndConfigs(MDriver newDriver, RepositoryTransaction tx); |
...