Page History

...

JIRA: https://issues.apache.org/jira/browse/SQOOP-1516

Summary

Configs are exposed in code via the Connectors and Drivers ( the two CONFIGURABLES supported). They annotate the config classes with "@Config" annotation and that is how sqoop registers these entities into the repository during the server startup. If a connector is already existing in the sqoop repository (or during upgrade path) then the connector's upgrade API is invoked to update the attributes of the config object.

...

     +----------------------------+
     | SQ_INPUT                   |
     +----------------------------+
     | SQI_ID: BIGINT PK AUTO-GEN |
     | SQI_NAME: VARCHAR(64)      |
     | SQI_CONFIG: BIGINT         |FK SQ_CONFIG(SQ_CFG_ID)
     | SQI_INDEX: SMALLINT        |
     | SQI_TYPE: VARCHAR(32)      |"STRING"|"MAP"
     | SQI_STRMASK: BOOLEAN       |
     | SQI_STRLENGTH: SMALLINT    |
     | SQI_ENUMVALS: VARCHAR(100) |
     +----------------------------+
 
   +----------------------------+
     | SQ_LINK_INPUT              |
     +----------------------------+
     | SQ_LNKI_LINK: BIGINT PK    | FK SQ_LINK(SQ_LNK_ID)
     | SQ_LNKI_INPUT: BIGINT PK   | FK SQ_INPUT(SQI_ID)
     | SQ_LNKI_VALUE: LONG VARCHAR|
     +----------------------------+
     +----------------------------+
     | SQ_JOB_INPUT               |
     +----------------------------+
     | SQBI_JOB: BIGINT PK        | FK SQ_JOB(SQB_ID)
     | SQBI_INPUT: BIGINT PK      | FK SQ_INPUT(SQI_ID)
     | SQBI_VALUE: LONG VARCHAR   |
     +----------------------------+


 

The following table lists the type and number in (*) of configs exposed by each of the configurables. Each config object is represented as a list. Hence a connector can expose a FROM-CONFIG with more than one config objects in it. 

...

The current design proposal enhancement proposal to the existing functionality ( command line and rest apis)  to support reuse of config objects by providing hooks to perform RU ( Read and Update) operations on the config input objects independently. 

Requirements

• Read and Update the Config Inputs by Type and By Job

Non Goals

• ConfigSupporting CD ( create / delete ) of Config Inputs via the REST or command line. It is only allowed via the configurable code and supported annotations on the classes.

Design

Shell Commands

Read Config By Type and Job

show config --type from --jid 1
 

Edit Config By Type and Job

edit config --type from --jid 1

Rest API 

Read Config By Type and Job

GET
v1/config/link?configurableId=?&direction type= ( get all the config details for the given configurable )

Edit Config By Type and Job

POST

v1/config/link?configurableId=?&direction type= ( post data for the link config object)

Testing

• Unit test are almost non existent for the shell code. Hence we will rely on the basic manual testing
• Rest APIs can be tested via integration tests and that will be part of the work proposed