Versions Compared

Old Version 8

changes.mady.by.user Veena Basavaraj

Saved on

compared with

New Version 9

changes.mady.by.user Veena Basavaraj

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

CONFIGURABLELINK-CONFIGJOB-CONFIG
CONNECTOR

(1)

LINK-CONFIG

MLinkConfiggListMLinkConfigList

(2)

FROM-CONFIG

MFromConfigList

TO-CONFIG

MToConfiggListMToConfigList

DRIVERNONE

1

DRIVER-CONFIG

MDriverConfiggListMDriverConfigList

 

The current proposal enhances existing functionality ( command line and rest apis)  to support reuse of config objects by providing hooks to perform RU ( Read and Update) operations on the config input objects independently. 


Requirements
  • Read and Update the Config Inputs by Type and By Job /Submission ( since SQOOP-2025 we may be able to have configs by submissionId)
  • Support this in both shell command and Rest-API.
  • Only the "inputs" with attribute "USER-ONLY" or "ANY" as per SQOOP-1804 will be editable. 
  • Once the input values are edited, the new values will be used in the next job run, unless we maintain history as per SQOOP-2025.

Non Goals

  • Supporting CD ( create / delete ) of Config Inputs via the REST or command line. It is only allowed via the configurable code and supported annotations on the classes today and it should remain so.

Design and Implementation Details

Shell Commands

A new Enum will be added to list the config types - 

 from, to, link, driver

Read Config By Type and Job or Submission 

 

Code Block
// Supported type values ( from, to, link, driver )



1. show config --type from --jid 1 - will provide the config value for the last job run
2. show config --type from --sid 1




 

 

Edit Config By Type and Job ( prev submissions cannot be edited, hence we restrict editing to the last job run only)

Code Block
1. edit config --type from --jid 1 - will provide the config value for the last job run
// we can only edit the last job run values

...

Supporte type values ( from, to, link, driver )
...
Rest API changes
Read Config By Type and Job or Submission
 Code Block
GET
...

v1/config?jId=?&type=
...
 
Edit Config By Type and Job
 Code Block
POST
...

v1/config/link?jId=?&type=
Repository API changes
  • Add new API to get config inputs by submissionId and type ( read-only)
  • Add new API to get config inputs by jobId and type
  • Add new API to edit/post config inputs by jobId and type
Testing
  • Unit test are almost non existent for the shell code. Hence we will rely on the basic manual testing
  • Rest APIs can be tested via integration tests and that will be part of the work proposed
...