THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
| CONFIGURABLE | LINK-CONFIG | JOB-CONFIG |
|---|---|---|
| CONNECTOR | (1) LINK-CONFIG
| (2) FROM-CONFIG
TO-CONFIG
|
| DRIVER | NONE | (1) DRIVER-CONFIG
|
Requirements
- Read the Config Inputs by Type/SubType and By Job /Submission ( since SQOOP-2025 we may be able to have configs by submissionId)
- Update the Config Inputs by Type/SubType for the latest/last submission in the job. We should not allow editing previous submissions and it should be read only
- Only the "inputs" with attribute "USER-ONLY" or "ANY" as per SQOOP-1804 will be editable. Make sure to validate the condition of editable and adhere to cascading changes depending on the overrides attribute on each input.
- Once the input values are edited, the new values will be used in the next job run, unless we maintain history as per SQOOP-2025.
NOTE : Support all the above in both shell command and Rest-API.
Non Goals
- Supporting CD ( create / delete ) of Config Inputs via the REST or command line. It is only allowed via the configurable code and supported annotations on the classes today and it should remain so.
- Editing submission history
Design and Implementation Details
Shell Commands
SubType for MConfigType
SubTypes Aliases have been added to the MConfigType Enum to indicate the sub types
| Code Block |
|---|
/** * Represents the various config types supported by the system. */ @InterfaceAudience.Private @InterfaceStability.Unstable public enum MConfigType { /** Unknown config type */ OTHER, @Deprecated // NOTE: only exists to support the connector data upgrade path CONNECTION, /** link config type */ LINK("link"), /** Job config type */ JOB("from", "to", "driver"); private List<String> |
...
subType; MConfigType(String... aliases){ this. |
...
subType = Arrays.asList(aliases); } |
...
List<String> getSubTypes(MConfigType type) { return type. |
...
subType; } } |
Shell Commands
Read Config By Type and Job or Submission
...
// NOTE: all the job config inputs values are for the last job run only since we do not store the config values for each submission yet
| Code Block |
|---|
1. show config foo --type JOB --subType from --id |
...
1 //*. show config "foo" --type JOB --subType "from" --sid 1 ( SINCE we are not doing SQOOP-2025, this will not be in the current patch) 2. show config foo --type JOB --subType to --id |
...
1 3. show config foo --type JOB --subType driver --id 1 4. show config foo --type LINK --subType link --id 1 // planned, SQOOP-2046 as a sub ticket of SQOOP-1516 5. show input "foo" --config bar --type LINK --id 1 // id here refers to the link id |
Edit Config By Type and Job
NOTE: Prev ( prev submissions cannot be edited, hence we restrict editing to the last job run only)
| Code Block |
|---|
1. edit config foo --type JOB --subType from --id 1 // id here refers to the job |
...
id // planned, SQOOP- |
...
2046 as a sub ticket of SQOOP-1516 2. edit input foo --config bar --type LINK --id 1 // id here refers to the link id |
Rest API changes
Read Config By Type and Job or Submission
| Code Block |
|---|
GET v1/config/JOB?name=?&Id=?&subType= or GET v1/config?type=JOB&name=?&Id=?&subType= |
Edit Config By Type and Job
| Code Block |
|---|
POST v1/config/LINK?name=?&id=?&subType= |
...
or POST v1/config?type=LINK&name=?&id=?&subType= |
Repository API changes
...
Add new API to get config inputs by jobId and type, name
Add new API to edit/post config inputs by jobId and type, name
| Code Block |
|---|
public abstract MConfig findFromJobConfig(long jobId, String name); public abstract MConfig findToJobConfig(long jobId, String name); public abstract MConfig findDriverJobConfig(long jobId, String name); public abstract MConfig findLinkConfig(long linkId, String name); public abstract void updateFromJobConfig(long jobId, MConfig config); public abstract void updateDriverobConfig(long jobId, MConfig config); public abstract void updateToJobConfig(long jobId, MConfig config); public abstract void updateLinkConfig(long linkId, MConfig config); |
Testing
Unit test are almost non existent for the shell code. Hence we will rely on the basic manual testing
...