Page History

...

• Configs should be treated as a top level entity as they become more sophisticated with the SQOOP-1804 and in future with the implementation of SQOOP-1643. Currently editing and reading config/ config inputs can only happen via a job or link. This proposal aims at making the config objects as first class citizens, so they can be read and edited by their unique name.
• Config objects per type are MConfigType (i,e JOB and LINK)  are lists. So ability to edit per "CONFIG" is name  is easier than having to edit per "CONFIG LIST" associated with the type. Users do see the names when they list the configs per connector, we can also have a shell command/ rest-API to list all the configs per configurable to make it easier for users to reference configs by name
• Read  the Config Inputs by Type/SubType and By Job /Submission ( since SQOOP-2025 we may be able to have configs by submissionId)
• Update the Config Inputs by Type/SubType for the latest/last submission in the job. We should not allow editing previous submissions and it should be read only
• Only the "inputs" with attribute "USER-ONLY" or "ANY" as per SQOOP-1804 will be editable. Make sure to validate the condition of editable and adhere to cascading changes depending on the overrides attribute on each input.
• Once the input values are edited, the new values will be used in the next job run, unless we maintain history as per SQOOP-2025.

...

1. show config foo --type JOB --subType from --id 1 
  //*. 1 or show config 1 "foo" --type JOB --subType "from" --sidid 1 ( SINCENOTE: we arecan notuse doing SQOOP-2025, this will not be in either id or name )
  //*. show config "foo" --type JOB --subType "from" --sid 1 ( SINCE we are not doing SQOOP-2025, this will not be in the current patch) 
 
2. show config foo --type JOB --subType to --id 1 
 
3. show config foo --type JOB --subType driver --id 1
 
4. show config foo --type LINK --subType link --id 1   
 
//  planned, SQOOP-2046 as a sub ticket of SQOOP-1516
5. show input "foo" --config bar --type LINK --id 1 // id here refers to the link id    

...

1. edit config foo --type JOB --subType from --id 1 // id here refers to the job id 
 
// planned, SQOOP-2046 as a sub ticket of SQOOP-1516
2. edit input foo --config bar --type LINK --id 1 // id here refers to the link id
 

...

ALTERNATIVELY, It was suggested that for command line it is NOT good not to introduce config as a top level ( It was a preference from Gwen Shapira and Qian Xu).  Hence the command are

edit job --jid 1 --config foo --type from 
show job --jid 1 --config foo --type from 
edit job --jid 1 --config foo --type to
edit job --jid 1 --config foo --type driver
 
edit link --lid 1 --config foo --type link 



 

Rest API changes

The resource returned from the APIs is a config object  MConfig. 

Read Config By Type and Job or Submission

1. GET v1/config/JOBjob?name=?&Id=?&subTypetype=   
 
or   
 
2. GET v1/config?type=JOB&name=?&Id=?&subType=

Edit Config By Type and Job

1. POST v1/config/LINKlink?name=?&id=?&subTypetype=  
 
 or 
 
2. POST v1/config?type=LINK&name=?&id=?&subType=

We chose #1 in both the cases

Repository API changes

Add new API to get config inputs by jobId and type, name

...

  public abstract MConfig findFromJobConfig(long jobId, String name);
  public abstract MConfig findToJobConfig(long jobId, String name);
  public abstract MConfig findDriverJobConfig(long jobId, String name);
  public abstract MConfig findLinkConfig(long linkId, String name);
  public abstract void updateFromJobConfig(long jobId, MConfig config);
  public abstract void updateDriverobConfig(long jobId, MConfig config); name);
// the type exists to check who is editing the configs, the connector code via upgrade or the user ( from rest / command line )
  public abstract void updateToJobConfigupdateJobConfig(long jobId, MConfig config, MConfigUpdateEntity type);
  public abstract void updateLinkConfig(long linkId, MConfig config, MConfigUpdateEntity type));

Testing

Unit test are almost non existent for the shell code. Hence we will rely on the basic manual testing

...