Summary
We currently have two entities in sqoop that expose configs, connector and driver ( after it was renamed from framework-driver). Driver is not persisted in the repository like the various connectors. The proposal is to change this and treat both of them as configurable entities in the sqoop repository.
Requirements
- Do not use "null" as an primary key indicator to identify Driver
- Connectors and Drivers expose configs and hence both should be in one table and they can uniquely identified by the primary key
- Make sure the driver and its config upgrade works
Design & Implementation Details
Repository Schema changes
- Rename the SQ_CONNECTOR to SQ_CONFIGURABLE
- Rename relevant FK fields to reflect the SQ_CONFIGURABLE
- Driver is now added a first class row in the SQ_CONFIGURABLE, and has a real persistence ID, instead of a fake Id/ null
// SQOOP-1557 upgrade queries for table rename for CONNECTOR-> CONFIGURABLE // drop the SQ_CONFIG FK for connector table public static final String QUERY_UPGRADE_DROP_TABLE_SQ_CONFIG_CONNECTOR_CONSTRAINT = "ALTER TABLE " + TABLE_SQ_CONFIG + " DROP CONSTRAINT " + CONSTRAINT_SQ_CFG_SQC; // drop the SQ_LINK FK for connector table public static final String QUERY_UPGRADE_DROP_TABLE_SQ_LINK_CONSTRAINT = "ALTER TABLE " + TABLE_SQ_LINK + " DROP CONSTRAINT " + CONSTRAINT_SQ_LNK_SQC; // drop the SQ_CONNECTOR_DIRECTION FK for connector table public static final String QUERY_UPGRADE_DROP_TABLE_SQ_CONNECTOR_DIRECTION_CONSTRAINT = "ALTER TABLE " + TABLE_SQ_CONNECTOR_DIRECTIONS + " DROP CONSTRAINT " + CONSTRAINT_SQCD_SQC; // rename public static final String QUERY_UPGRADE_RENAME_TABLE_SQ_CONNECTOR_TO_SQ_CONFIGURABLE = "RENAME TABLE " + TABLE_SQ_CONNECTOR + " TO SQ_CONFIGURABLE"; public static final String QUERY_UPGRADE_RENAME_TABLE_SQ_CONFIG_COLUMN_1 = "RENAME COLUMN " + TABLE_SQ_CONFIG + "." + COLUMN_SQ_CFG_CONNECTOR + " TO " + COLUMN_SQ_CFG_CONFIGURABLE; public static final String QUERY_UPGRADE_RENAME_TABLE_SQ_LINK_COLUMN_1 = "RENAME COLUMN " + TABLE_SQ_LINK + "." + COLUMN_SQ_LNK_CONNECTOR + " TO " + COLUMN_SQ_LNK_CONFIGURABLE; // add a type column to the configurable public static final String QUERY_UPGRADE_TABLE_SQ_CONFIGURABLE_ADD_COLUMN_SQC_TYPE = "ALTER TABLE " + TABLE_SQ_CONFIGURABLE + " ADD COLUMN " + COLUMN_SQC_TYPE + " VARCHAR(32)"; // add the constraints back for SQ_CONFIG public static final String QUERY_UPGRADE_ADD_TABLE_SQ_CONFIG_CONFIGURABLE_CONSTRAINT = "ALTER TABLE " + TABLE_SQ_CONFIG + " ADD CONSTRAINT " + CONSTRAINT_SQ_CFG_SQC + " " + "FOREIGN KEY (" + COLUMN_SQ_CFG_CONFIGURABLE + ") " + "REFERENCES " + TABLE_SQ_CONFIGURABLE + " (" + COLUMN_SQC_ID + ")"; // add the constraints back for SQ_LINK public static final String QUERY_UPGRADE_ADD_TABLE_SQ_LINK_CONFIGURABLE_CONSTRAINT = "ALTER TABLE " + TABLE_SQ_LINK + " ADD CONSTRAINT " + CONSTRAINT_SQ_LNK_SQC + " " + "FOREIGN KEY (" + COLUMN_SQ_LNK_CONFIGURABLE + ") " + "REFERENCES " + TABLE_SQ_CONFIGURABLE + " (" + COLUMN_SQC_ID + ")";
Repository Schema upgrade changes
Upgrade code in derby added for renames of table and field names
Override public void createOrUpgradeRepository(Connection conn) { ..} private void renameConnectorToConfigurable(Connection conn) { // SQ_CONNECTOR to SQ_CONFIGURABLE upgrade runQuery(QUERY_UPGRADE_DROP_TABLE_SQ_CONFIG_CONNECTOR_CONSTRAINT, conn); runQuery(QUERY_UPGRADE_DROP_TABLE_SQ_LINK_CONSTRAINT, conn); runQuery(QUERY_UPGRADE_DROP_TABLE_SQ_CONNECTOR_DIRECTION_CONSTRAINT, conn); runQuery(QUERY_UPGRADE_RENAME_TABLE_SQ_CONNECTOR_TO_SQ_CONFIGURABLE, conn); runQuery(QUERY_UPGRADE_RENAME_TABLE_SQ_CONFIG_COLUMN_1, conn); runQuery(QUERY_UPGRADE_RENAME_TABLE_SQ_LINK_COLUMN_1, conn); runQuery(QUERY_UPGRADE_TABLE_SQ_CONFIGURABLE_ADD_COLUMN_SQC_TYPE, conn); runQuery(QUERY_UPGRADE_ADD_TABLE_SQ_CONFIG_CONFIGURABLE_CONSTRAINT, conn); runQuery(QUERY_UPGRADE_ADD_TABLE_SQ_LINK_CONFIGURABLE_CONSTRAINT, conn); runQuery(QUERY_UPGRADE_ADD_TABLE_SQ_CONNECTOR_DIRECTION_CONSTRAINT, conn); LOG.info("CONNECTOR TABLE altered and constraints added for CONFIGURABLE"); }
- Force register the driver as an entity for the "Driver" upgrade path to work across release
/** * Pre-register Driver since the 1.99.3 release NOTE: This should be used only * in the upgrade path */ @Deprecated protected long registerDriver(Connection conn) { if (LOG.isTraceEnabled()) { LOG.trace("Begin Driver loading."); } PreparedStatement baseDriverStmt = null; try { baseDriverStmt = conn.prepareStatement(STMT_INSERT_INTO_CONFIGURABLE, Statement.RETURN_GENERATED_KEYS); baseDriverStmt.setString(1, MDriver.DRIVER_NAME); baseDriverStmt.setString(2, Driver.getClassName()); baseDriverStmt.setString(3, "1"); baseDriverStmt.setString(4, MConfigurableType.DRIVER.name()); int baseDriverCount = baseDriverStmt.executeUpdate(); if (baseDriverCount != 1) { throw new SqoopException(DerbyRepoError.DERBYREPO_0003, Integer.toString(baseDriverCount)); } ResultSet rsetDriverId = baseDriverStmt.getGeneratedKeys(); if (!rsetDriverId.next()) { throw new SqoopException(DerbyRepoError.DERBYREPO_0004); } return rsetDriverId.getLong(1); } catch (SQLException ex) { throw new SqoopException(DerbyRepoError.DERBYREPO_0009, ex); } finally { closeStatements(baseDriverStmt); } }
Repository API changes
registerDriver
method, makes a actual call to the repository and persists and entry into the SQ_CONFIGURABLEfindDriver
similarly fetches from the repository