Versions Compared

Old Version 5

changes.mady.by.user Veena Basavaraj

Saved on

compared with

New Version 6

changes.mady.by.user Veena Basavaraj

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The rest of the document will focus on the main public facing entities and  repository APIs 

 

Sqoop Entities

Represents the sqoop connector's link information. Link encapsulates the details required to connect to the the data source the connector represents. It has one associated config MLinkConfig

Top Level EntityRepresents the sqoop connector's link information. Link encapsulates the details required to connect to the the data source the connector represents. It has one main component the LINK CONFIG
EntityModelRelationshipDescription
ConfigurableCONFIGURABLEConfigurable.java ( abstract class)

Top Level Entity


Entity Represents a core entity that exposes config objects and used in sqoop job lifecycle.

Configurable have a associated version that acts as a identifier for connector config upgrades.

Code Block
titleMConfigurableType
collapsetrue
/**
 * Represents the sqoop entities that can own configs
 */
public enum MConfigurableType {
  /** Connector as a owner of config keys */
  CONNECTOR,
  /** Driver as a owner of config keys */
  DRIVER;
}
CONNECTORMConnector.java
  • associated with Configurable
  • has HAS 1-n config CONFIG objects

is a type of configurable

There can be many connectors registered to the sqoop server

DRIVER MDriver.java
  • associated with Configurable
  • has HAS 1-n config CONFIG objects

is a type of configurable

 

There is only one Driver object representing sqoop in the system

CONFIGMConfig.java and @Config annotation

 Top Level Entity

MConfigType are the supported config types as of 1.99.5

Code Block
titleMConfigType
collapsetrue
public enum MConfigType {
  /** Unknown config type */
  OTHER,
  @Deprecated
  // NOTE: only exists to support the connector data upgrade path
  CONNECTION,
  /** link config type */ ( should have been  called connector config type !)
  LINK,
  /** Job config type */
  JOB;
}
INPUT

MInput.java an abstract class and @Input annotation

Concrete classes for each supported types

MIntegerInput.java

MStringInput.java

  • Associated with a CONFIG object

Represents the key-value pairs for a given config.

MInputTypes supported are

Code Block
languagejava
titleMInputType
collapsetrue
public enum MInputType {
  /** Unknown input type */
  OTHER,
  /** String input type */
  STRING,
  /** Map input type */
  MAP,
  /** Integer input type */
  INTEGER,
  /** Boolean input type */
  BOOLEAN,
  /** String based input that can contain only predefined values **/
  ENUM,
  ;
}

It holds the key-values for the given InputType.

LINK

MLink.java

MLinkConfig.java		has 1-n configs-inputs objects
  • Associated with a CONNECTOR
  • HAS a CONFIG-INPUT object


JOB

MJob.java

MFromConfig.java

MToConfig.java

MDriverConfig.java

  • HAS 3 CONFIG-INPUT objects
  • HAS

has 1-n configs-inputs objects

has
  • 1-n
submissions
  • SUBMISSIONS

Represents the sqoop job. It encapsulates all the required configs to run the sqoop job.

Primarily the sqoop job has the 3 main components, the FROM, TO and the DRIVER.

FROM and its related FROM-CONFIG MFromConfig represent the config-inputs-values required to Extract data from the source

TO and its related TO-CONFIG MToConfig represent the config-inputs-values required to load data to the destination

DRIVER and its related DRIVER-CONFIG represent MDriverConfig the config-inputs-values required by the execution engine that runs the sqoop job optimally.

 

SUBMISSIONMSubmission.java 

Represents the job run details. Includes the job status, job counters and metrics from the job execution engine

http://sqoop.apache.org/docs/1.99.4/RESTAPI.html#v1-submissions-jname-jname-get-get-submissions-by-job

...