THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Code Block |
---|
/** Basic table lineage which has catalog and table in it. */ public abstract class TableLineage implements ConnectorLineage { /* The catalog of the table lineage. */ public Catalog catalog(); /* The table of the table lineage. */ public CatalogBaseTable table(); } /** Source lineage for table. */ @PublicEvolving public class TableSourceLineage extends TableLineage implements SourceLineage { @Override public StorageIdentifier connector(); /* Output columns for the source and the detailed column information such as data type are in the table. */ public List<String> columns(); } /** Sink lineage for table. */ @PublicEvolving public class TableSinkLineage extends TableLineage implements SinkLineage { @Override public StorageIdentifier connector(); /* The source lineages for the sink table. */ @Override public List<SourceLineage>List<TableSourceLineage> sources(); /* Modify type, INSERT/UPDATE/DELETE. */ String modifyType(); /* Update mode, APPEND/RETRACT/UPSERT. */ String updateMode(); boolean overwrite(); /* The output columns for the sink table. */ public List<String> columns(); /* The source column lineages for each target column in sink table, this will be supported by Flink in the future. */ public Map<String, List<TableColumnLineage>> columnLineages(); /* Source table and columns for the target column in sink lineage. Multiple source table columns would generate one sink column. */ @PublicEvolving public class TableColumnLineage { /* The source table for column lineage. */ public TableSourceLineage source(); /* The columns in source lineage for the sink column. */ public List<String> columns(); } } |
...
Multiple listeners are independent, and client/JobManager will notify the listeners synchronously. It is highly recommended NOT to perform any blocking operation inside the listeners. If blocked operations are required, users need to perform asynchronous processing in their customized listeners.
Rejected Alternatives
Report lineage in JobListener
...