Page History

...

-- Query upstream table snapshot for given table and snapshot
SELECT S.database, S.table, S.snapshot_id FROM
    source_snapshot_lineage S
    JOIN sink_snapshot_lineage T
    ON S.job=T.job AND S.barrier_id=T.barrier_id
    where T.`database`='myDatabase' and T.`table`='myTable' and T.snapshot_id=123;

-- Query downstream table snapshot for given table and snapshot
SELECT T.database, T.table, T.snapshot_id FROM
    source_job_lineage S
    JOIN sink_job_lineage T
    ON S.job=T.job AND S.barrier_id=T.barrier_id
    where S.`database`='myDatabase' and S.`table`='myTable' and S.snapshot_id=123;

System Database

...

Implementation

1. Concurrently Writing In Lineage Tables

There will be multiple jobs concurrently writing records to data lineage tables in System Database. To avoid conflicts, the data lineage tables such as source_snapshot_lineage and sink_snapshot_lineage use job name as partition field. The source reader operator and commit operator for each Flink ETL job writes data to different partitions independently.

2. Snapshots In Lineage Tables

3. Actions For Lineage Tables

Tables in system database are read-only and users can query data from system tables by SQL just like a regular table. But users can not alter the system tables and update the data in them. We provide Flink actions for users to delete data from system tables as follows

Table Lineage

We introduce a new option table-lineage for Paimon catalog, users can set this option when they create a new catalog.

...