Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

For example, there is a wide table t1 which has a primary key `a`:{

Code Block

...

languagesql
 create table t1 (
    `a` BIGINT,
    `b` STRING,
    `c` STRING,
    `d` STRING,
    `e` DOUBLE,
    `f` BIGINT,
    `g` INT,
    primary key(`a`) not enforced
  ) with (
    ...
  )

  create table t1 (
    `a` BIGINT,
    `b` STRING,
    `c` STRING,
    `d` STRING,
    `e` DOUBLE,
    `f` BIGINT,
    `g` INT,
    primary key(`a`) not enforced
  ) with (
    ...
  )

{code}

where column b, c, d from s1 table, column e, f, g from s2 table,
by different processing data from the source table s1 and s2,  the two insertions specify a different column list written to the wide table, ideally the two insertions will not affect each others' column{

Code Block

...

languagesql
insert into t1 (a, b, c, d)

...


select a, b, c, d from s1 where ...;

...



insert into t1 (a, e, f, g)

...


select a, e, f, g from s2 where ...;

{code}The current problem is that connectors cannot distinguish whether the null value of a column is really from the user's data or whether it is a null value populated because of partial insert behavior.

Let's see the execution plan fragment corresponding to the first insert statement in the above example:{

Code Block

...

languagetext
Sink(table=[default_catalog.default_database.t1], fields=[a, b, c, d, EXPR$3, EXPR$4, EXPR$5])

...


+- Calc(select=[a, b, c, d, null:VARCHAR(2147483647) AS EXPR$3, null:BIGINT AS EXPR$4, null:INTEGER AS EXPR$5])

{code}
The current connector implementor has no way of knowing that the last three fields were added by the planner and not from real user data. The user has to declare several different schemas (containing only partial column information and no overlap except for the primary key) to get around the current problem. 

By adding targetColumnList information to the DynamicTableSink#Context, this problem can be solved.

Public Interfaces

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

...