Page History

...

The Command#run method will translate the logical plan to hoodie’s api call. For example the InsertIntoHudiTableCommand will translate to hoodie’s dataframe api for  insert data to hoodie.

Process for meta fields

Hoodie will append five meta fields to the head of the table schema. These are table property fields. Users may not use them in the insert statement, However spark sql will invalidate

the target table fields size  which contains the meta fields with the select statement output field size which will result to size not match exception. eg.  insert into h0 select 1, 'a1', 10 from s

will failed in the validate stage, Because the select statement does not have the meta fields while the h0 contains.

In order to solve this problem, we add a rewrite rule for the insert logical relation which will append the five meta fields to the head of the select projects.

HoodieSparkSessionExtension

...