Versions Compared

Old Version 15

changes.mady.by.user Zelin Yu

Saved on

compared with

New Version 16

changes.mady.by.user Zelin Yu

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Current design of tag just store full data manifests, so it's not able to support streaming reading now. 

Record Data file's Commit Snapshot in ManifestEntry

We choose to record it in DataFileMeta instead. This is because every time Paimon generate a snapshot, it will create new manifest entries for data files. Assuming we record it in ManifestEntry, consider this scenario: we commit data file A to snapshot #1, we will get manifest entry Entry#1 as [ADD, A, commit at #1]. Then we commit -A to snapshot #2, we will get manifest entry Entry#2 as [DELETE, A, ?], which cannot know the commit snapshot for A. So we have to record it to data file meta directly.