THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Definition
A storage model / table type where commitA def~table-type where a def~table's def~commits are merged when read (#todo improve)
Design details
...
into def~table when read / viewed / queried.
This can be seen as "delayed ingestion": "compaction" happens delayed, on demand.
#todo improve to summarize semantics relative to def~commits lifecycle, before and after
Design details
| Excerpt |
|---|
In this def~table-type, records written to the def~table, are quickly first written to def~log-files, which are at a later time merged with the def~base-file, using a def~compaction action on the timeline. Various def~query-types can be supported depending on whether the query reads the merged snapshot or the change stream in the logs or the un-merged base-file alone. At a high level, |
...
135860486 writer goes through same stages as |
...
...
on-write (COW) writer in ingesting data. The |
...
updates are appended to latest log (delta) file belonging to the latest file slice without merging. For inserts, Hudi supports 2 modes: |
...
|
...
|
...
|
...
|
...
|
...
As in the case of |
...
...
on-write (COW), the input tagged records are partitioned such that all upserts destined to |
...
a def~file-id are grouped together. This upsert-batch is written as one or more log-blocks written to |
...
def~log-files. Hudi allows clients to control log file sizes |
...
. |
...
The WriteClient API is same for |
...
both def~copy- |
...
on-write (COW) and 135860486 writers. With |
...
135860486, several rounds of data-writes would have resulted in accumulation of one or more log-files. All these log-files along with base-parquet (if exists) constitute |
...
a def~file-slice which represents one complete version of the file. This table type is the most versatile, highly advanced and offers much flexibility for writing (ability specify different compaction policies, absorb bursty write traffic etc) and querying (e.g: tradeoff data freshness and query performance). At the same time, it can involve a learning curve for mastering it operationally. |
Kind of
...
Related concepts
...