Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Motivation

Each Paimon task in Flink/Spark job will create TableWriteImpl for Paimon Table, which uses FileStoreWrite to write data to file store. Paimon will create KeyValueFileStoreWrite for key-value file store and initialize memory pool for the writer.

...

Code Block
languagejava
/**
 * Paimon can allocate heap memory itself, and when the operator exits, JVM will release the allocated heap memory. 
 * But for off-heap memory, Paimon cannot release the memory itself. Engines such as Flink and Spark should allocate 
 * off-heap memory for Paimon, and release the memory when the task goes to termination.
 */ 
@Public
public final class MemorySegment {
    /* Remove this method, Paimon cannot free memory itself. */
    public static MemorySegment allocateOffHeapMemory(int size);

    /* RemoteRemove this method, Paimon cannot free memory itself. */
    public void free();
}

...