Motivation

Load Action is a synchronous import method where users send requests via HTTP protocol to import local files or data streams into Paimon. Load Action executes the import synchronously and returns the results of the import. Users can directly determine the success of the import from the response body of the request.

Load Action is primarily suitable for importing local files.

SCENARIO

Analysts and business development personnel often encounter temporary small-scale data scenarios in their work. Faced with one or multiple CSV files, these files typically have small data volumes and are scattered across various systems. The traditional process for handling these files involves first uploading them to OSS storage, then using Flink or Hive SQL for table creation and data import, before analysis can begin. This process is not only complex and tedious but also requires specific prior knowledge. To simplify this process, we have introduced the 'Load Action' service. This service enables the rapid import of small-scale data, directly creating Paimon tables, without the dependence on any other data processing engines.

Data is submitted and transmitted via the HTTP protocol. This is demonstrated here using the curl command to show how to submit data for import.

Users can also operate using other HTTP clients.

curl --location-trusted -u user:passwd [-H ""...] -T data.file -XPUT http://load_host:http_port/api/{db}/{table}/load

Proposed Changes

public interface WriteStrategy<T> extends Serializable {
   
    void init(Configuration conf) throws WriteStrategyException;

    
    void write(Table table, T record) throws WriteStrategyException;

   
    Schema defineTableSchema() throws WriteStrategyException;

    
    Catalog fetchTableCatalog(String path) throws WriteStrategyException;

    
    void updateTableCatalog(Schema schema, Identifier identifier) throws WriteStrategyException;

   
    List<String> listFileNames(String path) throws WriteStrategyException;
}

Compatibility, Deprecation, and Migration Plan

No

Test Plan

UT and IT

Rejected Alternatives

No