Table of Contents

Status

Current state: [One of "Under Discussion", "Accepted", "Rejected"]

...

Also, the user-side http connection service can be extended based on this interface in next stage.

3-Metadata Cache

In the previous implementation, all external tables were persisted in Doris' metadata.

In the new implementation, these metadata are no longer persistent, but the metadata of the external data source is obtained through real-time access to the datasource.

Of course, this may have performance issues, so we need to design a metadata caching strategy to improve efficiency.

4-Qualifier

The qualified names in SQL is divided into three categories:

...

In this way, in order not to affect the original syntax, we need to add an additional syntax similar to `use db`, first set the datasource to which the current session belongs, and then all non-query operations are completed within this datasource:

`switch datasource`

...

5-Privilege

The current Doris privilege management refers to the hierarchical design of mysql, with only has 3 level(global, database and table). Here we need to add a new level for data source.

...

6-Forward Compatibility

This solution needs to ensure that the original cluster can be upgraded normally.
First, after the old cluster is upgraded, all existing databases in the original cluster will be included under the default internal datasource. This ensures that the original function is completely unchanged. New datasources are created through the new create datasource syntax.

In addition, the previous external table functions are all retained, which can be understood as the external tables created before belong to the Internal Datasource.

...

7-Information_schema

information_schema remains unique across the cluster. Because the datasource level is already included in the information_schema, multiple datasources can be stored through the information_schema.

Detailed Design

1-Create datasource

create [external] catalog my_datasource properties("k" = "v", ...);
drop catalog my_datasource;
show catalogs;
modify catalog my_datasource set("k" = "v", ...);

2-Privilege Management

the detailed design of the function.

Scheduling

...

# No additional syntax will be added. If you need to authorize privilege under other datasource, you need to switch ds first.
# During initialization, you can use the admin account to switch to the new ds to create the permissions of the initial account.
grant all on db.tbl to user@%;

3-Basic data structure

Scheduling

1. Define basic data structures, including DataSource interface, ExternalScanNode, ExternalTable, etc.
2. POC: Access to the hive metastore data source using these basic data structure.
3. Support dynamic loading user defined datasource using SPI.
4. SQL parser: Support datasource qualifier.
5. Privilege: Support privilege management for datasource level.
6. Metadata Cache