Current state: Released
Discussion thread:
JIRA or Github Issue:
Released: 1.2.0
Google Doc: <If the design in question is unclear or needs to be discussed and reviewed, a Google Doc can be used first to facilitate comments from others.>
Doris currently supports the external table. This feature is through the mapping of metadata, so that Doris can access external data sources. We take the iceberg data source as an example, which has two metadata mapping methods:
We can see that the above two methods complete the metadata mapping at the database level and table level respectively. If the iceberg cluster has multiple databases, at least we need to manually establish multiple database-level mapping relationships. This is inconvenient to use in some scenarios.
In many scenarios, the user hopes to implement the "data source" level mapping, that is, through the "data source" level mapping relationship, Doris can directly access all data under the corresponding data source, including database and table. In this way, the user can rely on Doris and have unified access to multiple "data sources".
Therefore, the motivation of this feature is to add a new metadata level called "datasource" to Doris to support connecting and accessing to other data sources.
Just like Presto, there is Catalog→ Database→Table hierarchical relationships to manage metadata.
In Doris, I would like to introduce a new metadata called DataSource, which is exactly same as Catalog in Presto.
The current metadata hierarchy of Doris is Database->Table->Column.
We add a new level: DataSource->Database->Table→Column.
A DataSource represents a data source, such as an Iceberg cluster, a Hive cluster, an ES cluster, or another Doris cluster.
We can define data sources into the following two categories:
Here we discuss about how to manager datasources, for example, how to add a new databases.
To sum up, we will combine the above methods to carry out the first stage design: Built-in data source + SPI interface
Also, the user-side http connection service can be extended based on this interface in next stage.
In the previous implementation, all external tables were persisted in Doris' metadata.
In the new implementation, these metadata are no longer persistent, but the metadata of the external data source is obtained through real-time access to the datasource.
Of course, this may have performance issues, so we need to design a metadata caching strategy to improve efficiency.
The qualified names in SQL is divided into three categories:
After adding the datasource level, all qualified names need to add a extra level.
For Tables and UDFs, adding one level does not create semantic ambiguity:
For Column, because we will support nested types in the future, and fieds of nested types are still connected using DOT. Then there may be ambiguity here, give an example.
Column col: Struct type Col <M1, M2>
Then it might be written as follows:
You can see that the second and third formats are the same, but the content user want to express is different. So we need a priority-based qualified name resolution rule.
3 parts or less: Match from right to left. This approach ensures forward compatibility. After the datasource level is added, the user does not need to change the original SQL statement.
Example:
4 parts or more: Match from left to right.
Example:
The modification of the qualified name should only affect the SQL query, because we want to support data federation query between multiple datasources.
But for other functions, we can temporarily not support cross-datasource operations, such as DDL and DML.
In this way, in order not to affect the original syntax, we need to add an additional syntax similar to `use db`, first set the datasource to which the current session belongs, and then all non-query operations are completed within this datasource:
`switch datasource`
The current Doris privilege management refers to the hierarchical design of mysql, with only has 3 level(global, database and table). Here we need to add a new level for data source.
This solution needs to ensure that the original cluster can be upgraded normally.
First, after the old cluster is upgraded, all existing databases in the original cluster will be included under the default internal datasource. This ensures that the original function is completely unchanged. New datasources are created through the new create datasource syntax.
In addition, the previous external table functions are all retained, which can be understood as the external tables created before belong to the Internal Datasource.
information_schema remains unique across the cluster. Because the datasource level is already included in the information_schema, multiple datasources can be stored through the information_schema.
create [external] catalog my_datasource properties("k" = "v", ...);
drop catalog my_datasource;
show catalogs;
modify catalog my_datasource set("k" = "v", ...); |
# No additional syntax will be added. If you need to authorize privilege under other datasource, you need to switch ds first. # During initialization, you can use the admin account to switch to the new ds to create the permissions of the initial account. grant all on db.tbl to user@%; |
+---------------------+
+--->|HMSExternalDataSource|
| +---------------------+
+-----------------------+ | +---------------------+
|ExternalDataSource | +--->|EsExternalDataSource |
| +---+ +---------------------+
+-----> listDatabaseNames()| | +---------------------++
| | tableExists() | +--->|OdbcExternalDataSource|
+-------------------+ | | getTableSchema() | | +----------------------+
| DataSourceIf | | | ... | | +----------------------+
| +----+ +-----------------------+ +--->|HttpExternalDataSource|
| getDatabase() | +----------------------+
| listDatabases()+----+ +-----------------------+
| ... | | |InternalDataSource |
+-------------------+ | | |
+-----> self-managed meta |
| |
+-----------------------+
+-------------------+ +-----------------------+
| DatabaseIf +---------->ExternalDatabase |
| | +-----------------------+
| getTable() |
| listTable() | +-----------------------+
| ... +---------->Database |
+-------------------+ +-----------------------+
+-------------------+ +-----------------------+
| TableIf +---------->ExternalTable |
| | +-----------------------+ +----------+
| getColumn() | +---->|OlapTable |
| getSchema() | +-----------------------+ | +----------+
| ... +---------->Table +---+ +----------+
+-------------------+ +-----------------------+ +---->|OdbcTable |
+----------+ |
Interfaces:
Classes:
At the end of June, support access to hive, iceberg, hudi through the new framework, and the performance is better than before.
==================================================