Status

Current state: [One of "Under Discussion", "Accepted", "Rejected"]

Discussion thread: 

JIRA or Github Issue: 

Released: <Doris Version>

Google Doc: <If the design in question is unclear or needs to be discussed and reviewed, a Google Doc can be used first to facilitate comments from others.>

Motivation

Doris currently supports the external table. This feature is through the mapping of metadata, so that Doris can access external data sources. We take the iceberg data source as an example, which has two metadata mapping methods:

1. Create table directly
   In this way, the one-to-one mapping relationship between the table in the Doris metadata and the table in the Iceberg metadata is completed directly by establishing an external table.
2. Create database with external datasource property
   Set the url of iceberg metadata service(such as hive metastore) in the database property. This is equivalent to establishing a mapping relationship between database in Doris metadata and a database (or schema) in iceberg.
   After that, the Doris will automatically synchronize all tables under this database in background thread, and automatically complete the one-to-one mapping relationship of metadata of all tables.

We can see that the above two methods complete the metadata mapping at the database level and table level respectively. If the iceberg cluster has multiple databases, at least we need to manually establish multiple database-level mapping relationships. This is inconvenient to use in some scenarios.

In many scenarios, the user hopes to implement the "data source" level mapping, that is, through the "data source" level mapping relationship, Doris can directly access all data under the corresponding data source, including database and table. In this way, the user can rely on Doris and have unified access to multiple "data sources".

Therefore, the motivation of this feature is to add a new metadata level called "datasource" to Doris to support connecting and accessing to other data sources.

Related Research

Just like Presto, there is Catalog→ Database→Table hierarchical relationships to manage metadata.

In Doris, I would like to introduce a new metadata called DataSource, which is exactly same as Catalog in Presto.

1-DataSource

The current metadata hierarchy of Doris is Database->Table->Column.
We add a new level: DataSource->Database->Table→Column.

A DataSource represents a data source, such as an Iceberg cluster, a Hive cluster, an ES cluster, or another Doris cluster.

We can define data sources into the following two categories:

1. Internal DataSource
   Each Doris cluster will have an Internal DataSource by default, which manages all databases under the current cluster. When the upgrade Doris from old version, this Datasource will be added by default.
2. External Datasource
   All data sources that are not self-manager by Doris will be referred to as External Datasource. Such as hive, iceberg, hudi, es, odbc, etc. Users can create several External DataSources.
   External DataSource is a base class, we can provide different subclass implementations according to different data sources, such as:
   1. Hive Metastore: Used to access hive metastore compatible data sources.
   2. ES: Support ES data source
   3. ODBC: Supports accessing external data sources through odbc.
   4. Doris: Other Doris cluster

2-How to manager DataSource

Here we discuss about how to manager datasources, for example, how to add a new databases.

1. Built-in
   Like the various external data sources currently implemented by Doris, each external data source requires separate development and adaptation, and user need to upgrade Doris to new version to access new data source. This method is expensive, and it is difficult to solve diversity problems such as different versions of hive.
2. SPI
   Similar to Presto's connector, the interface is defined, and the user completes the specific implementation, and then accesses it through dynamic class loader. This approach can solve the problem of data source diversity. But here are a few issues to consider:
   1. What are the needs of most users: For most users, there are only a few data sources that may be concerned, such as hive, iceberg, hudi, odbc, ES. If we have built-in support for some or all of them, then there is nothing need to be done by user.
   2. We can use SPI on FE side. But on BE side, use dynamic library will have many compatibility problems. Therefore, only the metadata part can be customized by users, but the data access on BE side cannot be customized.
      Therefore, in the data access layer, we need to unify the access methods to a limited number of ways as much as possible to ensure that the built-in data access layer (BE side) can support most of the data sources, for example:
          1. HDFS
          2. S3
          3. ODBC
      For more access methods, we can consider another form, that is to deploy an Http Connector Service on the user's data source side. The specificity of data source access is masked through this HTTP Service. For Doris, it boils down to HTTP access. This method can bring more flexible scalability and reduce the intrusion to the Doris cluster. For example, it can handle network connectivity issues, authorization verification, data caching and even data filtering. Of course, this method brings extra maintenance costs to users, and also has high requirements on the availability and performance of the http connection service itself.

To sum up, we will combine the above methods to carry out the first stage design: Built-in data source + SPI interface

1. Built-in data sources meet most usage scenarios, and SPI meets diverse needs
2. Unify the interface definition of built-in data source and SPI to ensure the unity of code logic

Also, the user-side http connection service can be extended based on this interface in next stage.

Detailed Design

the detailed design of the function.

Scheduling

specific implementation steps and approximate scheduling.