Table of Contents

Status

Current state: [One of "Under Discussion", "Accepted", "Rejected"]Released

Discussion thread: 

JIRA or Github Issue: 

Released: <Doris Version>1.2.0

Google Doc: <If the design in question is unclear or needs to be discussed and reviewed, a Google Doc can be used first to facilitate comments from others.>

...

# No additional syntax will be added. If you need to authorize privilege under other datasource, you need to switch ds first.
# During initialization, you can use the admin account to switch to the new ds to create the permissions of the initial account.
grant all on db.tbl to user@%;

3-Basic data structure

Scheduling

                                                                +---------------------+
                                                           +--->|HMSExternalDataSource|
                                                           |    +---------------------+
                               +-----------------------+   |    +---------------------+
                               |ExternalDataSource     |   +--->|EsExternalDataSource |
                               |                       +---+    +---------------------+
                         +----->    listDatabaseNames()|   |    +---------------------++
                         |     |    tableExists()      |   +--->|OdbcExternalDataSource|
+-------------------+    |     |    getTableSchema()   |   |    +----------------------+
| DataSourceIf      |    |     |    ...                |   |    +----------------------+
|                   +----+     +-----------------------+   +--->|HttpExternalDataSource|
|    getDatabase()  |                                           +----------------------+
|    listDatabases()+----+     +-----------------------+
|    ...            |    |     |InternalDataSource     |
+-------------------+    |     |                       |
                         +----->    self-managed meta  |
                               |                       |
                               +-----------------------+

+-------------------+          +-----------------------+
| DatabaseIf        +---------->ExternalDatabase       |
|                   |          +-----------------------+
|    getTable()     |
|    listTable()    |          +-----------------------+
|    ...            +---------->Database               |
+-------------------+          +-----------------------+


+-------------------+          +-----------------------+
| TableIf           +---------->ExternalTable          |
|                   |          +-----------------------+         +----------+
|    getColumn()    |                                      +---->|OlapTable |
|    getSchema()    |          +-----------------------+   |     +----------+
|    ...            +---------->Table                  +---+     +----------+
+-------------------+          +-----------------------+   +---->|OdbcTable |
                                                                 +----------+

Interfaces:

• DataSourceIf: provide getDatabase(), listDatabases(), etc.
• DatabaseIf: provide getTable(), listTables(), etc.
• TableIf: provide getColumn(), getSchema(), etc.

Classes:

• InternalDataSource implements DataSourceIf: contains all self-managed metadata.
• ExternalDataSource implements DataSourceIf: contains all not-self-managed metadata which should be got from external data source.
  • HMSExternalDataSource: hive metastore compatible data sources.
  • EsExternalDataSource: elasticsearch
  • OdbcExternalDataSource: odbc compatible data sources
  • HttpExternalDataSource: for http api
  • ...

• ExternalDatabase implements DatabaseIf: database got from external data source.
• Database implements DatabaseIf: the self-managed databases.

• ExternalTable implements TableIf: table got from external data source.
• Table implements TableIf: the self-managed tables
  • OlapTable
  • MysqlTable
  • OdbcTable
  • ...

• ExternalScanNode: for external table scan task.

• MetaObjCache: To cache and manage all external meta objects.

Scheduling

1. Creation of DataSource (In review)
   1. create/drop/alter/show/switch syntax support
   2. Persistence of DataSource
   3. https://github.com/apache/incubator-doris/pull/10033
2. InternalDataSource Support (Already merged)
   1. The change of the original interface ensures that the previous use logic remains unchanged.
   2. https://github.com/apache/incubator-doris/pull/9953
   3. https://github.com/apache/incubator-doris/pull/10044
3. HMSExternalDataSource Support (Under development, 6.17)
   1. Ability to synchronize HiveMetaStore data sources
   2. Able to show databases, show tables, desc table, information schema
   3. Can be converted to the original Hive/HudiScanNode for data query
   4. https://github.com/apache/incubator-doris/pull/10028
4. FE side ExternalFileScanNode development (Under development, 6.17)
   1. Unify hive/iceberg/hudi scan node
   2. Support partition prunning and other functions
   3. https://github.com/apache/incubator-doris/pull/9973
5. BE side ExternalFileScanNode development (6.24)
   1. Replace the BrokerScanNode
   2. Only supported in the vectorized engine, with the vectorized parquet/orc reader
6. DataSource permission management (Under development, 6.17)
   1. Support DataSource-level permission management
7. SQL syntax support (6.24)
   1. SQL syntax supports three-level qualified names
   2. Support query across DataSource
8. Parquet predicate pushdown (Designing, 6.24)
   1. Support column pruning and predicate filtering for parquet files
9. Test and Bug Rush (6.30)
   1. [3-7] test of part of the work
   2. Functionally the same as before

At the end of June, support access to hive, iceberg, hudi through the new framework, and the performance is better than before.

==================================================

1. Read optimization of ORC/Text file format (ByteDance)
2. ESExternalDataSource support
3. ODBCExternalDataSource support
4. Support for appearance statistics
5. Collection and display interface of statistics
6. Statistics framework that needs to be combined with the new optimizer
7. Meta Object
8. Define basic data structures, including DataSource interface, ExternalScanNode, ExternalTable, etc.
9. POC: Access to the hive metastore data source using these basic data structure.
10. Support dynamic loading user defined datasource using SPI.
11. SQL parser: Support datasource qualifier.
12. Privilege: Support privilege management for datasource level.
13. Metadata Cache