Versions Compared

Old Version 5

changes.mady.by.user Unknown User (morningman)

Saved on

compared with

New Version 6

changes.mady.by.user Unknown User (morningman)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
# No additional syntax will be added. If you need to authorize privilege under other datasource, you need to switch ds first.
# During initialization, you can use the admin account to switch to the new ds to create the permissions of the initial account.
grant all on db.tbl to user@%;


3-Basic data structure

Code Block
                                                                +---------------------+
                                                           +--->|HMSExternalDataSource|
                                                           |    +---------------------+
                               +-----------------------+   |    +---------------------+
                               |ExternalDataSource     |   +--->|EsExternalDataSource |
                               |                       +---+    +---------------------+
                         +----->    listDatabaseNames()|   |    +---------------------++
                         |     |    tableExists()      |   +--->|OdbcExternalDataSource|
+-------------------+    |     |    getTableSchema()   |   |    +----------------------+
| DataSourceIf      |    |     |    ...                |   |    +----------------------+
|                   +----+     +-----------------------+   +--->|HttpExternalDataSource|
|    getDatabase()  |                                           +----------------------+
|    listDatabases()+----+     +-----------------------+
|    ...            |    |     |InternalDataSource     |
+-------------------+    |     |                       |
                         +----->    self-managed meta  |
                               |                       |
                               +-----------------------+

+-------------------+          +-----------------------+
| DatabaseIf        +---------->ExternalDatabase       |
|                   |          +-----------------------+
|    getTable()     |
|    listTable()    |          +-----------------------+
|    ...            +---------->Database               |
+-------------------+          +-----------------------+


+-------------------+          +-----------------------+
| TableIf           +---------->ExternalTable          |
|                   |          +-----------------------+         +----------+
|    getColumn()    |                                      +---->|OlapTable |
|    getSchema()    |          +-----------------------+   |     +----------+
|    ...            +---------->Table                  +---+     +----------+
+-------------------+          +-----------------------+   +---->|OdbcTable |
                                                                 +----------+


Interfaces:

  • DataSourceIf: provide getDatabase(), listDatabases(), etc.
  • DatabaseIf: provide getTable(), listTables(), etc.
  • TableIf: provide getColumn(), getSchema(), etc.


Classes:

  • InternalDataSource implements DataSourceIf: contains all self-managed metadata.
  • ExternalDataSource implements DataSourceIf: contains all not-self-managed metadata which should be got from external data source.
    • HMSExternalDataSource: hive metastore compatible data sources.
    • EsExternalDataSource: elasticsearch
    • OdbcExternalDataSource: odbc compatible data sources
    • HttpExternalDataSource: for http api
    • ...


  • ExternalDatabase implements DatabaseIf: database got from external data source.
  • Database implements DatabaseIf: the self-managed databases.


  • ExternalTable implements TableIf: table got from external data source.
  • Table implements TableIf: the self-managed tables
    • OlapTable
    • MysqlTable
    • OdbcTable
    • ...


  • ExternalScanNode: for external table scan task.


  • MetaObjCache: To cache and manage all external meta objects.

Scheduling

  1. Define basic data structures, including DataSource interface, ExternalScanNode, ExternalTable, etc.
  2. POC: Access to the hive metastore data source using these basic data structure.
  3. Support dynamic loading user defined datasource using SPI.
  4. SQL parser: Support datasource qualifier.
  5. Privilege: Support privilege management for datasource level.
  6. Metadata Cache

...