...

• MetaObjCache: To cache and manage all external meta objects.

Scheduling

1. Creation of DataSource (In review)
   1. create/drop/alter/show/switch syntax support
   2. Persistence of DataSource
   3. https://github.com/apache/incubator-doris/pull/10033
2. InternalDataSource Support (Already merged)
   1. The change of the original interface ensures that the previous use logic remains unchanged.
   2. https://github.com/apache/incubator-doris/pull/9953
   3. https://github.com/apache/incubator-doris/pull/10044
3. HMSExternalDataSource Support (Under development, 6.17)
   1. Ability to synchronize HiveMetaStore data sources
   2. Able to show databases, show tables, desc table, information schema
   3. Can be converted to the original Hive/HudiScanNode for data query
   4. https://github.com/apache/incubator-doris/pull/10028
4. FE side ExternalFileScanNode development (Under development, 6.17)
   1. Unify hive/iceberg/hudi scan node
   2. Support partition prunning and other functions
   3. https://github.com/apache/incubator-doris/pull/9973
5. BE side ExternalFileScanNode development (6.24)
   1. Replace the BrokerScanNode
   2. Only supported in the vectorized engine, with the vectorized parquet/orc reader
6. DataSource permission management (Under development, 6.17)
   1. Support DataSource-level permission management
7. SQL syntax support (6.24)
   1. SQL syntax supports three-level qualified names
   2. Support query across DataSource
8. Parquet predicate pushdown (Designing, 6.24)
   1. Support column pruning and predicate filtering for parquet files
9. Test and Bug Rush (6.30)
   1. [3-7] test of part of the work
   2. Functionally the same as before

At the end of June, support access to hive, iceberg, hudi through the new framework, and the performance is better than before.

==================================================

1. Read optimization of ORC/Text file format
2. ESExternalDataSource support
3. ODBCExternalDataSource support
4. Support for appearance statistics
5. Collection and display interface of statistics
6. Statistics framework that needs to be combined with the new optimizer
7. Define basic data structures, including DataSource interface, ExternalScanNode, ExternalTable, etc.
8. POC: Access to the hive metastore data source using these basic data structure.
9. Support dynamic loading user defined datasource using SPI.
10. SQL parser: Support datasource qualifier.
11. Privilege: Support privilege management for datasource level.
12. Metadata Cache