Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

State

[progress record] :

Proposed time : 2022/03/01

...

4. Link tests of data sources are completed through linkis metastore server service, which now provides corresponding metadata query service

Changes


Modification Detail
1
Modification of maven module

  • New module linkis-datasource-query-common is added, and new datasource data structure, exception class, and tool class are added
  • A new module, linkis-datasource-quwey-server, is added to manage data sources. It provides functions such as adding, deleting, checking, modifying, and testing data sources through restful interfaces
  • Added the linkis-metadata-manager-common module, and added the metadata data structure, exception class, and tool class
  • The linkis-metadata-manager-server module is added to provide metadata management services and query metadata databases, tables, and columns through restful interfaces
  • Added a new linkis-metadata-manager-service-es module to provide the elasticsearch metadata management service
  • The linkis-metadata-manager-service-hive module is added to provide the metadata query service for hive
  • Add a new module linkis-metadata-manager-service-kafka to provide metadata query service for kafka
  • A new module linkis-metadata-manager-service-mysql is added to provide metadata query services for mysql
  • A new datasource management Java client module linkis-datasource-client is added to facilitate datasource management using sdk
2Modification of HTTP interface
  • Added the interface for querying metadata d
  • New data source add delete change search function
3Modification of the client interface

LinkisDataSourceRemoteClient interface

  • GetAllDataSourceTypesResult getAllDataSourceTypes (GetAllDataSourceTypesAction) query all data types
  • QueryDataSourceEnvResult queryDataSourceEnv(QueryDataSourceEnvAction) Queries the cluster configurations that can be used by the data source
  • GetInfoByDataSourceIdResult getInfoByDataSourceId (GetInfoByDataSourceIdAction) : through the data source id query data source information
  • QueryDataSourceResult QueryDataSourceAction (QueryDataSourceAction) Queries data sources
  • GetConnectParamsByDataSourceIdResult getConnectParams (GetConnectParamsByDataSourceIdAction) get connection configuration parameters
  • CreateDataSourceResult createDataSource(CreateDataSourceAction) Creates a data source
  • DataSourceTestConnectResult getDataSourceTestConnect (DataSourceTestConnectAction) to test whether or not the data source connection is established properly
  • DeleteDataSourceResult deleteDataSource(DeleteDataSourceAction) Deletes a data source
  • ExpireDataSourceResult expireDataSource(ExpireDataSourceAction) Sets the data source to the expired state
  • GetDataSourceVersionsResult getDataSourceVersions (GetDataSourceVersionsAction) query list data source configuration version
  • PublishDataSourceVersionResult publishDataSourceVersion (PublishDataSourceVersionAction) released data source configuration version
  • UpdateDataSourceResult UpdateDataSourceAction (UpdateDataSourceAction) Updates data sources
  • UpdateDataSourceParameterResult updateDataSourceParameter (UpdateDataSourceParameterAction) to update the data source configuration parameters
  • GetKeyTypeDatasourceResult getKeyDefinitionsByType (GetKeyTypeDatasourceAction) data source type of a query need configuration properties

LinkisMetaDataRemoteClient interface

  • MetadataGetDatabasesResult getDatabases (MetadataGetDatabasesAction) query the database list
  • MetadataGetTablesResult getTables(MetadataGetTablesAction) Queries table data
  • MetadataGetTablePropsResult getTableProps (MetadataGetTablePropsAction)
  • MetadataGetPartitionsResult getPartitions (MetadataGetPartitionsAction) query partition table
  • MetadataGetColumnsResult getColumns(MetadataGetColumnsAction) Queries the columns of the data table
4Modification of database table structure
  • No table to modify
  • The new table structure is as follows:

Image Modified

5Modification of configuration item
6Modification Error code 
7Modifications for Third Party Dependencies

Compatibility, Deprecation, and Migration Plan

...