| Discussion thread | - |
|---|
| Vote thread | - |
|---|
| JIRA | - |
|---|
| Release | - |
|---|
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
In our production, we create Flink Session Cluster and users can perform OLAP query in it. Users like to use Jdbc Driver in their applications to submit queries and get results. Currently we use the old flink-sql-gateway[1] and flink-jdbc-driver[2], but when we try to upgrade our Flink Session Cluster, We meet compatibility problem. Flink Sql Gateway has been a submodule in Flink, and the API has been greatly optimized. The old Flink JDBC Driver cannot connect to the new Gateway directly which cause we cannot upgrade our Flink version. In this FLIP, I'd like to introduce Flink Jdbc Driver module in Flink to connect Gateway, users can use Jdbc Driver to submit their queries and get results like a database in their applications.
Proposal
Currently users can use Flink Sql Client to connect Sql Gateway as followed
eyJleHRTcnZJbnRlZ1R5cGUiOiIiLCJnQ2xpZW50SWQiOiIiLCJjcmVhdG9yTmFtZSI6IkZhbmcgWW9uZyIsIm91dHB1dFR5cGUiOiJibG9jayIsImxhc3RNb2RpZmllck5hbWUiOiJGYW5nIFlvbmciLCJsYW5ndWFnZSI6ImVuIiwiZGlhZ3JhbURpc3BsYXlOYW1lIjoiIiwic0ZpbGVJZCI6IiIsImF0dElkIjoiMjQwODg0MTMwIiwiZGlhZ3JhbU5hbWUiOiJjbGllbnQtZ2F0ZXdheSIsImFzcGVjdCI6IiIsImxpbmtzIjoiYXV0byIsImNlb05hbWUiOiJGTElQLTI5MzogSW50cm9kdWNlIEZsaW5rIEpkYmMgRHJpdmVyIEZvciBTcWwgR2F0ZXdheSIsInRic3R5bGUiOiJ0b3AiLCJjYW5Db21tZW50IjpmYWxzZSwiZGlhZ3JhbVVybCI6IiIsImNzdkZpbGVVcmwiOiIiLCJib3JkZXIiOnRydWUsIm1heFNjYWxlIjoiMSIsIm93bmluZ1BhZ2VJZCI6MjQwODg0MTI3LCJlZGl0YWJsZSI6ZmFsc2UsImNlb0lkIjoyNDA4ODQ0NTgsInBhZ2VJZCI6IiIsImxib3giOnRydWUsInNlcnZlckNvbmZpZyI6eyJlbWFpbHByZXZpZXciOiIxIn0sIm9kcml2ZUlkIjoiIiwicmV2aXNpb24iOjEsIm1hY3JvSWQiOiIyODg0OWU3YS00ODA2LTRjZmEtOTcwNy05Y2JiMTc2MWIxNTMiLCJwcmV2aWV3TmFtZSI6ImNsaWVudC1nYXRld2F5LnBuZyIsImxpY2Vuc2VTdGF0dXMiOiJPSyIsInNlcnZpY2UiOiIiLCJpc1RlbXBsYXRlIjoiIiwid2lkdGgiOiIyODEiLCJzaW1wbGVWaWV3ZXIiOmZhbHNlLCJsYXN0TW9kaWZpZWQiOjE2NzY2MDM1NDMwMDAsImV4Y2VlZFBhZ2VXaWR0aCI6ZmFsc2UsIm9DbGllbnRJZCI6IiJ9
SqlGateway starts Rest Service, and SqlClient can connect to it. Executor creates RestClient according to the address of Rest Service, then it can create Session, Submit Sql and Fetch Result according to RestClient.
We propose Flink Jdbc Driver interacts with SqlGateway through Executor too.
eyJleHRTcnZJbnRlZ1R5cGUiOiIiLCJnQ2xpZW50SWQiOiIiLCJjcmVhdG9yTmFtZSI6IkZhbmcgWW9uZyIsIm91dHB1dFR5cGUiOiJibG9jayIsImxhc3RNb2RpZmllck5hbWUiOiJGYW5nIFlvbmciLCJsYW5ndWFnZSI6ImVuIiwiZGlhZ3JhbURpc3BsYXlOYW1lIjoiIiwic0ZpbGVJZCI6IiIsImF0dElkIjoiMjQwODg0MTMyIiwiZGlhZ3JhbU5hbWUiOiJkcml2ZXItZ2F0ZXdheSIsImFzcGVjdCI6IiIsImxpbmtzIjoiYXV0byIsImNlb05hbWUiOiJGTElQLTI5MzogSW50cm9kdWNlIEZsaW5rIEpkYmMgRHJpdmVyIEZvciBTcWwgR2F0ZXdheSIsInRic3R5bGUiOiJ0b3AiLCJjYW5Db21tZW50IjpmYWxzZSwiZGlhZ3JhbVVybCI6IiIsImNzdkZpbGVVcmwiOiIiLCJib3JkZXIiOnRydWUsIm1heFNjYWxlIjoiMSIsIm93bmluZ1BhZ2VJZCI6MjQwODg0MTI3LCJlZGl0YWJsZSI6ZmFsc2UsImNlb0lkIjoyNDA4ODQ0NTgsInBhZ2VJZCI6IiIsImxib3giOnRydWUsInNlcnZlckNvbmZpZyI6eyJlbWFpbHByZXZpZXciOiIxIn0sIm9kcml2ZUlkIjoiIiwicmV2aXNpb24iOjEsIm1hY3JvSWQiOiJiOTQyOGY1MS01ZGIxLTQ3ZjUtYjNlNS0xMTE2NjA2MDgxNTciLCJwcmV2aWV3TmFtZSI6ImRyaXZlci1nYXRld2F5LnBuZyIsImxpY2Vuc2VTdGF0dXMiOiJPSyIsInNlcnZpY2UiOiIiLCJpc1RlbXBsYXRlIjoiIiwid2lkdGgiOiIyODEiLCJzaW1wbGVWaWV3ZXIiOmZhbHNlLCJsYXN0TW9kaWZpZWQiOjE2NzY2MDM1ODUwMDAsImV4Y2VlZFBhZ2VXaWR0aCI6ZmFsc2UsIm9DbGllbnRJZCI6IiJ9
There are four main classes in Flink Jdbc Driver: FlinkDriver, FlinkConnection, FlinkStatement and FlinkResultSet which implement jdbc interface Driver, Connection, Statement and Result.
- FlinkDriver parses gateway address from url, and creates FlinkConnection
- FlinkConnection creates Executor according to gateway address. When the Connections is closed, it can close the connection with gateway by Executor
- FlinkStatement can get Executor from FlinkConnection, and submit sql query to it. After query is executed, FlinkStatement can get StatementResult from Executor, and create FlinkResultSet
- FlinkResultSet is an iterator, it gets results from StatementResult and return them to users
The call relationship between them are as followed.
eyJleHRTcnZJbnRlZ1R5cGUiOiIiLCJnQ2xpZW50SWQiOiIiLCJjcmVhdG9yTmFtZSI6IkZhbmcgWW9uZyIsIm91dHB1dFR5cGUiOiJibG9jayIsImxhc3RNb2RpZmllck5hbWUiOiJGYW5nIFlvbmciLCJsYW5ndWFnZSI6ImVuIiwiZGlhZ3JhbURpc3BsYXlOYW1lIjoiIiwic0ZpbGVJZCI6IiIsImF0dElkIjoiMjQwODg0MTM0IiwiZGlhZ3JhbU5hbWUiOiJjbGFzc2VzLWRyaXZlciIsImFzcGVjdCI6IiIsImxpbmtzIjoiYXV0byIsImNlb05hbWUiOiJGTElQLTI5MzogSW50cm9kdWNlIEZsaW5rIEpkYmMgRHJpdmVyIEZvciBTcWwgR2F0ZXdheSIsInRic3R5bGUiOiJ0b3AiLCJjYW5Db21tZW50IjpmYWxzZSwiZGlhZ3JhbVVybCI6IiIsImNzdkZpbGVVcmwiOiIiLCJib3JkZXIiOnRydWUsIm1heFNjYWxlIjoiMSIsIm93bmluZ1BhZ2VJZCI6MjQwODg0MTI3LCJlZGl0YWJsZSI6ZmFsc2UsImNlb0lkIjoyNDA4ODQ0NTgsInBhZ2VJZCI6IiIsImxib3giOnRydWUsInNlcnZlckNvbmZpZyI6eyJlbWFpbHByZXZpZXciOiIxIn0sIm9kcml2ZUlkIjoiIiwicmV2aXNpb24iOjEsIm1hY3JvSWQiOiIxYThkNzlhZS0wZjIwLTQ1Y2EtOWM5Ni0xYTZmMWRjNWQzYTMiLCJwcmV2aWV3TmFtZSI6ImNsYXNzZXMtZHJpdmVyLnBuZyIsImxpY2Vuc2VTdGF0dXMiOiJPSyIsInNlcnZpY2UiOiIiLCJpc1RlbXBsYXRlIjoiIiwid2lkdGgiOiI0ODEiLCJzaW1wbGVWaWV3ZXIiOmZhbHNlLCJsYXN0TW9kaWZpZWQiOjE2Nzc3NTA5MzkwMDAsImV4Y2VlZFBhZ2VXaWR0aCI6ZmFsc2UsIm9DbGllbnRJZCI6IiJ9
[1] https://github.com/ververica/flink-sql-gateway
[2] https://github.com/ververica/flink-jdbc-driver