THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Status
Current state: "Under Discussion"
...
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
With efforts in FLIP-24 - SQL Client and FLIP-91: Support SQL Client Gateway, Flink SQL client supports submitting queries but lacks further support for their lifecycle afterwards, which is crucial for streaming use cases. That means Flink SQL client users have to turn to other clients (e.g. CLI) or APIs (e.g. REST API) to manage the queries, like triggering savepoints or canceling queries, which makes the user experience of SQL client incomplete.
Therefore, this proposal aims to complete the capability of SQL client by adding query lifecycle statements. With these statements, users could manage queries and savepoints through pure SQL in SQL client.
Public Interfaces
- New Flink SQL Statements
Proposed Changes
Architecture Overview
The overall architecture of Flink SQL client would be as follow:
...
Most parts are remained unchanged, only SQL Parser and Planner need to be modified to support new statements, and a new component ClusterClientFactory is introduced in Executor to enable direct access to Flink clusters.
Query Lifecycle Statements
Query lifecycle statements mainly interact with deployments (clusters and jobs) and have few connections with Table/SQL concepts, thus it’d be better to keep them SQL-client-only like jar statements.
SHOW QUERIES
SHOW QUERIES statements list the queries in the Flink cluster, which is similar to flink list in CLI.
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
+----------------------------------+-------------+----------| | query_id | query_name | status | +----------------------------------+-------------+----------| | cca7bc1061d61cf15238e92312c2fc20 | query1 | RUNNING | | 0f6413c33757fbe0277897dd94485f04 | query2 | FAILED | +----------------------------------+-------------+----------| |
STOP QUERY
STOP QUERY statements stop a non-terminated query, which is similar to `flink stop` in CLI. As stop command has a `--drain` option, we should introduce a table config like `sql-client.stop-with-drain` to support the same functionality.
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
+--------------------------------------------------------| | savepoint_path | +--------------------------------------------------------| | /tmp/flink-savepoints/savepoint-cca7bc-bb1e257f0dab | +--------------------------------------------------------| |
CANCEL QUERY
CANCEL QUERY statements cancel a non-terminated query, which is similar to `flink cancel` in CLI.
...
Since CANCEL QUERY doesn’t trigger a savepoint, the result would be a simple OK, like the one returned by DDL.
TRIGGER SAVEPOINT
TRIGGER SAVEPOINT statements trigger savepoints for the specified query, which is similar to `flink savepoint` in CLI.
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
+------------------------------------------------------| | savepoint_path | +------------------------------------------------------| | /tmp/flink-savepoints/savepoint-cca7bc-bb1e257f0dab | +------------------------------------------------------| |
DISPOSE SAVEPOINT
DISPOSE SAVEPOINT statements delete the specified savepoint, which is similar to `flink savepoint –dispose` in CLI.
...
The result would be a simple OK.
SQL Parser & Planner
To support the new statements, we need to introduce new SQL operators for SQL parser and new SQL operations for the planner.
SQL operator | SQL operation |
SqlShowQueries | ShowQueriesOperation |
SqlStopQuery | StopQueryOperation |
SqlCancelQuery | CancelQueryOperation |
SqlTriggerSavepoint | TriggerSavepointOperation |
SqlDisposeSavepoint | DisposeSavepointOperation |
Executor
Executor would need to convert the query lifecycle operations into ClusterClient commands.
SQL operation | Cluster Client Command |
ShowQueriesOperation | ClusterClient#listJobs |
StopQueryOperation | ClusterClient#stoplWithSavepoint |
CancelQueryOperation | ClusterClient#cancel |
TriggerSavepointOperation | ClusterClient#triggerSavepoint |
DisposeSavepointOperation | ClusterClient#disposeSavepoint |
Implementation Plan
The implementation plan would be simple:
- Support the new statements and operations in SQL parser and Planner.
- Extend Executor to support the new operations.
Rejected Alternatives
An alternatives approach to query monitoring is that the SQL client or gateway book keeps every query and is responsible for updating the query status through polling or callbacks. In that way, the query status is better maintained, and we wouldn’t lose track of the queries in cases that they’re cleaned up by the cluster or the cluster is unavailable.
...