Versions Compared

Old Version 4

changes.mady.by.user Timo Walther

Saved on

compared with

New Version 5

changes.mady.by.user Timo Walther

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Implementation Plan extended

...

 No compatibility changes or other deprecation necessary.

Implementation Plan

1. Basic Embedded SQL Client 

Add the basic features to play around with Flink's streaming SQL.

 

  • Add CLI component that reads the configuration files
    • "Pre-registered table sources"
    • "Job parameters"
  • Add executor for retrieving pre-flight information and corresponding CLI SQL parser
    • SHOW TABLES
    • DESCRIBE TABLE
    • EXPLAIN

  • Add streaming append query submission to executor

    • Submit jars and run SELECT query using the ClusterClient

    • Collect results on heap and serve them on the CLI side

    • EXECUTE 

  • Create SQL client in embedded mode with results stored on Heap

  • CLI functionality:

    • Query Submission

    • History

    • [Optional] Highlighter

  • Included SQL operations:

  • SELECT (Batch, Append, Retract/Upsert)

  • SHOW TABLES

  • SHOW FUNCTIONS

    • EXECUTE

    • (for executing a SQL statement stored in a local file)

  • DESCRIBE TABLE

  • DESCRIBE FUNCTION

  • EXPLAIN

  • Support INSERT INTO

    • Afterwards: Discuss further features
    Such as:

2. Full Embedded SQL Client

Add important features to fully use the SQL client for a variety of use cases.

  • Add support for streaming retract/upsert queries
  • Add support for batch queries

  • Add user-defined function support

    • Support scalar, aggregate, and table functions
    • Submit corresponding jars to the cluster
    • DESCRIBE FUNCTION
  • Add CLI query history
  • Add CLI query code completion/highlighting
  • Add support for INSERT INTO
    • Read configuration about "Pre-registered table sinks"
    • Allow submission without collect()

3. Discuss/design further features

Discuss and prioritize other features that are not part of this FLIP. 

  • Introduce gateway mode with REST API

  • Add support for catalogs

  • Allow creating Materialized Views

  • Support other materialization backends

  • Create a JDBC API

  • Further SQL DDL statements:

    • CREATE TABLE

    • CREATE TYPE

  • Further CLI features:

    • Auto-completion

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.