THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
To ensure that Flink SQL is semantic consistent in streaming mode and batch mode, combining the current situation of Flink and the needs of our business, choosing LEVEL-1 as the default behavior for Flink streaming and batch mode. If the user requires LEVEL-2 atomicity, this ability can be achieved by enabling the table.ctas-or-rtas.atomicity-enabled option. In general, batch mode usually requires LEVEL-2 atomicity. In a nutshell, Flink provides two level atomicity guarantee, LEVEL-1 as the default behavior.
...
Due to Create Table As Select syntax doesn't guarantee atomicity default, to support atomic semantic, Catalog should support serialization so as it can be serialized as a part of JobGraph and pass to JM side. Hence, here proposing a new interface SupportsAtomicCatalog AtomicCatalog which extands java Serializable interface. If you need atomic Create Table As Select, your catalog should implement this interface. By the way, this interface doesn't provide any specific method currently, but in the future we may introduce some new methods to support isolation of CTAS and RTAS through this API.
...
TableConfigOptions
Add table.ctas-or-rtas.atomicity-enabled option to allow users to enable atomicity when using create table as select syntax.
...
- InMemoryCatalog: Due to the CatalogDatabase and CatalogBaseTable etc can't be serialized by java serialization mechanism directly, so the InMemoryCatalog doesn't support to serialize which means it can not support atomic semantic.
- JdbcCatalog: The required member variables to construct Catalog object are directly serializable, such as username, password, base url, etc. The JdbcDialectTypeMapper interface need extends the serializable, so this Catalog can implement the SupportsAtomicCatalog AtomicCatalog interface.
- HiveCatalog: All member variables can be serialized directly, except for the HiveConf object, which cannot be serialized directly. We can refer to JobConfWrapper to solve the serialization problem of HiveConf, so this catalog also can implement the SupportsAtomicCatalog AtomicCatalog interface.
User-defined Catalog:
- User-defined catalog that require support for CREATE TABLE AS SELECT (CTAS) syntax should implement SupportsAtomicCatalog AtomicCatalog interface to support atomic semantic.
...