THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Discussion thread: https://lists.apache.org/thread/9k1sl2519kh2n3yttwqc00p07xdfns3h
JIRA: Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-28779
Released: <Flink Version>
...
As discussed in the mailing thread, we're planning to introduce an unified hint to support both sync|async lookup and with|without retry, the support matrix will be:
| lookup support | async | retry |
| sync w/o retry | N | N |
| sync w/ retry | N | Y |
| async w/o retry | Y | N |
| async w/ retry | Y | Y |
Non target:
- The lookup sources that can be connected as streams (can use other types of join)
- Async lookup with retry is not capable for fixed delayed processing for all input data (should use other lighter ways to solve, e.g., pending source consumption or use sync lookup with retry)
- Do not support retry on exception (let the sql connectors handle it)
...
The available hint options:
| option type | option name | optional | value type | default value | description |
|---|---|---|---|---|---|
| table name | table | N | string | N/A | the table name of the lookup source |
async | async | Y | boolean | N/A | value can be 'true' or 'false' to suggest the planner choose the corresponding lookup function. If the backend lookup source does not support the suggested lookup mode, it will take no effect. |
| output-mode | Y | string | ordered | value can be 'ordered' or 'allow_unordered'. 'allow_unordered' means if users allow unordered result, it will attempt to use AsyncDataStream.OutputMode.UNORDERED when it does not affect the correctness of the result, otherwise ORDERED will be still used. It is consistent with `ExecutionConfigOptions#TABLE_EXEC_ASYNC_LOOKUP_OUTPUT_MODE`. | |
| capacity | Y | integer | 100 | the buffer capacity for the backend asyncWaitOperator of the lookup join operator. | |
| timeout | Y | duration | 300s | timeout from first invoke to final completion of asynchronous operation, may include multiple retries, and will be reset in case of failover | |
retry | retry-predicate | Y | string | N/A | can be 'lookup_miss' which will enable retry if lookup result is empty. |
| retry-strategy | Y | string | N/A | can be 'fixed_delay' | |
| fixed-delay | Y | duration | N/A | delay time for the 'fixed_delay' strategy | |
| max-attempts | Y | integer | N/A | max attempt number of the 'fixed_delay' strategy |
For these connectors which can have both capabilities of async and sync lookup, our advice for the connector developers are implementing both sync and async interfaces if both capabilities have suitable use cases, the planner will prefer the async one by default, and users can give different option value 'async'='true|false' via the LOOKUP query hint to suggest the planner, otherwise choose one interface to implement.
...