Status

Current state: "Under Discussion"

Discussion thread: 

JIRA: here (<- link to )

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

@PublicEvolving
public interface SupportsDynamicFiltering {

  /**
   * applyReturns the candidate filter fields intothis thepartition table source, andsupported. return the accepted fields. The
 * data corresponding the filter fields will be provided in runtime, which can be used to filter
 * the partitions and the input data.
 */
 List<String>This method can tell the
   * planner which fields can be used as dynamic filtering fields, the planner will pick some
   * fields from the returned fields based on the query, and create dynamic filtering operator.
   */
  List<String> listAcceptedFilterFields();

  /**
   * Applies the candidate filter fields into the table source. The data corresponding the filter
   * fields will be provided in runtime, which can be used to filter the partitions or the input
   *  data.
   *
   * <p>NOTE: the candidate filter fields are always from the result of {@link
   * #listAcceptedFilterFields()}.
   */
  void applyDynamicFiltering(List<String> candidateFilterFields);
}

...

    @Override
    public void handleSourceEvent(int subtaskId, int attemptNumber, SourceEvent sourceEventSourceEvent sourceEvent) {
        if (sourceEvent instanceof DynamicFilteringEvent) {
         if   LOG.info(sourceEvent"Received instanceof DynamicFilteringEvent): {}", subtaskId);
            createSplitAssigner(((DynamicFilteringEvent) sourceEvent).getData());
        }
    }

    private void createSplitAssigner(@Nullable DynamicFilteringData dynamicFilteringData) {
        DynamicFileEnumerator fileEnumerator = fileEnumeratorFactory.create();
        if (dynamicFilteringData != null) {
            fileEnumerator.setDynamicFilteringData(dynamicFilteringData);
        }
        // create splits
        splitAssigner = splitAssignerFactory.create(splits);
      }
}

• DynamicFileEnumerator will do the partition filter of all partitions based on DynamicFilteringData, the pseudocode for HiveSourceDynamicFileEnumerator looks like:

...