Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Status

Current state Under Discussion

Discussion threadhttp://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Discussion-flink-elasticsearch-connector-supports-td42082.html#a42106

...

  • We split es type(it is index in high version, because every index just has one default type, and default type is docs) into different ElasticSearchInputSplit. One split corresponds to one shard.
  • The split doesn't contain slice info, which you can see below 
    • You can read here https://www.jianshu.com/p/d32e17dab90c, which is
      chinese.But you can konw that slice api has poor performance in es-hadoop
      project .
    • And i found that es-hadoop(https://github.com/elastic/elasticsearch-hadoop) has removed this and disable sliced scrolls by
      default. you can see below, which i found in the lastest es-hadoop release
      version
    • Configuration Changes
      `es.input.use.sliced.partitions` is deprecated in 6.5.0, and will be removed
      in 7.0.0. The default value for `es.input.max.docs.per.partition` (100000)
      will also be removed in 7.0.0, thus disabling sliced scrolls by default, and
      switching them to be an explicitly opt-in feature.

...