Status

...

...

JIRA:

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

This FLIP introduces new public configuration option `taskmanager.memory.managed.consumer-weights`. See Configure Weights for Use Cases for details.

In addition, this FLIP contains changes to existing public involving configuration options for python use cases. See Compatibility, Deprecation, and Migration Plan for details.

Proposed Changes

...

• `taskmanager.memory.managed.consumer-weights`: A map of managed memory use cases and their integer weights. ATM, valid use cases are:
  • BATCH_OP/ROCKSDB : Name to be decided. DATAPROC : This is a combined configuration option for RocksDB and batch operators, which are guaranteed not mixed.
  • PYTHON

...

If not explicitly specified, the following weights will be used by default.

The default weights should result in the same behavior as before in existing scenarios (without python operators). All managed memory should be used for RocksDB state backend for streaming jobs, since it’s the only occurring use case with non zero weight. Same for batch operators in batch jobs.

...