Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
languagejava
@Public
public interface Function extends java.io.Serializable {
    /**
     * Returns false if it is guaranteed that the function will not store and access
     * reference to the output value.
     */ 
    default boolean isOutputValueStored() {
        return falsetrue;
    }
}

@Public
public interface RichFunction extends Function {

    /**
     * RichFunction is able to put the values to the state backend so the method returns true by
     * default. For RichFunction that doesn't store output value to the state backend, it can return
     * false.
     */
    @Override
    default boolean isOutputValueStored() {
        return true;
    }
}


4) Update the description of pipeline.object-reuse  to mention that when it is false, Flink will decide whether to use object reuse based on the operator attributes.

Here is the updated description:

When it is true, objects that Flink internally uses for deserialization and passing data to user-code functions will be reused. When it is false, Flink will decide whether to use object reuse based on the operator attributes. Keep in mind that this can lead to bugs when the user-code function of an operation is not aware of this behavior.

Proposed Changes

1) Update OperatorChain to take advantage of the isOutputStreamRecordValueStored and isInputStreamRecordStored attribute.

...

POC and Analysis

We implement a POC and run the flink-benchmarks against the POC with global object-reuse disabled. We verify that, with the change proposed in the FLIP, many of the operators in the benchmark can enable object-reuse without code change. Only the custom operator and the AbstractUdfStreamOperator that contains a RichFunction cannot enable object-reuse to its output. In order to enable object-reuse of those operator/function, couple lines of code are needed to set the operator attributes accordingly. After that, all the operator in flink-benchmarks can enable object-reuse with the global object-reuse disable.

...