Status
Current state: Under Discussion
Discussion thread:
https://lists.apache.org/thread/4w36oof8dh28b9f593sgtk21o8qh8qx4
https://lists.apache.org/thread/t0bdkx1161rlbnsf06x0kswb05mch164
Vote thread: https://lists.apache.org/thread/0bh530j5ob11lzj48vpm883sqwgmstp8
JIRA: JIRA: here (<- link to https://issues.apache.org/jira/browse/FLINK-29844XXXX)
Released: 1.17
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
Add an extra method in TypeSerializerSnapshot.java and provide a default implementation before removing the deprecated TypeSerializerSnapshot#resolveSchemaCompatibility(TypeSerializer<T> newSerializer).
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@PublicEvolving public interface TypeSerializerSnapshot<T> { // Check whether the serializer is compatible with the old one. default TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(TypeSerializerSnapshot<T> oldSerializerSnapshot) { // Use the old logic to resolve. return oldSerializerSnapshot.resolveSchemaCompatibility(restoreSerializer()); } Return INCOMPATIBLE as default before removing the deprecated method. return TypeSerializerSchemaCompatibility.incompatible(); } } |
Mark the old method as deprecated and provide a default implementation.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@PublicEvolving public interface TypeSerializerSnapshot<T> { @Deprecated default TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(TypeSerializer<T> newSerializer) { // Use Returnnew INCOMPATIBLEmethod asto defaultresolve before removing thethis deprecated method. return TypeSerializerSchemaCompatibility.incompatible( return newSerializer.snapshotConfiguration().resolveSerializerSchemaCompatibility(this); } } |
Make the method abstract after removing the deprecated TypeSerializerSnapshot#resolveSchemaCompatibility(TypeSerializer<T> newSerializer).
...
Because TypeSerializerSnapshot is a public interface, we have several steps to migrate the logic.
Step 1
...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@PublicEvolving public interface TypeSerializerSnapshot<T> { // Check whether the serializer is compatible with the old one. default TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(TypeSerializerSnapshot<T> oldSerializerSnapshot) { // Use the old logic to resolve. Return INCOMPATIBLE as default before removing the deprecated method. return TypeSerializerSchemaCompatibility.incompatible(); } @Deprecated default TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(TypeSerializer<T> newSerializer) { // Use new method to resolve before removing this deprecated method. return oldSerializerSnapshotnewSerializer.resolveSchemaCompatibilitysnapshotConfiguration(restoreSerializer).resolveSerializerSchemaCompatibility()this); } } |
- Add an extra method (TypeserializeSnapshotr#resolveSchemaCompatibility(TypeSerializerSnapshot<T> oldSerializerSnapshot)
...
- ) in TypeSerializerSnapshot.java as above, and return INCOMPATIBLE as default.
- Mark the original method as deprecated and it will use new method to resolve as default.
- Implement the new method for all built-in TypeserializerSnapshots.
For the new customized serializer, users
All places where use TypeSerializerSnapshot#resolveSchemaCompatibility(TypeSerializer<T> newSerializer) to check the compatibility will call Typeserializer#resolveSchemaCompatibility(TypeSerializerSnapshot<T> oldSerializerSnapshot).
Users could implement their own logic of resolving compatibility in their new serializer and it will be used by flink.
For the old customized serializer, if If users haven't implement TypeSerializerSnapshot#resolveSchemaCompatibility(TypeSerializerSnapshot<T> oldSerializerSnapshot), all behaviors are still same as before.
Step 2
It will confuse users there are two similar methods (TypeSerializerSnapshot#resolveSchemaCompatibility(TypeSerializer<T> newSerializer) and Typeserializer#resolveSchemaCompatibility(TypeSerializerSnapshot<T> oldSerializerSnapshot)).
So we propose deprecating TypeSerializerSnapshot#resolveSchemaCompatibility(TypeSerializer<T> newSerializer) for the long run.
if they implement it and remove their implementation of old method, the new logic will be used by flink as well.
Step 2
After several stable versionIn the step 2, we need to:
- Remove
- Implement the new method in all inner TypeserializerSnapshots
- Mark TypeSerializerSnapshot#resolveSchemaCompatibility(TypeSerializer<T> newSerializer) as deprecated and provide a default implementation for it as below.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@PublicEvolving
public interface TypeSerializerSnapshot<T> {
@Deprecated
default TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(TypeSerializer<T> newSerializer) {
// Return INCOMPATIBLE as default before removing the deprecated method.
return TypeSerializerSchemaCompatibility.incompatible();
}
} |
Step 3
...
- related implementation.
- Make all places where use TypeSerializerSnapshot#resolveSchemaCompatibility(TypeSerializer<T> newSerializer)
...
- to check the compatibility call Typeserializer#resolveSchemaCompatibility(TypeSerializerSnapshot<T> oldSerializerSnapshot).
- Remove the default implementation of the new method.
...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@PublicEvolving
public interface TypeSerializerSnapshot<T> {
// Check whether the serializer is compatible with the old one.
public abstract TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(TypeSerializerSnapshot<T> oldSerializerSnapshot);
} |
...
- Existing UT of resolving compatibility could verify whether the old behavior could work as before.
- Add extra UT to verify the correctness of customized serializer.
- Add ITCase and e2e test case to mock the user behavior.
Rejected Alternatives
Introduce a new method in TypeSerializer
Compred to introducing a new method in TypeSerializer, we could get benefits if we introduce it in TypeSerializerSnapshot:
- TypeSerializerSnapshot still owns the responsibility of resolving schema compatibility, TypeSerializer could just pay attention to its serialization and deserialization as before.
- It's very convenient to implement it based on current implementation by all information in TypeSerializerSnapshot and tools which is also helpful for users to migrate their external serializer.
Introduce a method without default implementation
Why not:
We should make sure that user jobs still work without modifying any codes before removing the deprecated method.
So we need to have two steps to complete the migration and make sure that the first version will not break changeNone.