...
- Give users full control over k8s pod template (no mapping/whitelisting)
- Layering/merging of pod templates (operator itself could also apply cluster wide defaults)
kind: FlinkDeployment |
Java Operator SDK
The Flink operator should be built using the java-operator-sdk . The java operator sdk is the state of the art approach for building a Kubernetes operator in Java. It uses the Fabric8 k8s client like Flink does and it is open source with Apache 2.0 license.
Compatibility, Deprecation, and Migration Plan
- What impact (if any) will there be on existing users?
- If we are changing behavior how will we phase out the older behavior?
- If we need special migration tools, describe them here.
- When will we remove the existing behavior?
As this is a completely new standalone component, no migration will be necessary strictly speaking. Compatibility is to be seen and will depend on any changes required to the Flink Kubernetes integration.
Test Plan
Describe in few sentences how the FLIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?
Rejected Alternatives
...
Using Go to implement the operator
While Go is often a natural fit for implementing k8s operators and there are already some open-source examples of Flink operators implemented in Go we still feel that Java is more suitable for this new component.
Main reasons for choosing Java over Go
- Direct access to Flink Client libraries for submitting, managing jobs and handling errors
- Most Flink developers have strong Java experience while there are only few Go experts
- Easier to integrate with existing build system and tooling
- Required k8s clients and tools for building an operator are also available in Java
References
- [1] Discussion thread: https://lists.apache.org/thread/l1dkp8v4bhlcyb4tdts99g7w4wdglfy4
- [2] Flink embedded k8s integration: https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/resource-providers/native_kubernetes/
- [3] Flink native k8s support blog: https://flink.apache.org/2021/02/10/native-k8s-with-ha.html
- [4] Flink Kubernetes HA: https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/resource-providers/native_kubernetes/#high-availability-on-kubernetes
- [5] Lyft Flink operator: https://github.com/lyft/flinkk8soperator
- [6] Google Flink operator: https://github.com/GoogleCloudPlatform/flink-on-k8s-operator and its Spotify fork: https://github.com/spotify/flink-on-k8s-operator