...

Kubernetes (k8s) has become the predominant deployment platform for Flink. Over the past several years, a number of integrations have been developed that aim to help operationalize mission-critical Flink applications on k8s. Given the increasing adoption of k8s for Flink deployments and as discussed in [1], there is interest to provide a k8s native solution as part of Flink that can benefit from the rich experience of community members and ultimately make Flink easier to adopt.

Currently Flink comes with built-in embedded k8s support, also referred to as Flink native k8s [2, 3, 4], offering an alternative to the traditional standalone deployment mode. Independently a number of k8s operators have been developed outside of Apache Flink, including [5, 6]. These implementations are not under the umbrella of a neutral entity like the ASF and as a result, tend to lack wider community participation and projects go stale after the maintainers shift focus.

However, the custom resource and operator concepts are central to a Kubernetes native deployment experience. It allows to manage Flink applications and their lifecycle through k8s tooling like kubectl. A key feature of an operator is the automation of application upgrades, which cannot be achieved through the "Flink native" integration alone.

Public Interfaces

N/AThe public interface is essentially the custom resource descriptor (CRD), see below.

Proposed Changes

We are proposing to provide an Flink k8s operator implementation as part of Flink that is maintained by the community and closely integrated with the Flink ecosystem. This implementation will benefit from the extensive experience of Flink community members with large scale mission critical Flink deployments and learnings from existing operator implementations. As part of Flink, the operator will have a better chance to follow the development of Flink core, influence changes to Flink core and benefit from the established collaboration processes of the project. We are proposing a dedicated github repository for the operator with operator versioning and releases separate from core Flink (like flink-statefun or flink-shaded). 

...