Page properties | |||||
---|---|---|---|---|---|
|
Table of Contents |
---|
Status
Current state: "Under Discussion"
Discussion thread:
...
|
...
|
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Table of Contents |
---|
Motivation
The integration with docker in Flink is currently addressed in many places which often intersect, repeat each other or apply different approaches. This makes it really hard to follow the whole topic for users and maintainers. This FLIP suggests how to unify this topic. It means having one place which has the Dockerfile, all necessary scripts and docs following each other in a consistent way without any repetitions or contradictions.
...
- flink-container/kubernetes relies on flink-container/docker
- Kubernetes standalone job doc example relies on flink-container/docker
- Kubernetes standalone session doc example relies on apache/flink-docker image
- Native kubernetes integration relies on apache/flink-docker image
- Flink helm example is based on Kubernetes standalone session doc example and relies on apache/flink-docker image
- e2e tests rely on build scripts from flink-contrib/docker-flink and flink-container/docker in common_docker.sh
There are also custom Dockerfiles for tests and development purposes. This FLIP keeps them out of scope of this effort.
...
The idea is to keep all docker related resources in apache/flink-docker. It already has a detailed Dockerfile which is well suited for common use cases or at least serves as a good starting point. The suggestion is to make it extensible for other concerns which are currently addressed in other places. This mainly means refactoring of the existing code and introducing more docs as a first step. This effort should enable further improvements and follow-ups for the docker integration with Flink.
This would subsequently mean to adjust all other places to rely on or refer to apache/flink-docker code or docs. Eventually, all other purely docker related places can be completely removed: flink-contrib/docker-flink, flink-container/docker and docs/ops/deployment/docker.md.
...
Implementation steps
- Document the the official docker hub image and examples of how to run it (as of now)
- Document examples of how to extend the official extend the official docker hub image (as of now)
- Remove flink-contrib/docker-flink
- Extend entry point script and docs with job cluster mode and user job artefacts, document it
- Remove flink-container/docker
Tentative improvements:
- Modify the log4j-console.properties to also output logs into the files for WebUI
- Make custom logging properties possibleconfigurable
- Split stdout/stderr file container logsRemove the existing docker/compose/swarm/bluemix scripts in flink-contrib/docker-flink and flink-container/docker
Test Plan
Firstly, mostly manual testing. Later we can think of more extensive docker CI tests.
Future road map
We can still do move improvements to the user experience with docker in Flink:
- Investigate how to support developers to build a custom image for a snapshot version, e.g. for a certain commit in Flink repo
- Rewrite Flink options in flink-conf.yaml by environment variables when Flink process starts
- Refactor Flink bash scripts into one thin script which uses a Java bootstrap utility to prepare and configure started Flink process (similar to BashJavaUtils for memory setup)
For more details see also this FLIP discussion thread and more detailed proposal doc.
Rejected Alternatives
None so far.