Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The changes can affect entry point of the existing docker hub image. Additionally, custom Dockerfile entry point and custom user Dockerfile, which extends official image, can use new documented bash utilities to perform standard actions.

Proposed Changes

The idea is to keep all docker related resources in apache/flink-docker. It already has a detailed Dockerfile which is well suited for common use cases or at least serves as a good starting point. The suggestion is to make it extensible for other concerns which are currently addressed in other places.

...

          RUN flink_docker_utils install_flink --flink-version  0.10.0 --scala-version 2.11
    • install python version 2 or 3
          RUN flink_docker_utils install_python 2
    • install hadoop shaded (version)
          RUN flink_docker_utils install_shaded_hadoop (2.8.3_10.0)
    • possibly more shortcuts to install opt dependencies to lib, like
          RUN flink_docker_utils install_queryable_state
  • for entry point:
    • start JM (job/session) or TM

...

          flink_docker_utils configure "option.name" “value”

...

          flink_docker_utils set_rpc_address “address”
    • set various standard RPC/WebUI ports
          flink_docker_utils set_web_ui_port 8081

Entry point

The default entry point script can accept a command to start JM (job/session) or TM. Additionally, users can pass other arguments to customise starting the process , like setting ports etcby e.g. setting environment vars. This would already allow users to run the default docker image in various modes without creating a custom image, like:

docker run flink session_jobmanager --webui_port 8081env ENABLE_BUILT_IN_PLUGINS=true --env-file ./env.list

User docs

User docs can be extended in markdown files of apache/flink-docker. The docs should explain:

  • the mainstream docker hub image
  • how to run it in various modes 
    • session JM
    • single job JM (plus packing job artifacts)
    • TM
    • other options
    • environment variables
      • FLINK_PROPERTIES
      • ENABLE_BUILT_IN_PLUGINS
      • Custom jar paths
      • Custom logging conf
  • how to extend it to build a custom image
    • install python/hadoop
    • install optional /lib dependencies from /opt or externally
    • install /plugins
    • add user job jar for single job mode
    how to use flink_docker_utils to create a custom entry point
  • add docs with examples of running compose/swarm
    • give script examples (mention job/session)

...