The Flink documentation is build nightly via Github Actions.

The Flink-StateFun/-ML documentation is build nightly in the Apache buildbot (https://ci2.apache.org/) environment.

The content is pushed to nightlies.apache.org/flink.

Build Environment

Github Actions

The following files configure the workflow for building the docs. Check it out for adjusting the build configuration.

Apache Buildbot

The following files configure buildbot for checking out branches and building the docs. Check it out for adjusting the build configuration.

Branch configuration

The configuration files above contain a flink*_branches_nightly variable that configures the branches to build nightly:

# format: (branch, alias) or branch
flink_branches_nightly = [
 ('master', 'release-1.6'),
 ('release-1.5', 'stable'),
 'release-1.4',
 'release-1.3',
 'release-1.2',
 'release-1.1',
 'release-1.0'
]

or, for Stateful Functions:

# format: (branch, alias) or branch
flink_statefun_branches_nightly = [
 ('master', 'release-2.1'),
 ('release-2.0', 'stable')
]

or, for Flink ML:

# format: (branch, alias) or branch
flink_ml_branches_nightly = [
 ('master', 'release-2.1'),
 ('release-2.0', 'stable')
]

The Flink docs will be served under the following URLs:

  • https://nightlies.apache.org/flink/flink-docs-BRANCH (e.g. flink-docs-master)
  • https://nightlies.apache.org/flink/flink-docs-ALIAS (e.g. flink-docs-release-1.6)

The StateFun docs will be served under the following URLs:

  • https://nightlies.apache.org/flink/flink-statefun-docs-BRANCH (e.g. flink-statefun-docs-master)
  • https://nightlies.apache.org/flink/flink-statefun-docs-ALIAS (e.g. flink-statefun-docs-stable)

The Flink ML docs will be served under the following URLs:

  • https://nightlies.apache.org/flink/flink-ml-docs-BRANCH (e.g. flink-ml-docs-master)
  • https://nightlies.apache.org/flink/flink-ml-docs-ALIAS (e.g. flink-ml-docs-stable)

Triggering a build manually

Navigate to github actions and click "run workflow", then select a branch.

Triggering a build manually CURRENTLY BROKEN

You can manually trigger a build via an IRC buildbot.

  1. Go to: https://web.libera.chat/
  2. Pick a nickname (e.g. uce) and channel #flink-testing
  3. For Flink docs, type flink-buildbot: force build flink-docs-BRANCH
  4. For StateFun docs, type flink-sf-buildbot: force build flink-statefun-docs-BRANCH

The command in the 3rd step will force build the docs of the provided builder, e.g. flink-docs-release-1.5 or flink-statefun-docs-release-2.0 in the above example.

For Flink, you can check the progress of the build at, for example, https://ci2.apache.org/#/builders/17/builds/1.

For StateFun, you can check the progress of the build at, for example, https://ci2.apache.org/#/builders/14/builds/43.

For Flink ML, you can check the progress of the build at, for example, https://ci2.apache.org/#/builders/103/builds/2.

You can check the progress of all builders at https://ci2.apache.org/#/builders.

Cleaning up content

See https://nightlies.apache.org/authoring.html for instructions on how to delete content from nightlies.apache.org.

Note that directories containing an index.html file cannot be deleted; you first need to delete said file before deleting the directory.


Currently it is not possible to delete directories/files uploaded via Github Actions.

Instead one needs to run a workflow that deletes the directory via rsync.

As this is incredibly error prone with fatal consequences (i.e., wiping the entire documentation for all versions), this should NOT be attempted unless you absolutely know what you're doing.

Manual upload

As an emergency measure it is possible to manually upload files to nightlies.apache.org, as explained in https://nightlies.apache.org/authoring.html.

This must be used sparingly because each file needs to be uploaded individually.

As an example, the following was used to update the 1.9 documentation to contain the warning about being outdated:

# executed in docs/
for i in $(find . -name "*.html" -type f); do
        x=$(echo $i | cut -c 3-)
        curl --netrc-file ~/nightlies -T $x https://nightlies.apache.org/flink/flink-docs-release-1.9/$x
done

net-rc-file contents: machine nightlies.apache.org login <username> password <password>

Testing changes

Github Actions:

Modify the workflow to skip the rsync steps and trigger a manual build.

Buildbot:

Changes to the website build process can be tested by pushing to the docs_experimental__docs  and docs_experimental__docs_compile  branches (of the Flink repository) and triggering a manual build. The first branch will only run build_docs.sh , while the latter will also compile Flink and generate the java/scala/python docs.

Neither of these will push any content to nightlies.apache.org.

Building legacy documentation

This section contains instructions for building the documentation of older Flink versions.

Flink 1.0 -> 1.7:

docs/build_docs.sh:

-bundle install --path .rubydeps
+bundle install --full-index --path .rubydeps


docker run --rm --volume="$PWD:/srv" --expose=4000 -p 4000:4000 -it ruby:2.3.1 bash -c "cd /srv && ./build_docs.sh"

Flink 1.8 -> 1.12 / StateFun 2.0 -> 2.2:

docker run --rm --volume="$PWD:/srv" --expose=4000 -p 4000:4000 -it ruby:2.6.3 bash -c "cd /srv && ./build_docs.sh"
  • No labels