Page History

The Flink documentation is build nightly via Github Actions.

The Flink(-StateFun) /-ML documentation is build nightly in the Apache buildbot (https://ci2.apache.org/)  environmentenvironment.

The content is pushed to nightlies.apache.org/flink.

Build Environment

Github Actions

The following files configure the workflow for building the docs. Check it out for adjusting the build configuration.

• flink: https://github.com/apache/flink/blob/master/.github/workflows/docs.yml

Apache Buildbot

The following files configure buildbot for checking out branches and building the docs. Check it out for adjusting the build configuration.

• flink-statefun.conf: https://github.com/apache/infrastructure-bb2/blameblob/master/flink-statefun.py
• flink-statefunml.conf:  https://github.com/apache/infrastructure-bb2/blob/master/flink-statefunml.py

Branch configuration

The configuration files above contain a flink*_branches_nightly variable that configures the branches to build nightly:

...

# format: (branch, alias) or branch
flink_statefun_branches_nightly = [
 ('master', 'release-2.1'),
 ('release-2.0', 'stable')
]

or, for Flink ML:

# format: (branch, alias) or branch
flink_ml_branches_nightly = [
 ('master', 'release-2.1'),
 ('release-2.0', 'stable')
]

...

• https://nightlies.apache.org/flink/flink-statefun-docs-BRANCH (e.g. flink-statefun-docs-master)
• https://nightlies.apache.org/flink/flink-statefun-docs-ALIAS (e.g. flink-statefun-docs-stable)

The Flink ML docs will be served under the following URLs:

• https://nightlies.apache.org/flink/flink-ml-docs-BRANCH (e.g. flink-ml-docs-master)
• https://nightlies.apache.org/flink/flink-ml-docs-ALIAS (e.g. flink-ml-docs-stable)

Triggering a build manually

Navigate to github actions and click "run workflow", then select a branch.

Triggering a build manually CURRENTLY BROKEN

You can manually trigger a build via an IRC buildbot.

...

The command in the 3rd step will force build the docs of the provided builder, e.g. flink-docs-release-1.5 or flink-statefun-docs-release-2.0 in the above example.

[09:56] == uce [02cd5693@gateway/web/freenode/ip.2.205.86.147] has joined #flink-testing
[09:57] <uce> flink-buildbot: force build flink-docs-release-1.5
[09:57] <@flink-buildbot> build #7 forced
[09:57] <@flink-buildbot> I'll give a shout when the build finishes

Builder status

For Flink, you can check the progress of the build at, for example, https://ci2.apache.org/#/builders/17/builds/1.

For StateFun, you can check the progress of the build at, for example, https://ci2.apache.org/#/builders/14/builds/43.

For Flink ML, you can check the progress of the build at, for example, https://ci2.apache.org/#/builders/103/builds/2.

You can check the progress of all builders at https://ci2.apache.org/#/builders.

...

Note that directories containing an index.html file cannot be deleted; you first need to delete said file before deleting the directory.

Currently it is not possible to delete directories/files uploaded via Github Actions.

Instead one needs to run a workflow that deletes the directory via rsync.

As this is incredibly error prone with fatal consequences (i.e., wiping the entire documentation for all versions), this should NOT be attempted unless you absolutely know what you're doing.

Manual upload

As an emergency measure it is possible to manually upload files to nightlies.apache.org, as explained in https://nightlies.apache.org/authoring.html.

...

net-rc-file contents: machine nightlies.apache.org login <username> password <password>

Testing changes

Github Actions:

Modify the workflow to skip the rsync steps and trigger a manual build.

Buildbot:

Changes to the website build process can be tested by pushing to the docs_experimental__docs  and docs_experimental__docs_compile  branches (of the Flink repository) and triggering a manual build. The first branch will only run build_docs.sh , while the latter will also compile Flink and generate the java/scala/python docs.

...