Page History

The Flink documentation is build nightly via Github Actions.

The Flink-StateFun/-ML documentation is build nightly in the Apache buildbot (https://cici2.apache.org/buildbot.html) environment and served by it as well) environment.

The content is pushed to nightlies.apache.org/flink.

Build Environment

Github Actions

The following files configure the workflow for building the docs. Check it out for adjusting the build configuration.

• flink: https://github.com/apache/flink/blob/master/.github/workflows/docs.yml

Apache Buildbot

The file flink.conf configures following files configure buildbot for checking out Flink branches and building the docs. Check it out for adjusting the build configuration.

• flink-statefun.conf: https://svngithub.apache.org/repos/infra/infrastructure/buildbot/aegis/buildmaster/master1/projects/flink.confcom/apache/infrastructure-bb2/blob/master/flink-statefun.py
• flink-ml.conf: https://github.com/apache/infrastructure-bb2/blob/master/flink-ml.py

Branch configuration

The flink.conf file mentioned above contains a flinkconfiguration files above contain a flink*_branches_nightly variable that configures the branches to build nightly:

# format: (branch, alias) or branch
flink_branches_nightly = [
 ('master', 'release-1.6'),
 ('release-1.5', 'stable'),
 'release-1.4',
 'release-1.3',
 'release-1.2',
 'release-1.1',
 'release-1.0'
]

or, for Stateful Functions:

# format: (branch, alias) or branch
flink_statefun_branches_nightly = [
 ('master', 'release-2.1'),
 ('release-2.0', 'stable')
]

or, for Flink ML:

# format: (branch, alias) or branch
flink_ml_branches_nightly = [
 ('master', 'release-2.1'),
 ('release-2.0', 'stable')
]

The Flink docs will be served under the following URLs:

• https://cinightlies.apache.org/projects/flink/flink-docs-BRANCH (e.g. flink-docsmasterdocs-master)
• https://cinightlies.apache.org/projects/flink/flink-docs-ALIAS (e.g. flink-docs-release-1.6)

The StateFun docs will be served under the following URLs:

• https://nightlies.apache.org/flink/flink-statefun-docs-BRANCH (e.g. flink-statefun-docs-master)
• https://nightlies.apache.org/flink/flink-statefun-docs-ALIAS (e.g. flink-statefun-docs-stable)

The Flink ML docs will be served under the following URLs:

• https://nightlies.apache.org/flink/flink-ml-docs-BRANCH (e.g. flink-ml-docs-master)
• https://nightlies.apache.org/flink/flink-ml-docs-ALIAS (e.g. flink-ml-docs-stable)

Triggering a build manually

Navigate to github actions and click "run workflow", then select a branch.

Triggering a build manually CURRENTLY BROKEN

You can manually trigger a build via an IRC buildbot.

1. Go to: https://webchatweb.freenodelibera.netchat/
2. Pick a nickname (e.g. uce) and channel #flink-testing
3. Type For Flink docs, type flink-buildbot: force build flink-docs-BRANCH
4. For StateFun docs, type flink-release-1.5 build me the docs pleasesf-buildbot: force build flink-statefun-docs-BRANCH

The command in the 3rd step will force build the docs of the provided builder, e.g. flink-docs-release-1.5 or flink-statefun-docs-release-2.0 in the above example.

[09:56]

...

==

...

uce

...

[02cd5693@gateway/web/freenode/ip.2.205.86.147]

...

has

...

joined

...

#flink-testing
[09:57]

...

<uce>

...

flink-buildbot:

...

force

...

build

...

flink-docs-release-1.5

...

[09:57]

...

<@flink-buildbot>

...

build

...

#7

...

forced
[09:57]

...

<@flink-buildbot>

...

I'll

...

give

...

a

...

shout

...

when

...

the

...

build

...

finishes

...

Builder status

For Flink, you You can check the progress of the build at at, for example, https://cici2.apache.org/#/builders/flink-docs-release-1.5/builds/7/17/builds/1.

For StateFun, you can check the progress of the build at, for example, https://ci2.apache.org/#/builders/14/builds/43.

For Flink ML, you can check the progress of the build at, for example, https://ci2.apache.org/#/builders/103/builds/2.

 where the last number is the build number. You can check the progress of all builders at https://cici2.apache.org/#/builders.

Cleaning up content

See https://nightlies.apache.org/authoring.html for instructions on how to delete content from nightlies.apache.org.

Note that directories containing an index.html file cannot be deleted; you first need to delete said file before deleting the directory.

Currently it is not possible to delete directories/files uploaded via Github Actions.

Instead one needs to run a workflow that deletes the directory via rsync.

As this is incredibly error prone with fatal consequences (i.e., wiping the entire documentation for all versions), this should NOT be attempted unless you absolutely know what you're doing.

Manual upload

As an emergency measure it is possible to manually upload files to nightlies.apache.org, as explained in https://nightlies.apache.org/authoring.html.

This must be used sparingly because each file needs to be uploaded individually.

As an example, the following was used to update the 1.9 documentation to contain the warning about being outdated:

# executed in docs/
for i in $(find . -name "*.html" -type f); do
        x=$(echo $i | cut -c 3-)
        curl --netrc-file ~/nightlies -T $x https://nightlies.apache.org/flink/flink-docs-release-1.9/$x
done

net-rc-file contents: machine nightlies.apache.org login <username> password <password>

Testing changes

Github Actions:

Modify the workflow to skip the rsync steps and trigger a manual build.

Buildbot:

Changes to the website build process can be tested by pushing to the docs_experimental__docs  and docs_experimental__docs_compile  branches (of the Flink repository) and triggering a manual build. The first branch will only run build_docs.sh , while the latter will also compile Flink and generate the java/scala/python docs.

Neither of these will push any content to nightlies.apache.org.

Building legacy documentation

This section contains instructions for building the documentation of older Flink versions.

Flink 1.0 -> 1.7:

docs/build_docs.sh:

-bundle install --path .rubydeps
+bundle install --full-index --path .rubydeps

docker run --rm --volume="$PWD:/srv" --expose=4000 -p 4000:4000 -it ruby:2.3.1 bash -c "cd /srv && ./build_docs.sh"

Flink 1.8 -> 1.12 / StateFun 2.0 -> 2.2:

docker run --rm --volume="$PWD:/srv" --expose=4000 -p 4000:4000 -it ruby:2.6.3 bash -c "cd /srv && ./build_docs.sh"