The Flink documentation is build nightly via Github Actions.
The Flink-StateFun/-ML documentation is build nightly in the Apache buildbot (https://cici2.apache.org/buildbot.html) environment and served by it as well) environment.
The StateFun documentation is built nightly and served at content is pushed to nightlies.apache.org/flink.
Build Environment
Github Actions
The following files configure the workflow for building the docs. Check it out for adjusting the build configuration.
- flink: https://
...
...
Apache Buildbot
The following files configure buildbot for checking out branches and building the docs. Check it out for adjusting the build configuration.
- flink-statefun.conf: https://svngithub.apache.org/repos/infracom/apache/infrastructure-bb2/buildbot/aegis/buildmaster/master1/projectsblob/master/flink-statefun.confpy
- flink-statefunml.conf: https://github.com/apache/infrastructure-bb2/blob/master/flink-statefunml.py
Branch configuration
The configuration files above mentioned above contains contain a flink*_branches_nightly variable that configures the branches to build nightly:
...
# format: (branch, alias) or branch
flink_statefun_branches_nightly = [
('master', 'release-2.1'),
('release-2.0', 'stable')
]
or, for Flink ML:
# format: (branch, alias) or branch
flink_ml_branches_nightly = [
('master', 'release-2.1'),
('release-2.0', 'stable')
]
The Flink docs will be served under the following URLs:
- https://cinightlies.apache.org/projects/flink/flink-docs-BRANCH (e.g. flink-docs-master)
- https://cinightlies.apache.org/projects/flink/flink-docs-ALIAS (e.g. flink-docs-release-1.6)
...
- https://nightlies.apache.org/flink/flink-statefun-docs-BRANCH (e.g. flink-statefun-docs-master)
- https://nightlies.apache.org/flink/flink-statefun-docs-ALIAS (e.g. flink-statefun-docs-stable)
The Flink ML docs will be served under the following URLs:
- https://nightlies.apache.org/flink/flink-ml-docs-BRANCH (e.g. flink-ml-docs-master)
- https://nightlies.apache.org/flink/flink-ml-docs-ALIAS (e.g. flink-ml-docs-stable)
Triggering a build manually
Navigate to github actions and click "run workflow", then select a branch.
Triggering a build manually CURRENTLY BROKEN
You can manually trigger a build via an IRC buildbot.
- For Flink, go to: https://webchat.freenode.net/. For StateFun, go
Go to: https://web.libera.chat/ Pick a nickname (e.g. uce) and channel #flink-testingFor Flink docs, type flink-buildbot: force build flink-docs-BRANCHFor StateFun docs, type flink-sf-buildbot: force build flink-statefun-docs-BRANCH
The command in the 3rd step will force build the docs of the provided builder, e.g. flink-docs-release-1.5 or flink-statefun-docs-release-2.0 in the above example.
[09:56]
...
==
...
uce
...
[02cd5693@gateway/web/freenode/ip.2.205.86.147]
...
has
...
joined
...
#flink-testing
[09:57]
...
<uce>
...
flink-buildbot:
...
force
...
build
...
flink-docs-release-1.5
[09:57]
...
<@flink-buildbot>
...
build
...
#7
...
forced
[09:57]
...
<@flink-buildbot>
...
I'll
...
give
...
a
...
shout
...
when
...
the
...
build
...
finishes
Builder status
For Flink, you can check the progress of the build at at, for example, https://cici2.apache.org/#/builders/flink-docs-release-1.517/builds/7 where the last number is the build number. You 1.
For StateFun, you can check the progress of all builders at the build at, for example, https://cici2.apache.org/#/builders/14/builds/43.
For StateFunFlink ML, you can check the progress of the build at, for example, https://ci2.apache.org/#/builders/14103/builds/432.
You can check the progress of all builders at https://ci2.apache.org/#/builders.
Cleaning up content
See https://nightlies.apache.org/authoring.html for instructions on how to delete content from nightlies.apache.org
.
Note that directories containing an index.html
file cannot be deleted; you first need to delete said file before deleting the directory.
Currently it is not possible to delete directories/files uploaded via Github Actions.
Instead one needs to run a workflow that deletes the directory via rsync.
As this is incredibly error prone with fatal consequences (i.e., wiping the entire documentation for all versions), this should NOT be attempted unless you absolutely know what you're doing.
Manual upload
As an emergency measure it is possible to manually upload files to nightlies.apache.org
, as explained in https://nightlies.apache.org/authoring.html.
This must be used sparingly because each file needs to be uploaded individually.
As an example, the following was used to update the 1.9 documentation to contain the warning about being outdated:
Code Block |
---|
# executed in docs/
for i in $(find . -name "*.html" -type f); do
x=$(echo $i | cut -c 3-)
curl --netrc-file ~/nightlies -T $x https://nightlies.apache.org/flink/flink-docs-release-1.9/$x
done |
net-rc-file contents: machine nightlies.apache.org login <username> password <password>
Testing changes
Github Actions:
Modify the workflow to skip the rsync steps and trigger a manual build.
Buildbot:
Changes to the website build process can be tested by pushing to the docs_experimental__docs
and docs_experimental__docs_compile
branches (of the Flink repository) and triggering a manual build. The first branch will only run build_docs.sh
, while the latter will also compile Flink and generate the java/scala/python docs.
Neither of these will push any content to nightlies.apache.org
.
Building legacy documentation
This section contains instructions for building the documentation of older Flink versions.
Flink 1.0 -> 1.7:
docs/build_docs.sh
:
Code Block |
---|
-bundle install --path .rubydeps
+bundle install --full-index --path .rubydeps |
Code Block |
---|
docker run --rm --volume="$PWD:/srv" --expose=4000 -p 4000:4000 -it ruby:2.3.1 bash -c "cd /srv && ./build_docs.sh" |
Flink 1.8 -> 1.12 / StateFun 2.0 -> 2.2:
Code Block |
---|
docker run --rm --volume="$PWD:/srv" --expose=4000 -p 4000:4000 -it ruby:2.6.3 bash -c "cd /srv && ./build_docs.sh" |