Introduction

The website build pipeline consists of Makefiles, Sphinx, a variety of Sphinx plugins, a Sphinx website theme (and associated client-side modifications), API-specific documentation generation tools such as doxygen, roxygen, scaladocs and lein, shell scripts, Python scripts, Jenkins CI, a GitHub repo for the website artifacts, and Apache infrastructure to host the website.

The core content of the website, that is the home page, tutorial pages, and the like, are all sourced from Markdown files. These are transpiled by Sphinx into html. The content in the API sections utilizes a mix of Markdown and rst. The rst are triggers and special functions of Sphinx that can pull in data from API-specific source code.

The primary engine here is Sphinx. It builds the project from source, it calls all of the other documentation build tools, and it outputs the html of the website.

At a high level you can imagine this flow:

branch/tag of source code → md (rst mixed in) → Sphinx → rst intermediate → Sphinx plugins calling other documentation engines → Sphinx website theme applied → html representing one version of the website (a version)

Wrapped around this are two shell scripts:

  1. build_all_version.sh: this builds the list of versions (branches or tags) from a fork, and places them into subfolders by names
    1. inputs: versions list, names, and fork
    2. outputs: each version of the website in its own folder
  2. update_all_version.sh: this sets the default version of the site, injects the root URL, injects the versions dropdown, and copies in various website artifacts
    1. inputs: the output of previous build script, the folder name of the default version, and the root URL (for development this would be http://your-ip-address/)
    2. outputs: the fully updated website in a `VersionedWeb` folder.

Note: Ideally, this injection process would not happen. It would be handled as part of the website's overall layout as dynamically loaded content. However, the website's current design expects this injection process, and cleaning this up requires a website redesign.

Building the full site with your fork

The previous example building a single version was a precursor to the more complicated process of building the full site and its many versions.
The scripts used by CI and that can be used yourself for building dev version of the site are in build_version_doc and are still build_all_version.sh and update_all_version.sh. 

These scripts call make docs which in turn calls make html from the docs directory. So, they inherit the properties described in the settings file. The version tags get passed down from the shell script into the docs Makefile invocation, which then is read in the Sphinx plugin mxdoc.py, which reads the settings file, which is what then calls all of the other docs generation tools. 

Building Each Version and Optionally Using Your Fork

Build the content of the 1.2.0 branch in the main project repo to the 1.2.1 folder:

./build_all_version.sh "v1.2.0" "1.2.1"

Using the main project repo, map the 1.2.0 branch to output to a 1.2.1 directory; others as is:

./build_all_version.sh "v1.2.0;v1.1.0;master" "1.2.1;1.1.0;master"

Using a custom branch and fork of the repo, map the branch to master, map 1.2.0 branch to 1.2.1 and leave 1.1.0 in 1.1.0:

./build_all_version.sh "sphinx_error_reduction;v1.2.0;v1.1.0" "master;1.2.1;1.1.0" https://github.com/aaronmarkham/incubator-mxnet.git

Updating Each Version and Optionally Using Your Fork

Assuming you build 1.2.1, 1.1.0, 1.0.0, and master, you need to inject the versions dropdown to reflect these options for each page on the site:

./update_all_version.sh "1.2.1;1.1.0;1.0.0;master" master http://mxnet.incubator.apache.org/

It doesn't use your fork at this point, but it will pull the latest project README.md from your current branch and use that for the root of the output site.

Troubleshooting: When running locally in your fork, you may run into errors trying to use the upstream branches. They may not exist in your local repo. You can get around this by using tags instead. Just drop the "v" and use the plain version number. Or you can manually fetch each one that you intend to use so that you have the appropriate branch locally when you run these scripts.

Review the intermediate outputs and the final outputs

Both are really useful and can save you time later.

Intermediate build outputs

The intermediate outputs would the result of each version build. Navigate to docs/build_version_doc/<fork-username>. For my fork this would be docs/build_version_doc/aaronmarkham and inside I would see folders for master, 1.2.1, and 1.1.0 after running each earlier example. The contents of the master folder would have come from my sphinx_error_reduction branch. The contents of the other versions would be as follows: the 1.2.1 folder would have the content from the v1.2.0 branch, and the other two (1.1.0 and 1.0.0) would have the content from their respective branches. Renaming is useful with the branch names don't exactly match what's in them.

Final "updated" site outputs

The updated site outputs can be found by navigating to docs/build_version_doc/VersionedWeb. Note the structure has a version of the website as the root, then there is a versions folder that has each named version inside. Also note the .htaccess file. This is one of the artifacts that is copied during the update step and always placed in the root. It handles the redirects. Another artifact is the README.md file.

A full site example

Now that you have seen a few examples, let's take a look at a sample script the deploys the full site on a dev server. Let's call this fawkes_fly.sh.

It will accomplish the following tasks:

  1. Run from the docs/build_version_doc/ folder.
  2. Execute the build script passing in all of the versions you want to run. In this example it uses tags instead of branches. It also uses a branch called "sphinx_error_reduction" which on the custom fork provided as the last parameter. You would want to switch these to elements out for your branch and fork respectively.
  3. Execute the update script setting the versions to include in the website, the default version, and the root URL.
  4. Delete the previously hosted root website (important to cleanup and delete old files when there are API docs updates).
  5. Copy the full site build to the web serving folder.
cd ~/incubator-mxnet/docs/build_version_doc/
./build_all_version.sh "v1.4.x;v1.3.x;v1.2.0;v1.1.0;v1.0.0;v0.12.0;v0.11.0;master" "1.4.0;1.3.1;1.2.1;1.1.0;1.0.0;0.12.1;0.11.0;master" https://github.com/aaronmarkham/incubator-mxnet.git
./update_all_version.sh "1.4.0;1.3.1;1.2.1;1.1.0;1.0.0;0.12.1;0.11.0;master" master http://34.201.8.176/
sudo rm -rf /var/www/html/*
sudo cp -a ~/incubator-mxnet/docs/build_version_doc/VersionedWeb/* /var/www/html/

After you have run this, you should be able to open your web browser and visit your site by its IP. Note it now has a versions dropdown.

Build and serve multiple features at once

A handy feature described in the previous example is using your own branch and fork to test your branch's latest updates. Instead of using "master" as the name for the custom branch, use the custom branch name, or whatever you want to appear on the site, so that people can use the dropdown to switch to that feature's view. In this way, you could deploy a site will a variety of branches to do A/B comparisons. 

If you have already built various versions and you don't want to recreate them, then don't do the delete step in the earlier example. Retain the website. Each version is in the website's /version folder. Also, each version that was built and pre-update is in the docs/build_version_doc/<fork-username> folder. You can also run the one-off site build using make html. Copy your newly created versions of the site into aptly named subfolders within the the docs/build_version_doc/<fork-username> folder. Then rerun update_all_version.sh with the versions you want to appear in the dropdown list. Take that output and serve it, or even make a new directory in /var/www/html/versions and copy it in there, so you can run many flavors of the site at will.

Troubleshooting

  • Sometimes the submodules get updated in master and you have to specifically update them yourself. If you see a build error that crashes on mshadow or another submodule's build you likely need to run the following command in your MXNet repo:
     git submodule update --init --recursive
  • Another more rare occurrence is someone did something fairly major and you get a build error in MXNet like "no rule to make target...". Try running the following in the MXNet repo root of the version that is failing:
     make clean
  • You might get nagged about a file that is going to get overwritten with a checkout when you've made local changes in your branch, but are trying to run a test build from the upstream branch. At this point it is better to set the build_all_version script to use your fork, so you can test it out.

Next up

Website Updating Guide (pre v1.6)


  • No labels