You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 7 Next »

Use the right build command to build only the things that changed:

# Regenerate cmake files.
./buildall.sh -cmake_only

# Rebuild impalad binary only:
make -j ${IMPALA_BUILD_THREADS} impalad

# Rebuild a backend test binary only:
make -j ${IMPALA_BUILD_THREADS} buffered-block-mgr-test

# Rebuild frontend only:
make fe

Pass the right command line flags to skip unnecessary work:

  • Add -notests to buildall.sh command to skip building backend tests
  • Pass -noclean to buildall.sh if you don't need to do a full clean

Ccache - speeds up rebuilds when switching between branches

Ccache is a standard tool to accelerate C/C++ builds by caching previously-built object files. It can tremendously speed up builds when switching between branches of Impala.

# Install the ccache package (this example works on Ubuntu)
sudo apt-get install ccache

# Make sure you have a high limit on your max ccache size to ensure it is effective
ccache --max-size=50G

Ninja - speeds up incremental builds

Ninja is an alternative to make that can resolve dependencies much faster. This can save 10+ seconds per invocation compared to make, which is significant for incremental builds (the difference in speed between ninja and make for full builds is minor compared to the end-to-end runtime of the build).

# Install the ninja package (this example works on Ubuntu)
sudo apt-get install ninja-build

# Instead of ./buildall.sh ...:
./buildall.sh ... -ninja

# Instead of make -j ${IMPALA_BUILD_THREADS} <targets>:
ninja -j ${IMPALA_BUILD_THREADS} <targets>

# You can use an alias to make ninja use IMPALA_BUILD_THREADS by default
alias ninja='ninja -j ${IMPALA_BUILD_THREADS}'

Skip toolchain bootstrapping

Bootstrapping the toolchain and python dependencies can add latency to buildall.sh invocations even if nothing needs to be downloaded. You can disable this by setting SKIP_TOOLCHAIN_BOOTSTRAP=true in bin/impala-config-local.sh. Note that this means you need to manually set SKIP_TOOLCHAIN_BOOTSTRAP=false when running buildall.sh to download dependencies when they do change.

Using distcc

The instructions for using distcc are in bin/distcc/README.md

  • No labels