Prerequisites

  1. Download IntelliJ and CLion. You can request for an open source license for CLion.
  2. Bootstrap the system for Impala development. See this link.
  3. Always source $IMPALA_HOME/bin/impala-config.sh prior to launching IntelliJ or CLion.
  4. To be able to attach to a local process, follow the instruction here.

Front-end Development using IntelliJ

Setup

  1. Launch IntelliJ and click "Open".
  2. Select the $IMPALA_HOME/fe directory.

Run/Debug Unit Tests

  1. Go to Run --> Edit Configurations.
  2. Select Defaults --> JUnit
  3. Add LD_LIBRARY_PATH with the value of $IMPALA_HOME/be/build/latest/service.
  4. Run all JUnit tests.

Remote Debugging

  1. Go to Run --> Edit Configurations.
  2. Click on the + sign and select Remote.
  3. Add impalad-debug configuration to debug impalad with a port number of 30000.
  4. Add catalog-debug configuration to debug catalog with a port number of 30030.
  5. Set a breakpoint.
  6. Run impalad-debug.
  7. Execute a command in Impala shell.
  8. Start debugging.

Backend Development using CLion

Setup

  1. Launch CLion and click "Open".
  2. Select $IMPALA_HOME directory.

Local Debugging

  1. Go to Run --> Attach to Local Processes.


    If you see an error "ptrace: Operation not permitted" then follow the instructions here to allow ptracing of non-child processes.
  2. Set a breakpoint.
  3. Execute a command in Impala shell.
  4. Start debugging.


Impala Shell Development using CLion

Setup

  1. Go to File --> Settings.
  2. Go to Build, Execution, Deployment --> Python Interpreter.
  3. Add an existing virtual env environment.
  4. Select the python interpreter in $IMPALA_HOME/infra/python/env/bin/python

Local Debugging

  1. Go to Run --> Attach to Local Processes.

  2. Set a breakpoint.
  3. Execute a command in Impala shell.
  4. Start debugging.

Use PyCharm to Run/Debug impala_shell.py

Need to set Environment Variables like what $IMPALA_HOME/bin/impala-python did:

        1. bash -x $IMPALA_HOME/bin/impala-python

        2. Copy PYTHONPATH and paste it to Environment Variables in Run/Debug Configurations

      3. Environment Variables

    

     4. Run/Debug Configurations

      5. Setting break point and debug 

Using clang-format in IntelliJ and CLion

You can use the ClangFormatIJ plugin to run clang-format in IntelliJ and CLion

WIP: Using IntelliJ and CLion on Mac

The above content requires that you can bootstrap your system successfully. You may encounter many problems in MacOS, i.e. there are no pre-built toolchain for Mac and the toolchain may not be compilable on Mac.

However, you can still use IntelliJ and CLion on Mac for coding as long as you have compiled Impala successfully in a Ubuntu environment. If you have a Ubuntu machine, go and compile Impala in it first (link). If you don't have a Ubuntu machine, you can compile Impala in Docker.

Steps

  • After you compile Impala successfully in your Ubuntu environment, package these folders in $IMPALA_HOME: be/generated-sources, fe/generated-sources, toolchain. E.g. make a tar file for them. (Transferring lots of small files between file systems will be time consuming)
  • Transfer those tar files back to your MacOS laptop and unpack them in your $IMPALA_HOME. Now the contents of $IMPALA_HOME in your MacOS laptop should be the same as your Ubuntu environment. (Maybe you can use rsync as well. But be aware of those compilation outputs and cmake cache files. Don't need to copy them)
  • Apply this patch in your MacOS env: dev_env_for_Mac.patch (Command: git apply $somewhere/dev_env_for_Mac.patch). The patch comments out the dependency of LLVM (TODO: resolve this in IMPALA-8229) and fix some stale versions and java paths.

  • Then source $IMPALA_HOME/bin/impala-config.sh and launch CLion/IntelliJ in your terminal. E.g. 

    cd $IMPALA_HOME
source bin/impala-config.sh
/Applications/CLion.app/Contents/MacOS/clion    # or /Applications/IntelliJ\ IDEA.app/Contents/MacOS/idea
  • There're some conflicts for CLion and IntelliJ to work in the same folder, e.g. they both overwrite others' iml files. It's recommended that making dedicated Impala dir for them.
  • If all go well, your CLion/IntelliJ will still have some errors but they can build index successfully. So you should be able to navigating in the code base, e.g. finding definitions, finding usages, showing call hierarchy, etc.

Tips for development on Mac

  • Let's say you have two directories, impala-be and impala-fe, for CLion and IntelliJ respectively. You can make C++/Python changes in CLion, and Java changes in IntelliJ.
  • After you make some changes, use git-diff to generate a patch, e.g. use "git diff > be-tmp.patch"  in impala-be dir. Then send the patch files to your Ubuntu environment and apply them by git-apply, e.g. use "git apply be-tmp.patch".
  • You can't use local debugging in Mac since there still some errors to be resolve in CLion/IntelliJ (IMPALA-8229). However, you can enjoy debugging by logs (smile). See more tips here for how to add useful logs while debugging.