Page History

Prerequisites

1. Download IntelliJ and CLion. You can request for an open source license for CLion.
2. Bootstrap the system for Impala development. See this link.
3. Always source $IMPALA_HOME/bin/impala-config.sh prior to launching IntelliJ or CLion.
4. To be able to attach to a local process, follow the instruction here.

Front-end Development using IntelliJ

Setup

1. Launch IntelliJ and click "Open".
   Image Added
2. Select the $IMPALA_HOME/fe directory.
   Image Added

Run/Debug Unit Tests

1. Go to Run --> Edit Configurations.
   Image Added
2. Select Defaults --> JUnit
   Image Added
3. Add LD_LIBRARY_PATH with the value of $IMPALA_HOME/be/build/latest/service.
   Image Added
4. Run all JUnit tests.
   Image Added

Remote Debugging

1. Go to Run --> Edit Configurations.
   Image Added
2. Click on the + sign and select Remote.
   Image Added
3. Add impalad-debug configuration to debug impalad with a port number of 30000.
   Image Added
4. Add catalog-debug configuration to debug catalog with a port number of 30030.
   Image Added
5. Set a breakpoint.
   Image Added
6. Run impalad-debug.
   Image Added
7. Execute a command in Impala shell.
   Image Added
8. Start debugging.
   Image Added

Backend Development using CLion

Setup

1. Launch CLion and click "Open".
   Image Added
2. Select $IMPALA_HOME directory.
   Image Added

Local Debugging

1. Go to Run --> Attach to Local Processes.
   Image Added
   Image Added
   If you see an error "ptrace: Operation not permitted" then follow the instructions here to allow ptracing of non-child processes.
2. Set a breakpoint.
   Image Added
3. Execute a command in Impala shell.
   Image Added
4. Start debugging.
   Image Added

Impala Shell Development using CLion

Setup

1. Go to File --> Settings.
   Image Added
2. Go to Build, Execution, Deployment --> Python Interpreter.
   Image Added
3. Add an existing virtual env environment.
   Image Added
4. Select the python interpreter in $IMPALA_HOME/infra/python/env/bin/python
   Image Added

Local Debugging

1. Go to Run --> Attach to Local Processes.
   Image Added
   Image Added
2. Set a breakpoint.
   Image Added
3. Execute a command in Impala shell.
   Image Added
4. Start debugging.
   Image Added

Use PyCharm to Run/Debug impala_shell.py

Need to set Environment Variables like what $IMPALA_HOME/bin/impala-python did:

        1. bash -x $IMPALA_HOME/bin/impala-python

        2. Copy PYTHONPATH and paste it to Environment Variables in Run/Debug Configurations

Image Added

      3. Environment Variables

Image Added    

     4. Run/Debug Configurations

Image Added

      5. Setting break point and debug 

Using clang-format in IntelliJ and CLion

You can use the ClangFormatIJ plugin to run clang-format in IntelliJ and CLion

WIP: Using IntelliJ and CLion on Mac

The above content requires that you can bootstrap your system successfully. You may encounter many problems in MacOS, i.e. there are no pre-built toolchain for Mac and the toolchain may not be compilable on Mac.

However, you can still use IntelliJ and CLion on Mac for coding as long as you have compiled Impala successfully in a Ubuntu environment. If you have a Ubuntu machine, go and compile Impala in it first (link). If you don't have a Ubuntu machine, you can compile Impala in Docker.

Steps

• After you compile Impala successfully in your Ubuntu environment, package these folders in $IMPALA_HOME: be/generated-sources, fe/generated-sources, toolchain. E.g. make a tar file for them. (Transferring lots of small files between file systems will be time consuming)
• Transfer those tar files back to your MacOS laptop and unpack them in your $IMPALA_HOME. Now the contents of $IMPALA_HOME in your MacOS laptop should be the same as your Ubuntu environment. (Maybe you can use rsync as well. But be aware of those compilation outputs and cmake cache files. Don't need to copy them)
• Apply this patch in your MacOS env: dev_env_for_Mac.patch (Command: git apply $somewhere/dev_env_for_Mac.patch). The patch comments out the dependency of LLVM (TODO: resolve this in IMPALA-8229) and fix some stale versions and java paths.

• Then source $IMPALA_HOME/bin/impala-config.sh and launch CLion/IntelliJ in your terminal. E.g. 

  Code Block
  cd $IMPALA_HOME source bin/impala-config.sh /Applications/CLion.app/Contents/MacOS/clion # or /Applications/IntelliJ\ IDEA.app/Contents/MacOS/idea

  
  

• There're some conflicts for CLion and IntelliJ to work in the same folder, e.g. they both overwrite others' iml files. It's recommended that making dedicated Impala dir for them.
• If all go well, your CLion/IntelliJ will still have some errors but they can build index successfully. So you should be able to navigating in the code base, e.g. finding definitions, finding usages, showing call hierarchy, etc.

Tips for development on Mac

• Let's say you have two directories, impala-be and impala-fe, for CLion and IntelliJ respectively. You can make C++/Python changes in CLion, and Java changes in IntelliJ.
• After you make some changes, use git-diff to generate a patch, e.g. use "git diff > be-tmp.patch"  in impala-be dir. Then send the patch files to your Ubuntu environment and apply them by git-apply, e.g. use "git apply be-tmp.patch".
• You can't use local debugging in Mac since there still some errors to be resolve in CLion/IntelliJ (IMPALA-8229). However, you can enjoy debugging by logs . See more tips here for how to add useful logs while debugging.

This document is to help Impala front-end development using IntelliJ 

Prerequisites

• Please read the document before starting this: Eclipse Setup for Impala Development
• Please build your own dev. system: Bootstrapping an Impala Development Environment From Scratch
  • The build is required due to the dependent jars.
• Please skip this step if you already have IntelliJ license.
  • IntelliJ Community Edition is free for open-source developer. Please download Community Edition here: https://www.jetbrains.com/idea/download

Setup

...

Image Removed

...

Remote Debugging

1. Choose "Run-Edit Configurations"
   Image Removed
2. Press the + button. Scroll down the list menu and choose "Remote"
   Image Removed
3. Describe some information such as Name, Host, Port. Use port 30000 for the first impalad, port 30001 for the second impalad.
   Image Removed
4. Press debug icon on the top of the right side.
   Image Removed
5. Connected the Impala fronet end successfully!
   Image Removed
6. Let's set a breakpoint.
   Image Removed
7. Execute a query like this:
   Image Removed
8. Enjoy debugging!Image Removed

...