Page History

In addition to this guide, also see the latest Flink documentation on the website.

Requirements for developing and building Flink

• Unix-like environment (We use Linux, Mac OS X, and Cygwin)
• git
• Maven (at least version 3.1.1)
• Java 8
• Python

Clone the repository

Apache Flink’s source code is stored in a git repository which is mirrored to GitHub. The common way to exchange code on GitHub is to fork the repository into your personal GitHub account. For that, you need to have a GitHub account or create one for free. Forking a repository means that GitHub creates a copy of the forked repository for you. This is done by clicking on the Fork button on the upper right of the repository website. Once you have a fork of Flink’s repository in your personal account, you can clone that repository to your local machine.

git clone https://github.com/<your-user-name>/flink.git

The code is downloaded into a directory called flink.

Azure Pipelines for CI

Azure Pipelines has restricted usage of free plan accounts to a case-by-case basis, making use of Flink's CI setup on a personal fork difficult. While the community decides on the best next-steps, there is a workaround to leverage the Flink community's Azure account for in-progress work without creating noise on the main apache/flink GitHub repository.

To do so, you'll need to fork the flink-ci/flink-mirror repository to your personal GitHub account.

Then, you'll need to add the mirror as a git remote in your local machine's Flink project directory.

cd /path/to/your/flink
git remote add flink-mirror https://github.com/<your-user-name>/flink-mirror.git  

To run your changes on Flink's Azure CI:

• push your branch to your flink-mirror fork
• open a draft PR in the flink-ci/flink-mirror GitHub repository

git push -u flink-mirror your-branch

When you're satisfied with your changes, you can:

• close the draft PR in the flink-ci/flink-mirror repository
• push your branch to your apache/flink fork
• open a new PR against the apache/flink GitHub repository

git push -u origin your-branch

Proxy Settings

If you are behind a firewall you may need to provide Proxy settings to Maven and your IDE.

For example, the WikipediaEditsSourceTest communicates over IRC and need a SOCKS proxy server to pass.

Setup an IDE and import the source code of Java/Scala

The Flink committers use IntelliJ IDEA (recommended) and Eclipse IDE to develop the Flink Java/Scala code base.

Minimal requirements for an IDE are:

• Support for Java and Scala (also mixed projects)
• Support for Maven with Java and Scala

IntelliJ IDEA

The IntelliJ IDE supports Maven out of the box and offers a plugin for Scala development.

• IntelliJ download: https://www.jetbrains.com/idea/
• IntelliJ Scala Plugin: http://plugins.jetbrains.com/plugin/?id=1347

Check out our Setting up IntelliJ guide for details.

Eclipse Scala IDE

For Eclipse users, we recommend using Scala IDE 3.0.3, based on Eclipse Kepler. While this is a slightly older version, we found it to be the version that works most robustly for a complex project like Flink.

Further details and a guide to newer Scala IDE versions can be found in the How to setup Eclipse docs.

Note: Before following this setup, make sure to run the build from the command line once (mvn clean install -DskipTests; see below).

1. Download the Scala IDE (preferred) or install the plugin to Eclipse Kepler. See How to setup Eclipse for download links and instructions.
2. Add the “macroparadise” compiler plugin to the Scala compiler. Open “Window” -> “Preferences” -> “Scala” -> “Compiler” -> “Advanced” and put into the “Xplugin” field the path to the macroparadise jar file (typically “/home/-your-user-/.m2/repository/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar”). Note: If you do not have the jar file, you probably did not run the command line build.
3. Import the Flink Maven projects (“File” -> “Import” -> “Maven” -> “Existing Maven Projects”)
4. During the import, Eclipse will ask to automatically install additional Maven build helper plugins.
5. Close the “flink-java8” project. Since Eclipse Kepler does not support Java 8, you cannot develop this project.

Import the source code

Apache Flink uses Apache Maven as build tool. Most IDEs are capable of importing Maven projects.

Setup an IDE and import the source code of Python

The Flink committers use PyCharm(recommended) to develop Flink Python code base.

PyCharm

The PyCharm is a friendly IDE to supports python development.

• PyCharm download: https://www.jetbrains.com/pycharm/

Check out our Setting up PyCharm guide for details.

Import the source code

The Python code of Apache Flink can be opened by PyCharm directly. The open path should be pointed to the flink-python module.

Build the code

To build Flink from source code, open a terminal, navigate to the root directory of the Flink source code, and call:

mvn clean package

This will build Flink and run all tests(without python test case). Flink is now installed in build-target.

To build Flink without executing the tests you can call:

mvn -DskipTests clean package

Run Python Test

After building the Flink source code, you can run Python test in the flink-python module:

sh dev/lint-python.sh