Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

There are three areas where we shade our dependencies:

Internal Shading: Hadoop Dependencies

Internal shading is about hiding some of Hadoop's dependencies from Flink. 
Flink supports a broad range of Hadoop versions, which depend on different versions of certain dependencies.
Right now, we shade

...

NOTE: It is very important not to put guava, protobuf, netty and asm (== all shaded hadoop dependencies) into the dependencyManagement section of the parent pom. If they are there, they'll influence the version we'll pack into the shaded hadoop jar!

External Shading: Guava, ASM

External shading hides some of Flink's dependencies to Flink users. This allows users to use whatever version of Guava or ASM they want.
The external shading is controlled in the flink-parent pom, in the maven-shade-plugin. 

...

To add more dependencies to this external shading, add them to the artifactSet in the flink-parent/pom.xml

Final binary fat jar

Before the pull request mentioned above, Flink was not building a single fat-jar for the binary distribution. This was only done for YARN to reduce the number of files necessary to distribute to all containers.
With this change, we always create one fat-jar containing all flink jars and their dependencies.
If we would stick with the old approach (a jar file for each Flink module), we would end up with a very big binary distribution in the end, because all jars would contain all shaded dependencies (asm, guava etc.). 

...