You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Dependency Shading has been extended in this pull request: https://github.com/apache/flink/pull/454

Before the change, Flink had a "flink-shaded" module which contained a relocated guava dependency. All guava dependencies inside the individual modules were set to the 'provided' scope.

 

There are three areas where we shade our dependencies:

Internal shading is about hiding some of Hadoop's dependencies from Flink. 
Flink supports a broad range of Hadoop versions, which depend on different versions of certain dependencies.
Right now, we shade

  • everything in the com.google namespace (guava and protobuf) 
  • org.objectweb.asm
  • org.jboss.netty

They all end up in org.apache.flink.hadoop.shaded.*
The shading of hadoop dependencies is done in "flink-shaded-hadoop" and its sub-modules.
Originally, we wanted to to handle the different hadoop versions inside this module, but the maven-shade-plugin doesn't properly support maven build profiles when creating the dependency reduced pom (a pom file which doesn't contain the shaded dependencies. In our case they won't contain guava, asm or netty anymore). 
To resolve that, the "flink-shaded-hadoop" is the parent of some version specific flink-shaded-hadoop modules. The parent contains the configuration of the shading (the sub modules will inherit the configuration).
The submodules have the following purpose:

  • flink-shaded-hadoop1 Is for all hadoop 0.2X and 1.XX versions. It contains only hadoop-core + some dependency exclusions
  • flink-shaded-hadoop2 is for all hadoop versions starting from 2.x. It contains dependencies for hadoop-common, hadoop-hdfs, hadoop-mapreduce-client-core (for the hadoop compatibility stuff in flink-java). This module is only used for the hadoop 2.0.0-alpha build. It contains only some MapReduce interfaces and the HDFS client. It does not contain YARN.
  • flink-shaded-include-yarn This module is for Hadoop 2.2.0+. This module contains the same dependencies as the hadoop2 module above + some dependencies for YARN.
  • flink-shaded-include-yarn-tests This module is like flink-shaded-include-yarn but it contains all classes required for flink-yarn-tests. The dependencies are NOT in the 'tests' scope, because the maven-shade-plugin can not relocate them from there. We need this module to consistently shade all required classes when running the tests.

All other modules in flink which need something from hadoop need to depend on the right flink-shaded-hadoop submodule. Therefore the flink-parent pom exposes the shading-artifact.name variable. It is set depending on the requested hadoop dependency set.
To depend on Hadoop, add the following entry to your pom:

<dependency>
	<groupId>org.apache.flink</groupId>
	<artifactId>${shading-artifact.name}</artifactId>
	<version>${project.version}</version>
</dependency>

 

 

  • No labels