...

The primary format of a Flink release is the source distribution (this is true of all Apache projects). In a sense, the "source distribution" is the ground truth of a release, and the thing that really needs to get checked by the PMC during the release voting. 

Flink also provides a binary convenience distribution as well as Maven artifacts.

Each of these distributions must observe the rules described below for “bundled” software dependencies.The rules dictate the content of the LICENSE and NOTICE files for each distribution type.

All distributions must contain the following files, with varying rules depending on the distribution type:

• a LICENSE file, containing the Apache License
• a NOTICE file, containing a list of every bundled dependency
• a licenses directory, containing the license of each dependency listed in the NOTICE file (if they are not Apache licensed)

Source distribution

The contents of the source distribution must closely reflect the files in source control with a few minor exclusions.

Code that has been copied either in part or in whole from other projects into the Flink codebase is considered "bundled" in the source distribution.

If a bundled dependency/copied code is present in the source distribution, the license must be a Category A license [3].

All licensing files for the source distribution must be placed into the root directory of the distribution.

The LICENSE and NOTICE files used for the source distribution can be found in the project root directory. The root directory also contains the licenses directory which contains the LICENSE files of all bundled code dependencies.

If a bundled dependency is present in the source distribution, the license must be a Category A license [3].

Attention: The module flink-runtime-web bundles JavaScript dependencies for the Flink UI which are part of the source distribution. Run bower list in the flink-runtime-web/web-dashboard to get a list of all used JavaScript dependencies.

...

Dependencies that have been copied into the distribution (such as jar files) should be considered bundled, in addition to the bundled dependencies of the source archive.

The LICENSE and NOTICE (called NOTICE-binary) files for the binary distribution can be found in the project root directory.

Similar to the source distribution, the root directory contains a licenses-binary directory which contains all LICENSE files of the bundled dependencies by the binary distribution.

When assembling the binary distribution (done by the maven-assembly-plugin in flink-dist), Maven will rename the NOTICE-binary file into NOTICE, licenses-binary into licenses and include the LICENSE file in the resulting assembly. Therefore, make sure that NOTICE-binary and licenses-binary contain all the license information of the binary release's bundled dependencies. (further instructions)

If a bundled binary dependency is present in a binary distribution, the license must be a Category A [3] or Category B [4] license.

All licensing files for the binary distribution must be placed into the root directory of the distribution.

The LICENSE file for the binary distribution can be found in the project root directory, and is copied during the build process into the binary distribution.

The licenses directory and NOTICE file for the binary distribution are generated during the release process, see tools/releasing/create_binary_release.sh If a bundled dependency is present in a binary distribution, the license must be a Category A [3] or Category B [4] license .

Maven artifacts

Most modules of Flink generate at least one Maven artifact (jars) that is uploaded and available for download on Maven Central.  These artifacts follow a similar pattern to the binary distribution except that the bundled software is typically restricted to the individual module.

All licensing files for maven artifacts must be placed into the META-INF directory of the jar.

The LICENSE file for all Maven artifacts is pulled in automatically via the apache-jar-resource-bundle defined in the Apache parent pom.

...

For Maven artifacts that DO bundle (usually through the maven-shade-plugin) any dependencies the NOTICE file can be found in the src/main/resources/META-INF directory of the respective module. If the module does not have a NOTICE file yet, then add one under src/main/resources/META-INF.

License files

Requirements for LICENSE file

All modules The LICENSE file must contain a standard Apache License file, i.e., the one pulled in from the apache-jar-resource-bundle.

Special cases:

• LICENSE.txt files pulled in through dependencies should be excluded from the distribution.

Requirements for NOTICE file

All bundled dependencies must be listed in NOTICE, grouped by license type, using standard maven syntax (groupId:artifactId:[classifier:]version).

• This DOES include ASLv2 dependencies (for maintainability reasons, as it simplifies matching of entries in the dependency-tree and NOTICE files)
  
  

  Expand
  title Example
  flink-mesos Copyright 2014-2018 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). This project bundles the following dependencies under the Apache Software License 2.0. (http://www.apache.org/licenses/LICENSE-2.0.txt) - com.netflix.fenzo:fenzo-core:0.10.1 - org.apache.mesos:mesos:1.0.1 - com.fasterxml.jackson.core:jackson-annotations:2.4.0 - com.fasterxml.jackson.core:jackson-core:2.4.5 - com.fasterxml.jackson.core:jackson-databind:2.4.5 This project bundles the following dependencies under the BSD license. See bundled license files for details. - com.google.protobuf:protobuf-java:2.6.2

For all dependencies not using the ASLv2, include the license text in a LICENSE.<dependency_name> file under the src/main/resources/META-INF/licenses directory of the respective module.

Special cases:

• NOTICE.txt files pulled in through dependencies should be excluded from the distribution and manually merged into the dependents NOTICE file.
• netty:3.X.Y includes a NOTICE files that cover more than they should. These should be excluded and relevant parts merged into the dependents NOTICE file.

Requirements for licenses directory

For all dependencies not using the ASLv2, include the license text in a LICENSE.<dependency_name> file under the src/main/resources/META-INF/licenses directory of the respective module.

Special cases:

• netty:3.X.Y includes a license directory under META-INF. Jars that bundle this directory along with a LICENSE file under META-INF cannot be unpacked on case-insensitive filesystems. This directory should be excluded and merged into the dependents licenses directory.

...

Attention: flink-dist bundles various Flink modules and their transitive dependencies. Any transitive dependency that is pulled in this way which is not included (and thus accounted for!) in a bundled Flink module MUST be accounted for in flink-dist.

FAQ: What does all this mean in practice?

...

In order to generate a NOTICE file and licenses directory for all contained jars in the binary distribution, you can use tools/releasing/collect_license_files.sh. The script takes the directory of the binary distribution and an output directory as arguments. It extracts the NOTICE files and licenses directories from all jars contained in the specified directory and merges them. This can be used as a skeleton for creating the NOTICE-binary file and licenses-binary directory. The skeleton needs to be enriched with license information of differently bundled dependencies.

The merged NOTICE file might contain duplicate entries which should be removed. For example, flink-s3-fs-hadoop and flink-s3-fs-presto both pull in a NOTICE file from the same Hadoop version.

Attention: Example jars are also included in the binary distribution. They originate from flink-examples/flink-examples-batch and flink-examples/flink-examples-streaming. The pulled in dependencies from the example jars need to be accounted for in NOTICE-binary and licenses-binary.

...

I'm changing a Flink dependency, what do I need to do with the LICENSE and NOTICE files?

• If you are adding a new dependency, make sure it is licensed in category A or B [3]
• If you are modifying an existing dependency, make sure the license of the dependency is still the same
• Everything below applies to the changed dependency version AND new or updated transitive dependencies
  • If you are changing a dependency in a module that does not release a "fat jar" / shaded jar to Maven central, there are no additional checks needed for that module. Downstream modules (such as flink-dist) might still have version changes that need attention.
  • If the module is releasing a shaded jar, you need to update the src/main/resources/META-INF/NOTICE file: Run maven, check the output of the shade plugin. For each "Including groupId:artifactId:.. into shaded jar" line, there needs to be an entry in this file (see details above!)
    • Group dependencies by license
    • mention the version
    • only include dependencies in the NOTICE file that are really there.
    • using "mvn dependency:tree" or "mvn project-info-reports:dependencies" (see "target/site/dependencies.html" of that module) are a best practice for analyzing a module's dependencies
  • If your module is bundled in flink-dist (or some other binary), the same rules as for shaded jars apply.

I'm copying code from another project, StackOverflow or somewhere else. What do I need to do?

Make sure it is licensed with a category A license [3], add a comment in the code that it is copied somewhere. Also specifically ask for a review of this in the pull request.
You can not copy code from StackOverflow or the internet into Apache Flink.

I'm verifying a Flink release. What do I need to do?

• You need to ensure that all source and binaries Flink is distributing have been developed by Flink, or under a compatible license, and that we are fulfilling all requirements of these licenses. It is impractical to manually check all dependencies, code and binaries for every release.
• We recommend:
  • Checking all pom.xml changes between the release candidate and the last release, and then checking if the NOTICE files have been touched (if necessary because of shading)
  • Checking the NOTICE file contents if possible. For a specific module whose dependency has changed in this release,
    • By using "mvn package -DskipTests -pl <module> | grep -i Including", you could filter the output like "[INFO] Excluding commons-codec:commons-codec:jar:1.15 from the shaded jar." . 
    • From the list you could get the list of dependencies get shaded in. For each dependency, sometimes they'll include the used license in the pom.xml files and you could find the file under maven local repositories of the given dependency. If that is not true, you could always find the type of license in maven repo like https://mvnrepository.com/artifact/com.google.api-client/google-api-client-jackson2/1.32.2
  • Checking all (or a sample of) jars in the staging repository and Flink distribution: Are shaded or bundled jars mentioned in the NOTICE file?
    • check using "jar tf <jar file> | grep" or "jar tf <jar file> | less"
  • Checking all (or a sample of, or changes of) non-Java-source-code files (such as build setup, documentation, javascript, ...).
    • Their license needs to be mentioned properly
    • requirements from their licenses need to be fulfilled (NOTICE file forwards, copyright owner mentions, ...)
    • we need to make sure we did everything correctly with copied code

[1] http://apache.org/legal/src-headers.html#headers
[2] http://apache.org/legal/src-headers.html#3party
[3] http://www.apache.org/legal/resolved.html#category-a
[4] http://www.apache.org/legal/resolved.html#category-b