Release Retrospective to correct Apache MXNet source licenses in post MXNet 1.0 release.
A. Problem Description
Links to email threads and github issues that record the existing issues during various voting periods -
- https://github.com/apache/incubator-mxnet/issues/8913
- Email from Henri Yandell
- Email from Justin McClean
- Results of Apache RAT are in this text file - YetToFix.txt (Some of these are now fixed)
B. Apache Licensing Policies
- LICENSE file requirements
- LICENSE requirements for distribution artifacts with multiple licenses
- NOTICE file requirements (Check Copyright year) - see also https://www.apache.org/legal/src-headers.html
- Apache Legal
- Acceptable and Unacceptable Dependency Licenses
C. Open Pull Requests to fix these Issues
...
- PR to fix the top level LICENSE file - This one has been merged but a review from mentors/PMC would still be helpful
- bayard@: The last paragraph looks suspect. I doubt we've taken code from the BSD project.
- PR to update the PR to update the license_header.py file and add apache licenses where missing - Merged.
PR to add license for dockerfiles - Merged
PR to fix License headers in specific files - Part 1 - Merged
Open Issues/Questions/Doubts/Concerns (Help Requested!)
...
- Part 2 - Merged
- PR for some fixes based on Apache RAT failures - Part 3 - Merged
- PR for some fixes based on Apache RAT failures - Part 4 - Merged
- PR for adding headers to the perl-package - Merged
- PR for adding headers to the docs folder - Merged
- PR for some fixes based on Marco's email - Merged
- PR to fix the website - Merged
- PR to add copyrights for third party licenses to license file - Merged
- PR to add static-dependencies licenses - Merged
D. Open Issues/Questions/Doubts/Concerns (Help Requested!)
Question 1: What all can be excluded from RAT checks? Currently the following are being excluded from the RAT checks. Is this ok ?
...
these file types - *.xml ; *.css ; *.txt; *.md ; \..* ; *.ipynb ; *.html ; *.js ; *.json ; *.svg; *.config; *.names
...
Submodules - 3rdparty/* ; dmlc-core/* ; mshadow/* ; dmlc-core/* ; dlpack/* ; nnvm/* ; ps-lite/*
...
R-package since it is not released - R-package/*
...
src/operator/mkl/*
...
Ignore all Dockerfiles - docker/*; Dockerfile* ; docker_multiarch/*
...
perl-package/*
...
contrib/*
...
__init__.py files
...
docs/*
...
Name of File/Folder Excluded from RAT Check | Reason for Ignoring | Concerns | |
---|---|---|---|
1 | These File Types: *.css ; *.txt; \..* ; *.ipynb ; *.html ; *.json ; *.svg; *.names; *.csv | ||
2 | Submodules - 3rdparty/* | None | |
3 | R-package | Not a part of MXNet Release | None |
4 | Ignore all Dockerfiles - docker/*; Dockerfile* ; docker_multiarch/* | Dockerfiles cant have license | |
5 | __init__.py files | These files contain no text | None |
6 | theme.conf | A header might affect website. | Only one script here without a license - docs/_static/mxnet-theme/theme.conf |
7 | This file - src/operator/nn/pool.h | It was decided that this file should not have an Apache License and it was removed here - PR 9170 | None. But can be verified again. |
8 | This file - src/operator/special_functions-inl.h |
...
None. But can be verified again. | |
9 | example/rcnn/rcnn/cython |
...
/* | This is licensed under MIT but RAT doesnt pick that |
...
. ASF should not be added | Should MIT license text be added explicitly? | ||
10 | This Dataset - example/gluon/tree_lstm/dataset.cPickle | This is a dataset | None. But can be verified again. |
11 | CODEOWNERS | text file |
Question 2: There are still 2 files with unknown licenses as per Apache RAT - SEE TABLE BELOW for this LIst
This list assumes above excluded folders and formats are acceptable.
S.No File Name
List of Issues based on Feedback
...
Comments 1.CODEOWNERSMoved to excludes 2.appveyor.ymlAdded ASF header - Open PR 3.readthedocs.ymlDoes this need an Apache License ?4. snap.python Can I add Apache License here? 5.snapcraft.yamlCan I add Apache License here?6. python/mxnet/cython/base.pyi File format --? 7.tests/ci_build/pylintrcAdded ASF header - Open PR
Question 3: Issue 23 and Issue 37 in table below
This Folder needs to be reviewed - src/operator/contrib/ctc_include
E. More Details about the Issues & their Status
No | Category | Problem | Source Files | Suggested by | Leads | Comments |
---|---|---|---|---|---|---|
1. | Source tree | * Move the various git submodules into third-party/ or similar so it's
simpler to see what is Apache original source when we review a release.
| submodules | Henri Yandell | Haibin Lin | |
2. |
|
|
|
|
Source
tree
* Move the various git submodules into third-party/ or similar so it's
simpler to see what is Apache original source when we review a release.
submodules
Henri Yandell
Haibin Lin
2.
Copyright
* Deal with the Copyright statements in perl-package (copyright holder has approved doing this)
perl-package
Henri Yandell
3.
Formatting
* Lots of whitespace on end of NOTICE
NOTICE
Henri Yandell
Meghna Baijal
PR Merged
https://github.com/apache/incubator-mxnet/pull/9459
4.
Comments
* Comment added to CODEOWNERS to explain the file so we don't cause
community problems
CODEOWNERS
Henri Yandell
5.
LICENSE
* There was a suggestion to simplify the LICENSE to not explicitly list which packages are under each license. Something to consider.
LICENSE
Henri Yandell
Meghna Baijal
PR - https://
3. |
LICENSE
* Update the Paths to license files once submodules are moved
|
|
| Henri Yandell |
Meghna Baijal |
Automated Check
* Update the license_header.py script instead of manual exclusions
tools/license_header.py
ISSUES IN SPECIFIC FILES
6.
Specific
Files
* Resolve License Header - if you follow the URL mentioned in the file it is unclear
if the code came form that site or was written for the project by the author
of that site.
example/image-classification/predict-cpp/image-classification-predict.cc
Justin McClean
7.
Specific
Files
PR Merged | ||||||
4. | Comments | * Comment added to CODEOWNERS to explain the file so we don't cause
community problems | CODEOWNERS | Henri Yandell | Steffen Rochel | |
5. | LICENSE | * There was a suggestion to simplify the LICENSE to not explicitly list which packages are under each license. Something to consider. | LICENSE | Henri Yandell | Meghna Baijal | If this is done, it would resolve points 8-11 and 13-19 of this wiki |
LICENSE | * Update the Paths to license files once submodules are moved | LICENSE | Meghna Baijal | Haibin Lin | ||
Automated Check | * Update the license_header.py script instead of manual exclusions | tools/license_header.py |
Meghna Baijal | Meghna |
Baijal | makefiles, |
ISSUES |
IN |
SPECIFIC |
Specific
FILES | ||||
6. |
by the matching string. Does not need a fix.
8.
Specific |
Files | * |
Resolve License Header - if you follow the URL mentioned in the file it is unclear if the code came form that site or was written for the project by the author of that site. | example/image-classification/predict-cpp/image-classification-predict.cc | Justin McClean | ||
7 |
nnvm/tvm/HalideIR/LICENSE
Justin McClean
. | Specific Files |
|
|
|
| Meghna Baijal |
Possibly RAT got confused by the matching string. Does not need a fix. | |
8 |
. | Specific Files |
|
|
|
|
| Meghna Baijal |
Added to MIT license |
in LICENSE |
9. | Specific Files |
Justin McClean | Meghna Baijal |
Files licenses under different licenses, added to all in the top level LICENSE - /incubator-mxnet/example/rcnn/LICENSE |
10 |
. | Specific Files |
perl-package/AI-NNVMCAPI/Makefile.PL
Justin McClean
Justin McClean | Meghna Baijal | Files licenses under different licenses, added to all in the top level LICENSE - /incubator-mxnet/example/rcnn/LICENSE | ||
11 |
. | Specific Files |
Justin McClean |
Meghna Baijal | Files licenses under different licenses, added to all in the top level LICENSE - /incubator-mxnet/example/rcnn/LICENSE | |||||
12. |
|
|
| Justin McClean | ||
13 |
. | Specific Files |
|
|
|
| Justin McClean |
Wrongly Named in the LICENSE file as Warp-CTC, corrected | |
14 |
. | Specific Files |
|
| Justin McClean |
Wrongly Named in the LICENSE file as Warp-CTC, corrected | |
15 |
. | Specific Files | * |
Add |
to |
LICENSE |
- |
BSD |
license |
nnvm/tvm/dmlc-core/cmake/Modules/FindCrypto.cmake | Justin McClean |
Meghna Baijal | Repeat again for the same file?? |
16 |
. | Specific Files |
|
|
|
|
|
| Justin McClean | Already there in LICENSE file as MS coco | |
17. | Specific Files |
|
| Justin McClean |
Already there in LICENSE | ||||
18. | Specific Files |
|
| Justin McClean |
Submodule, no edits
Submodule, no edits
Packages not named in LICENSE anymoreAdded to LICENSE as BSD | ||||
19. | Specific Files |
|
| Justin McClean |
CUB exists as BSD | ||||
20. | Specific Files | * Incorrect License Header - Has ASF header but is it BSD | prepare_mkl.sh | Justin McClean |
Does not say BSD, add? | ||||
21. | Specific Files | * Incorrect License Header - Has ASF header but is it BSD | src/operator/nn/im2col.h | Justin McClean |
Created PR, in review | ||||
22. |
| * Incorrect License Header - Has ASF header but is it BSD |
| Justin McClean |
First was fixed for 1.0.0 here - https://github.com/apache/incubator-mxnet/pull/9170 | ||||
23. | Specific Files | * Incorrect License Header - Has ASF header but is it BSD | src/operator/contrib/ctc_include/contrib/moderngpu/include/mgpuenums.h | Justin McClean |
License seems to be correct. No ASF header. | ||||
24. | Specific Files | * Incorrect License Header - Has ASF header but is it BSD | example/ssd/dataset/pycocotools/coco.py | Justin McClean |
Removed ASF header in PR | ||||
25. | Specific Files | * Incorrect License Header - Has ASF header but is it MIT | example/rcnn/rcnn/cython/setup.py | Justin McClean |
Removed ASF Header in PR | ||||
26. | Specific Files | * Incorrect License Header - Has ASF header but is it MIT | example/rcnn/rcnn/cython/nms_kernel.cu | Justin McClean |
Removed ASF Header in PR | ||||
27. |
|
|
| Justin McClean |
Repeat |
Specific Files | Resolve License: should this file get an apache license? (RAT thinks so but why does the script skip it?) | src/operator/special_functions-inl.h |
APACHE RAT CHECK FAILURES |
28. | RAT |
Failure | * |
Fix |
Submodules |
- |
RAT |
detected |
almost |
2000 |
files |
with |
unknown |
licenses |
in |
submodules | Submodules |
(nnvm, |
dlpack, |
3rdparty, |
ps-lite, |
Decision needed on how to handle submodule licenses
29.
RAT Failure
* Check Docs - RAT detected almost 200 files with unknown licenses in the Docs directory
/docs
30.
RAT Failure
* Fix dockerfiles without license headers
/docker
Meghna Baijal
31.
RAT Failure
* Fix docker_multiarch - unknown license header
/docker_multiarch
32.
RAT Failure
* Fix scala-package - unknown license
/scala-package
33.
RAT Failure
* Fix tools
/tools
34.
RAT Failure
* Fix tests
/tests
35.
RAT Failure
* Fix examples
/examples
36.
RAT Failure
* Check file types that can be ignored - .txt; .js, .md, .json, .yml, .*,
/
Meghna Baijal
37.
RAT Failure
* Fix ctc_include
src/operator/contrib/ctc_include
This file already has a license but Apache RAT adds the AL too!
mshadow) and R-package | Decision needed on how to handle submodule licenses | |||||
29. | RAT Failure | * Check Docs - RAT detected almost 200 files with unknown licenses in the Docs directory | /docs | Excluded for now | ||
30. | RAT Failure | * Fix dockerfiles without license headers | /docker | Meghna Baijal | Added a top level License.md to this folder | |
31. | RAT Failure | * Fix docker_multiarch - unknown license header | /docker_multiarch | Meghna Baijal | Added a top level License.md to this folder | |
37. | RAT Failure | * Fix ctc_include | src/operator/contrib/ctc_include | |||
F. Static Linked Dependencies in Release
This section lists all dependencies used in the releases of pip and maven. Frank Liu pointed out some of them are using BSD3, GPL and customized License which should be placed in the official releases.
Currently, the licenses were covered when we created the static-linked so file which can be found in this PR.
OpenCV depends on following libraries, and can build them on the fly:
- OpenBLAS
- zlib (ver 1.2.6)
- libjpeg-turbo (ver 1.5.90)
- libpng (ver 1.6.34)
- libtiff (ver 42 - 4.0.9)
- eigen (ver 3.3.4)
ps-lite dependencies (USE_DIST_KVSTORE):
- zeromq - 4.1.4
- protobuf 2.5.0
- lz4
- cityhash
USE_S3
- libcurl
- libssl-dev
...