In the 4th (2019-10-19) and 5th (2020-01-29) donations, Oracle transferred the code of the following parts of NetBeans 8.2 to the Apache Software Foundation:

  • 4th Donation - Some parts of the C/C++ "cnd" cluster
  • 4th Donation - The Dark Look & Feel Module
  • 5th Donation - Dlight Modules
  • 5th Donation - Remote access support for git, mercurial, subversion and a remote file system
  • (This list is not exhaustive)

Since there were dependencies between the 4th and 5th donations, we were waiting for the 5th donation to be finalized before processing the code from the 4th donation.

Analysis of the CND cluster

The so called "cnd" cluster adds support for the C/C++/Fortran/Assembler languages, for "Makefile" based NetBeans projects and more.

The source code of the 4th and 5th donations was imported into the "cnd" branch of Apache NetBeans (see https://github.com/apache/netbeans/tree/cnd) in order to review it before merging it with the master branch of Apache Netbeans.

The analysis of the source code revealed the following issues:

Issue #1: No source code for AntlrV2 modified compiler and runtime

For historical reasons, the NetBeans 8.2 "cnd" cluster relies on a variation of the Antlr 2.X tool for generating lexers and parsers, in the so called "org.netbeans.modules.cnd.antlr" module.

The source code of this module was not donated to the ASF by Oracle, and instead it was suggested to use the binary version of the NetBeans 8.2 module, under the CDDL/GPLv2 license, in a new "libs.antlr.cnd" module that did not exist in NetBeans 8.2 (see https://github.com/apache/netbeans/blob/308b080ac972e8cdea16c97d1e16c2156cb6bf59/cnd/libs.antlr.cnd/nbproject/project.properties#L23).

Further analysis showed that the licenses of Antlr V2 itself had been problematic in the past:

Unfortunately, v2's open-source license was unclear and so projects such as Eclipse could not include v3 because of its dependency on v2. In the end, Sam Harwell converted all of the v2 grammars into v3 so that v3 was written in itself. Because v3 has a very clean BSD license, the Eclipse project okayed for inclusion in that project in the summer of 2011.

(Terrence Parr, Antlr V2 Author, "Why do we need ANTLR V4?" at https://github.com/antlr/antlr4/blob/master/doc/faq/general.md

The "org.netbeans.modules.cnd.antlr" module contained both

  • a variation of the AntlrV2 compiler (that generates Java code from Antlr v2 modified ".g" grammar specifications). This is used in modules "cnd.apt", "cnd.asm", "and "cnd.modelimpl" to compile the grammars for the C preprocessor, C, C++ and Fortran lexers, and added a build-time dependency  (see, for instance, https://github.com/apache/netbeans/blob/308b080ac972e8cdea16c97d1e16c2156cb6bf59/cnd/cnd.apt/build.xml#L34 for an example of how the variation of the AntlrV2 compiler is used to compile grammars).
  • as well as the runtime code supporting those compiled grammars, living under the "org.netbeans.modules.cnd.antlr.*" package, which is used in the modules "cnd.apt", "cnd.asm", "cnd.model.jclank.bridge" and "cnd.modelimpl"

In order to circumvent the dependency with this module in Apache NetBeans we explored different alternatives:

  1. Recreate the AntlrV2 compiler and runtime, that proved useless since the AntlrV2 code itself had licensing problems (see the comment by Terrence Parr above), and we don't know what the improvements to this compiler were done in the "org.netbeans.modules.cnd.antlr" compiler.
  2. Migrate the AntlrV2 compiler and runtime to the more modern Antlr V4 runtime. This path was explored and promised interesting performance improvements (see https://github.com/vieiro/cnd.nextapt ) but it would require difficult mainteinance of the grammars, and a refactoring of many modules in the CND cluster to support the new runtime. Adapting the Antlr V4 runtime to look like the Antlr V2 runtime with minimum impact could be difficult, because Antlr V4 uses either Parser Listeners or Parse Tree Visitors to build the AST, which is a major difference with Antlr V3 (and V2) ASTs.
  3. Use the clank library (https://github.com/java-port/clank) . This library is a pure Java port of LLVM39 that was presented in the "2017 EuroLLVM Developers' Meeting" and was in the process of being used in NetBeans 8.2, but was not clearly adopted. An analysis of the project showed the following issues:
    1. Licensing: the project had no clear license. In April, 2021 the Apache PMC contacted the authors and they clearly stated that the library was covered by the LLVM license (see https://github.com/java-port/clank/blob/master/modules/org.clang.analysis/src/org/clang/analysis/analyses/AnalysisClangGlobals.java) and some other parts had different licenses.
    2. Build instructions: There're no clear build instructions in the project.
    3. Support for newest LLVM versions: We don't know how to port the project to a more modern version of LLVM.
  4. Use the "libclang" https://clang.llvm.org/doxygen/group__CINDEX.html library inside NetBeans. By creating some JNI wrappers that could add support to syntax highlighting, code completion and more. This experiment is at https://github.com/vieiro/netbeans/tree/cnd-libclang-12.4/cnd/cnd.clang . Some analysis showed that "libclang" is not the proper approach if one wants "full control over the Clang AST" (https://clang.llvm.org/docs/Tooling.html), so this path was discarded.
  5. Use "sulong" in GraalVM (https://github.com/oracle/graal/tree/master/sulong) that can execute clang. This was proposed in the Apache NetBeans mailing list.
  6. Reuse some of the modules in the "cnd" cluster to add support for Makefile based projects (and more), and integrate them in the "cpplite" branch, that uses LSP servers (such as clangd) to add support for code completion, syntax highlighting and more. This is a work in progress. An initial version is athttps://github.com/vieiro/netbeans/tree/cnd-small and the list of modules to recover from the CND cluster is at https://github.com/vieiro/netbeans/blob/57d9c16bcf5532c6c9c9a75f9a69ad1895f10261/nbbuild/cluster.properties#L1071 

Issue #2: Two different parsers (C++ model implementations) in NetBeans 8.2 and missing tests

It seems that the NetBeans 8.2 "cnd" cluster supported two different parser implementations (and hence C++ "model implementations") of the "cnd.api.model" API (https://github.com/apache/netbeans/tree/cnd/cnd/cnd.api.model)

One could switch between the different parser implementations using a system property (see https://github.com/apache/netbeans/blob/308b080ac972e8cdea16c97d1e16c2156cb6bf59/cnd/cnd.modelimpl/run.sh#L48).

Issue #3: "NetBeans fs-server has CDDL License"

The 5th donation document stated that the "NetBeans FS-Server" was covered by the CDDL license, and so could not be relicensed to the Apache License by the ASF. This was solved in Unable to render Jira issues macro, execution error. NETBEANS-5634.

Some further research showed that this "fs-server" is a server-side native component (executable) that improved performance of the "Remote Filesystem" feature in CND. It could safely be removed from Apache NetBeans, at the cost of a reduced performance.

Issue #4: Native libraries and executables

Some modules in "cnd" rely on platform-specific libraries/executables that have to be compiled in different platforms. This includes Oracle Solaris 11, but ASF Infra has no current support for compiling binaries for this platform.

Summary:

The 4th and 5th donations of the NetBeans 8.2 source by Oracle to the ASF contain the "cnd" cluster, and presented some issues that have either been resolved or must still be addressed by the Apache NetBeans Team.

Several workarounds have been proposed and explored to circumvent these issues, with different complexity and mainteinance costs.

See cnd integration proposal: cnd-small for a proposal to integrate some of the functionalities of cnd in Apache NetBeans.



  • No labels