You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 32 Next »

Historically NetBeans project relies on (nb-)javac  for Java editing features, i.e., parsing and lexing, for features such as syntax coloring, code completion, refactorings, and the like. This has some positive aspects, but also some downsides. Originally nb-javac  was maintained in a separate repository: hg.netbeans.org/main/nb-javac - Since version 15, it is hosted by Arvind's team on GitHub: https://github.com/oracle/nb-javac - "nb-javac" consists of two libraries ( nb-javac-api.jar  and nb-javac-impl.jar ) located in "extra" cluster, see: Overview: NetBeans Structure.

Pros:

  • Adopting latest Java language features is simple(r)
  • Errors, hints, warnings in the editor match exactly the command line build
  • Close co-operation with JDK language team

Cons:

  • Distributing GPLv2+CPEx licensed component in Apache software is hard

    • it would be way easier to use plain javac  from a JDK

    • nb-javac  has to be downloaded by end-user on demand via autoupdate

  • On demand download is problematic

  • Testing matrix is complicated

    • each supported JDK needs to be tested twice - with nb-javac  and without nb-javac 

  • Every bug/problem one needs to know whether nb-javac  was or wasn't in use

  • nb-javac  is a fork of JDK's javac

    • nobody likes forks

    • ironically Arvind's team is part of JDK organization - e.g. it maintains own fork of JDK's javac 

Eliminating the need for nb-javac

Clearly there are numerous drawbacks and Apache NetBeans needs a way out. Let's get rid of nb-javac  as we know it. Let's replace it with JDK's own javac ! However there are some problems...

  •  javac  in JDK15 isn't good enough
  • compile on save doesn't work
  • re-compilation of a single method doesn't work
  • runs out of memory more often than `nb-javac`.

Before NetBeans can really get rid of nb-javac , the `javac` in JDK is needs to be good enough.

Using JDK 17 javac Instead

Let's now assume JDK17 offers good enough javac , now NetBeans can suggest people to use JDK17 when using Apache NetBeans IDE

  • not a big problem, JDK17 is LTS, but then?
  • if people wanted to use language features of JDK19, they'd have to run on 19!
  • that's not what competition does - they support latest language features running on JDK11 LTS or even JDK8 LTS

Requiring to execute the IDE on latest JDK to try latest features is serious disadvantage compared to competitors IDEs, but possibly the story may end here and it might even be a good enough story for Apache NetBeans IDE. However...

Automatically Generating nb-javac

However, I don't find the restriction of latest JDK satisfying. It is not good enough story yet. There are parties that want to run on the IDE on some Java LTS version and still support the latest Java features. To address their needs let's take JDK17's javac  and let run it on JDK8! Of course, there are issues:

  • latest javac  is written in the language syntax of modern Java
    • such syntax cannot be compiled to JDK8 bytecode with `javac`
  • latest javac  is using APIs not available on JDK8
    • one needs to rewrite these calls to some older APIs
    • the behavior needs to be tested to remain the same

The great revelation is that both these problems can be solved with existing Apache NetBeans tools! Rather than maintaining manual patches like nb-javac  does, let's write advanced refactoring rules and apply them automatically. For example Optional.isEmpty()  method has been added in JDK11. Let's add following rule:

$1.isEmpty() :: $1 instanceof java.util.Optional
=>
!$1.isPresent()
;;

That automatically rewrites all occurrences of optional.isEmpty()  to !optional.isPresent()  and that is going to compile on JDK8. Few more (~30) rules like this and the javac  is almost ready to run on JDK8! Run few tests to verify the behavior remains the same after the automatic transformation and that's all. People can use Apache NetBeans IDE with javac  from the latest JDK or they can use the automatic port of the same code running on JDK8. Ideally the behavior shall be identical. No more questions: Are you using nb-javac or not? No more duplicated testing matrix.

The PR-12 implements here-in proposed conversion.

What does nb-javac do that's different to vanilla javac?

A few years back, the NetBeans team wrote a page describing what nb-javac does in addition to/differently to vanilla javac. Some of the things listed above are not part of nb-javac anymore (e.g., cancelling and some part of the error recovery paragraph) but it still gives a reasonable overview and is good to get an idea what kinds of things nb-javac does. Here is the copy ready to be made up to date:

Recompleting Symbols from Sources

Consider the following usecase: let there be two classes, A and B available both in source code and (up-to-date) class files. These classes are interdependent (each refers to the other). Let there be a refactoring, that needs to work over both these files. The refactoring needs Trees (to access method bodies, to get offsets, etc), and these trees need to be attributed.

There are the following ways for the refactoring to work:

  • parse the source codes in two instances of javac (each time completing the other one from the class files). This does not require much memory (only needs memory to parse one file), but is quite slow.
  • create one instance of javac and parse both files in that instance - fast, but requires a memory to parse both/all the files, which is not reasonable in the IDE.
  • create an instance of javac, parse A from sources, complete B from the classfile. After processing of file A is done, check the available memory and either parse the source code for B in the same instance of javac (if there is enough memory), or discard the old javac and create a new one for B. This allows to use the available memory to speed up the refactoring, but allows to perform it (slowly) in memory required to process one file. Unfortunately, the standard javac does not allow to recomplete a Symbol from sources (i.e. the Symbol was originally created from a class file and now we parse and attribute a file containing the same Symbol).

Recompleting of Symbols is also used:

  • when an API client asks for Tree for Symbol that was loaded from classfiles
  • when Symbol originates from classfile without parameter names (debugging information), and parameter names are required. Source is parsed to fill in the parameter names.

The ability to recomplete a Symbol from source file is one of the most important part of the NetBeans fork/patch.


Stable annonymous innerclass numbers

When a Scope is created, the corresponding Tree is duplicated and attributed. This may lead into incorrect Symbols created for anonymous innerclasses. It is necessary to ensure that the anonymous innerclass numbers will match the numbers that would be produced by a batch compiler. This needs to be ensured even in case when the Scope is created before the tree is attributed.


Error Recovery

While editing, the code in the editor contains compilation errors almost all the time. It is therefore unacceptable to loose e.g. code completion only because of a missing semicolon, or because of an (unrelated) unresolvable symbol.

The list of current changes includes (not an exhaustive list):

  • JavacParser:
    • a variable declaration tree is produced even for cases where variable declaration is not allowed, e.g. in then/else section of if. The tree is wrapped in an erroneous tree and an error is reported. Trees produced before this change were too confusing for some hints (NB bug #192561). Relates to parseStatement and parseBlockStatement(s).
    • parseCompilationUnit is modified to handle multiple package clauses
  • Attr:
    • in "return <expr>" <expr> is always attributed, even for initializers and methods with return type void
    • the last-resort symbol filler PostAttrAnalyzer is modified to dive into erroneous trees (NB bug #152334)
  • Annotate:
    • modified to always attribute the annotation's attribute's values, even if the annotation itself is unresolvable (i.e. "@Undefined(@Another)" will attribute also @Another).


Cancellability

NetBeans need to be able to stop javac processing even inside one phase (parse/member enter/attribute). This is used e.g. when the user types into the editor, and the current instance of javac is already processing an obsolette source code. Then, there is no point in wasting time and memory in continuing the processing of the file.


Inferring Binary Names

Consider file A.java, containing classes B and C, and file D.java, referencing B and C. If file D.java is being parsed (and class file for B and C do not exist yet), the javac is currently unable to locate file A.java. Yet, the IDE may have the information about the content of the A.java file. So, the problem is how to pass this information into the javac. ClassNamesForFileOraculum is currently used for this.


Support for Reparsing Method Bodies

If the user changes are contained only inside one method body, it is desirable to reparse only the body of the one method. This leads into faster reparse times and less garbage on the heap. The NetBeans' fork/patch contains support for reparsing methods bodies.


Repair

In the NetBeans' fork/patch, there is a new phase, Repair, running after Flow. For source code with compilation errors, this phase converts the "uncompilable" trees into "compilable" trees.

New attributes are added to the classfiles to keep:

  • error types (stripped from the trees during repair to obtain valid trees)
  • parameters names, without depending on debug information
  • annotations with RetentionPolicy.SOURCE


Annotation Processing

In vanilla javac, new Symbols are created for classes/methods/fields in the sources for each annotation processing round. This is rewritten in the NetBeans' fork/patch to use the symbol recompleting. Symbols, once created, are used in all rounds on annotation processing and also for the final compilation. Annotation processing is also supported when completing symbols from sources. Exceptions thrown by annotation processors are logged, but do not stop the compilation.


IDE Mode

A special option, "ideMode", has been introduced by the NetBean's fork/patch to improve javac behavior in the following cases:

  • when there is no java.lang package (i.e. no platform), vanilla javac stops with an error. In IDE mode, the compiler handles the situation more gracefully. - !May not be needed anymore.
  • when loading 1.5+ classfile with sourcelevel <=1.4, vanilla javac throws away any 1.5 information stored in the classfile. In IDE mode, this information is preserved.


Miscallenous

  • small extensions to allow parsing and attributing a "standalone" statement, expression, etc. in the given context.
  • small extensions to JavacTaskImpl to allow per-file per-phase parsing of multiple sources in the same instance of javac
  • access restrictions loosened for a lot methods/classes (private->protected, etc.)
  • inlineTags cache for ParamTagImpl and ThrowsTagImpl
  • string folding is disabled (by "disableStringFolding" option) to so that concatenated strings are represented by full trees
  • ToolProvider loads classes from context classloader
  • TreeInfo.symbolFor works all trees that refer to a symbol


  • No labels