An experiment has been conducted to show a possible split of the main NetBeans repository into several smaller parts.

In the experiment, the sources were re-organized into the following structure:

  • clusters

    • platform
    • harness
    • ide
    • java
    • <etc>
  • ide
  • nbi
  • nbbuild

In this structure, there Would be a top-level repository (consisting of ide, nbi and nbbuild) and then a repository per cluster.

The code that does the splitting is here:
https://bitbucket.org/jlahoda/split-nb-repo
this is the actualy splitting code:
https://bitbucket.org/jlahoda/split-nb-repo/src/55c9cf672644d61d69cf7a81b8c9e8622d8d988a/src/split/nb/repo/SplitNbRepo.java?at=default&fileviewer=file-view-default
And this patch needs to be applied after splitting:
https://bitbucket.org/jlahoda/split-nb-repo/src/55c9cf672644d61d69cf7a81b8c9e8622d8d988a/adjustments.patch?at=default&fileviewer=file-view-default


After this, a full NetBeans build should be possible, and testing of java.hints should run.

To try this yourself, you'll need:

  • NetBeans 8.2 installation, in directory $NETBEANS_HOME
  • a clean checkout of NetBeans main repository, tip: b7050e57a3c2, in directory $NETBEANS_REPO
  • the target directory (empty), in $NETBEANS_TARGET

Then:

  1. clone the above repository, cd into it

  2. build the code like this:

    ant -Dvar.netbeans.home=$NETBEANS_HOME jar

  3. run the produced jar like this:

    java -cp $NETBEANS_HOME/platform/lib/org-openide-util.jar:$NETBEANS_HOME/ide/modules/org-netbeans-modules-project-ant.jar:$NETBEANS_HOME/platform/core/org-openide-filesystems.jar:$NETBEANS_HOME/platform/lib/org-openide-util-lookup.jar:$NETBEANS_HOME/ide/modules/org-netbeans-modules-projectapi.jar:$NETBEANS_HOME/ide/modules/org-netbeans-modules-projectapi-nb.jar:dist/split-nb-repo.jar split.nb.repo.SplitNbRepo $NETBEANS_REPO $NETBEANS_TARGET

  4. apply the adjustments patch:

    cd $NETBEANS_TARGET

    patch -p1 -i $SPLIT_NB_REPO/adjustments.patch

Please note that this was only an experiment: many things would need fixing/improvements (see an incomplete list below).

Notes on the experiment:

  • does not touch modules that are not in the standard distribution or contrib
  • if all modules (including contrib) would be re-organized under clusters, paths inside the scripts could be simplified (i.e. clusters/platform/openide.util could become platform/openide.util inside the scripts)
  • in the experiment, there is still a single global nbbuild/cluster.properties - would be better to split and place parts into individual repositories
  • the directories under "clusters" don't necessarily need to map 1-1 to clusters: a single repository may contain code for multiple clusters. Like maybe ide and extide => ide, cnd and cndext => cnd, or even platform and harness => platform
  • developers working on a cluster would need to checkout at least repositories for all clusters their cluster requires, transitivelly. Improvements can be considered, possibly using:
    http://wiki.netbeans.org/DevFaqOrphanedNetBeansOrgModules
  • the experiment will create an unversioned copy of the sources - that would need to be changed to keep history, of course
  • the main intent here is to experiment with a way to keep repository sizes smaller while keeping history. One of the risks here is that having (too many) repositories is also not convenient. So, some consolidation of clusters into repositories seems to be in order. As a data point, some other big project is proposing to move from multiple smaller repositories to a single bigger repository (i.e. closer to the current NetBeans state):
    http://mail.openjdk.java.net/pipermail/jdk9-dev/2016-October/004987.html

 

  • No labels

2 Comments

  1. Wade Chandler

    On the JDK JEP they mentioned thoughts on core being one repo, but rejected that in favor of the whole, still, I think NB versus the JDK is quite different in some key ways with regard to scope:

    1. The JDK, other than script support, and Nashhorn, is about compiling and running Java byte code related things only. That scope is "fairly" static; outside of debug tools or things like Visual VM.
    2. NetBeans is about editing, executing, and working with various files and components related to various technologies plus a true RCP type core. This set is already large, and if all goes as hoped at Apache, should be growing further.
    3. My guess is Visual VM and other things will not come into that JDK source repo. I see this in the non-goals as an example "Adding the FX sources to the JDK forest is not part of the proposal.". Too me, these things probably had thoughts behind them much like the reasons we can think to split along cluster or other logical lines. I see the RCP or base itself much like the JDK, and the other pieces much like FX in this context.

     

    1. Jan Lahoda

      All I was trying to say is that having too many repositories is also troublesome. jdk9/dev forest on OpenJDK has in total 8 repositories (top-level, corba, hotspot, jaxp, jaxws, jdk, langtools, nashorn), and tasks like keeping them all up-to-date, or configuring a CI server are quite cumbersome.

      Basically, as having a too big repository may cause trouble, having too many smaller repositories also may cause trouble.

      For the NetBeans case, I think that if we choose to split the current repo, we probably should not create a repository per cluster. But rather coalescence reasonably related clusters into a single repository. Like ide, extide and maybe even dlight could be one repository, possibly platform and harness (maybe even have platform, harness and nbbuild in one repository), cnd and cndext could belong to one repo, some of the Java technologies could be also share repository (maybe java, javafx, profiler, apisupport? could be joined and javacard and mobility could be joined), etc.