...

...

• nifi: will have been broken up into nifi-api, nifi-framework, nifi-extensions, nifi-release
  • Alternatively: this repo could be reduced to a single pom module containing the org.apache.nifi:nifi pom that inherits from org.apache:apache and can be a parent pom for other projects.
    
    
• nifi-minifi: will become a new assembly of nifi-framework and nifi-extensions that lives in the nifi-release repo

Benefits of Multiple Repositories

Multiple git repositories are not required in order to have multiple Maven projects. An alternative is a single git source code repository with multiple projects in sub directories. This was considered. Pros and cons of both approaches were also discussed on the discussion thread related to this proposal. This section of the proposal serves as a summary of that discussion and the thinking behind why this proposal recommends multiple repositories at this time.

Advantages of single repository:

• Easier to make cross-project changes, especially for new comers. Also for some, it will be easier to review changes that impact multiple projects if they are in a single PR.
• Git history for combined/entire project in one place
• Single focal point for entire community to collaborate on and follow
• Less infra to manage (though potentially mitigated now that access to repos is managed by ASF LDAP groups)

Advantages of multiple repositories:

• Easier for Release Mangers as they are performing source releases of an entire repository.
• Quicker to build an entire repository. Easier to setup reliable, fast CI builds. In a multi-repo setup, builds will never span multiple projects as commits are, by definition, isolated to a single repo. Also avoids having to "build all project" or setting up CI tools such as Travis or Jenkins to scope builds based on contents of commits.
• Separate repos forces pull requests to stay scoped to a given component, making for overall smaller PRs. Easier to look at a repo and see what work/contributions are still open.
• Easier to understand the history for a single project or module in the git history, e.g., "show me all changes to this project since the last tag / release"
• Lower learning curve for making changes to a single project, as the overall code base is smaller.

At this time (though it is still being discussed), the multiple repository approach seems to be merited, especially based on it being the optimized approach for managing source code releases.

To mitigate the downsides of multiple repositories, the following is recommended:

• Additional documentation and guidance be prepared for developers and contributors, especially getting started guides that target new comers. For example, document the project and module structure to help newcomers navigate repositories.
• Setup a CI job somewhere that publishes a SNAPSHOT build of every project's main branch, and allow development to depend on SNAPSHOT versions of dependencies. This will enable work to continue without releasing stable versions of dependencies. The CI infrastructure and where to host SNAPSHOT artifacts has not been identified. It is possible that GitHub Actions could be setup to run the CI jobs to publish SNAPSHOT artifacts to the Apache Maven repository managed by ASF infra.

A Note About Jira

This proposal only impacts source code projects and repositories. At this time, no change is being suggested for our Jira project structure or issue tracking system. It is recommended that even under different source code projects and repositories, issues are still tracked in the existing Jira project (i.e., NIFI) and differentiated as to which source code project/repo/artifact they impact via fields on the issue, such as component, label, fix version, etc.

An Alternative: Single project in a mono-repo

There are downsides to multiple projects, regardless of single or multiple repository. The main downside is development. If one project depends on another, and changes need to be made in the dependency, then the dependency must be updated, build, and installed as a SNAPSHOT in order for work to continue. This can be avoided by structuring the project in the other extreme, so that every component is part of one maven project. In order for this approach to be considered, someone would need to do exploratory work to understand how we could significantly speed up builds or facilitate partial builds of sub-modules automatically based on which files have been changed in a commit to the mono-repo.

Proposed Approach

Phase 1: nifi

...

1. Create a mapping of existing modules in the nifi repository/project into where they will live in the new structure
2. Use a utility utilities such as git filter-branch or git-subtree to move existing modules while retaining as much revision history as possible
3. Update the pom files for migrated modules to contain their new parents, etc.
4. Each new repository/project will get a new assembly module that just packages that artifact so that releases can be done from that project repository
5. Write new assemblies in the nifi-release project/repository that produces something that closely matches the existing assembly
6. Get all builds and tests working, although this should be minimal as no maven module coordinates are changing.

...

1. One or more community volunteers will manually attempt the above steps, taking detailed notes of commands they use and changes that they need to make in order to get everything working, such as modifications to pom files.
2. From those notes, a script will be made that streamlines (ideally, fully automated) the restructuring the nifi repository into new repositories. This will enable us to repeatedly perform the restructuring from any given revision of the master main branch.
3. The restructuring automation script(s) and instructions for use will be distributed on the Apache NiFi dev@ mailing list. The NiFi dev community will be asked to verify the procedure and resulting output in a manner similar to a RC verification vote. The final output assembly of the restructured process should match the output assembly of the current nifi-assembly module from the 1.x line on master todaythe main branch today.
4. The automation process will go through as many reviews as needed until it passes a vote by the NiFi PMC.
5. A specified date and time will be chosen for the restructuring. Any large branches from master main should be merged before that date to avoid burdensome rebasing that will be necessary for any changes not merged before the restructuring.
6. On the agreed upon date and time, with the assistance of ASF Infra, the existing nifi repository will be frozen from modification. The restructuring script will be run and the resulting repositories will be added as top level Github repositories, again, with the help of ASF Infra.
7. Unmerged pull requests to the nifi repo that were under active development will need to be recreated as PRs to the new repositories. This responsibility will fall to the author of each PR. PRs will stay open to the old nifi repository for a time as a historical record, but only those that are recreated against the new repositories by their authors will be considered to be merged.
8. Contributions continue as normal, but from forks of the new repositories.
9. The first release after the restructuring will be a release from all repositories/projects in the order dictated by the dependency graph shown above. After that, releases will only need to be performed as needed from each repository/project.
   • NOTE: There should be a community discussion on how versioning and releasing of the new, smaller projects should be handled. That is, should we always release everything so that versions stay aligned, even if there are not changes to a particular component such as nifi-api, which changes very infrequently, or should we follow semantic versioning for each new project and allow them to advance and be released independently of each other and only as needed.

Phase 2: nifi-standard-

...

libs

Introduce nifi-commons standard-libs as a home for shared, top-level, reusable libraries of components across the new collection of projects

Steps

1. A new nifi-standard-commons libs project and repository is introduced.
2. Similar code across nifi-framework and nifi-registry, and possibly other projects such as nifi-toolkit and nifi-extensions, is rewritten as generic, reusable libraries in sub-modules of nifi-standard-commonslibs, e.g., nifi-standard-commonslibs-security could contain shared authentication and authorization APIs and provider implementations.
3. Code from nifi-framework, nif-registry, etc. is removed and replaced with library implementations from nifi-standard-commons libs modules.

Phase 3: nifi-minifi

Migrate code modules from nifi-minifi into nifi-framework, nifi-extensions, and (possibly) nifi-standard-commonslibs. Migrate the nifi-minifi assembly into nifi-release.

...

Once the initial restructuring and decomposition of the nifi project/repo is complete, there may be opportunities for farther steps such as breaking nifi-extensions into smaller projects, or dropping support for some legacy extensions.

A Note About Jira

This proposal only impacts source code projects and repositories. At this time, no change is being suggested for our Jira project structure or issue tracking system. It is recommended that even under different source code projects and repositories, issues are still tracked in the existing Jira project (i.e., NIFI) and differentiated as to which source code project/repo/artifact they impact via fields on the issue, such as component, label, fix version, etc.