...

...

• Easier for Release Mangers as they are performing source releases of an entire repository.
• Quicker to build an entire repository. Easier to setup reliable, fast CI builds. In a mutlimulti-repo setup, builds will never span multiple projects as commits are, by definition, isolated to a single repo. Also avoids having to "build all project" or setting up CI tools such as Travis or Jenkins to scope builds based on contents of commits.
• Separate repos forces pull requests to stay scoped to a given component, making for overall smaller PRs. Easier to look at a repo and see what work/contributions are still open.
• Easier to understand the history for a single project or module in the git history, e.g., "show me all changes to this project since the last tag / release"
• Lower learning curve for making changes to a single project, as the overall code base is smaller.

...

To mitigate the downsides of multiple repositories, it the following is recommended that additional :

• Additional documentation and guidance be prepared for developers and contributors, especially getting started guides that target new comers.

...

• For example, document the project and module structure to help newcomers navigate

...

• repositories.
• Setup a CI job somewhere that publishes a SNAPSHOT build of every project's main branch, and allow development to depend on SNAPSHOT versions of dependencies. This will enable work to continue without releasing stable versions of dependencies. The CI infrastructure and where to host SNAPSHOT artifacts has not been identified. It is possible that GitHub Actions could be setup to run the CI jobs to publish SNAPSHOT artifacts to the Apache Maven repository managed by ASF infra.

A Note About Jira

This proposal only impacts source code projects and repositories. At this time, no change is being suggested for our Jira project structure or issue tracking system. It is recommended that even under different source code projects and repositories, issues are still tracked in the existing Jira project (i.e., NIFI) and differentiated as to which source code project/repo/artifact they impact via fields on the issue, such as component, label, fix version, etc.

An Alternative: Single project in a mono-repo

There are downsides to multiple projects, regardless of single or multiple repository. The main downside is development. If one project depends on another, and changes need to be made in the dependency, then the dependency must be updated, build, and installed as a SNAPSHOT in order for work to continue. This can be avoided by structuring the project in the other extreme, so that every component is part of one maven project. In order for this approach to be considered, someone would need to do exploratory work to understand how we could significantly speed up builds or facilitate partial builds of sub-modules automatically based on which files have been changed in a commit to the mono-repo.

Proposed Approach

Phase 1: nifi

...

1. Create a mapping of existing modules in the nifi repository/project into where they will live in the new structure
2. Use a utility utilities such as git filter-branch or git-subtree to move existing modules while retaining as much revision history as possible
3. Update the pom files for migrated modules to contain their new parents, etc.
4. Each new repository/project will get a new assembly module that just packages that artifact so that releases can be done from that project repository
5. Write new assemblies in the nifi-release project/repository that produces something that closely matches the existing assembly
6. Get all builds and tests working, although this should be minimal as no maven module coordinates are changing.

...

1. One or more community volunteers will manually attempt the above steps, taking detailed notes of commands they use and changes that they need to make in order to get everything working, such as modifications to pom files.
2. From those notes, a script will be made that streamlines (ideally, fully automated) the restructuring the nifi repository into new repositories. This will enable us to repeatedly perform the restructuring from any given revision of the master main branch.
3. The restructuring automation script(s) and instructions for use will be distributed on the Apache NiFi dev@ mailing list. The NiFi dev community will be asked to verify the procedure and resulting output in a manner similar to a RC verification vote. The final output assembly of the restructured process should match the output assembly of the current nifi-assembly module from the 1.x line on master todaythe main branch today.
4. The automation process will go through as many reviews as needed until it passes a vote by the NiFi PMC.
5. A specified date and time will be chosen for the restructuring. Any large branches from master main should be merged before that date to avoid burdensome rebasing that will be necessary for any changes not merged before the restructuring.
6. On the agreed upon date and time, with the assistance of ASF Infra, the existing nifi repository will be frozen from modification. The restructuring script will be run and the resulting repositories will be added as top level Github repositories, again, with the help of ASF Infra.
7. Unmerged pull requests to the nifi repo that were under active development will need to be recreated as PRs to the new repositories. This responsibility will fall to the author of each PR. PRs will stay open to the old nifi repository for a time as a historical record, but only those that are recreated against the new repositories by their authors will be considered to be merged.
8. Contributions continue as normal, but from forks of the new repositories.
9. The first release after the restructuring will be a release from all repositories/projects in the order dictated by the dependency graph shown above. After that, releases will only need to be performed as needed from each repository/project.
   • NOTE: There should be a community discussion on how versioning and releasing of the new, smaller projects should be handled. That is, should we always release everything so that versions stay aligned, even if there are not changes to a particular component such as nifi-api, which changes very infrequently, or should we follow semantic versioning for each new project and allow them to advance and be released independently of each other and only as needed.

Phase 2: nifi-

...

standard-libs

Introduce nifi-commons standard-libs as a home for shared, top-level, reusable libraries of components across the new collection of projects

Steps

1. A new nifi-commons standard-libs project and repository is introduced.
2. Similar code across nifi-framework and nifi-registry, and possibly other projects such as nifi-toolkit and nifi-extensions, is rewritten as generic, reusable libraries in sub-modules of nifi-standard-commonslibs, e.g., nifi-standard-commonslibs-security could contain shared authentication and authorization APIs and provider implementations.
3. Code from nifi-framework, nif-registry, etc. is removed and replaced with library implementations from nifi-commons standard-libs modules.

Phase 3: nifi-minifi

Migrate code modules from nifi-minifi into nifi-framework, nifi-extensions, and (possibly) nifi-commonsstandard-libs. Migrate the nifi-minifi assembly into nifi-release.

...