DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
This wiki page is based on a discussion on the dev@nifi mailing list. The discussion thread that prompted this is here:
Table of Contents
Goals
- Faster build times for dev/testing/review/release
- Smaller release artifacts
- Maintain or increase the productivity of dev community (i.e., the new project structure should not be too burdensome for existing and future contributors to adopt)
- Facilitate code reuse across projects, such as nifi and nifi-registry, e.g., shared nifi security provider implementations for authentication and authorization.
- Position the NiFi project to more easily support and take advantage of future Java language features, such as modules.
- Better separation of concerns & better defined interfaces / touch points between project components
- Promote the idea that each module contains its own code and unit tests against public APIs such that it can be developed, tested, and released independently
- Cross-module integration tests can exist outside of each module, in their own module that tests the integration of two or more modules
Proposed End State
Diagram of Repositories
Description of Repositories
The following repositories are proposed. Each repository would contain a top-level maven project that inherits from the org.apache:apache parent pom.
| Project / Repo Name | New? | Dependencies | Description |
|---|---|---|---|
| nifi-api | new - will be extracted from nifi | Contains any Java APIs that need to be agreed upon by multiple top-level projects, such as nifi-framework and nifi-extensions | |
| nifi-framework | new - will be extracted from nifi | nifi-api, nifi-maven, nifi-fds, nifi-registry, nifi-commons | Contains the NiFi web server and front end without any unneeded flow extension bundles such as processors, controller services, and reporting tasks. |
| nifi-extensions | new - will be extracted from nifi | nifi-api, nifi-maven, nifi-commons | Extension bundles for NiFi, such as processors, controller services, and reporting tasks. |
| nifi-release | new - will be extracted from nifi | nifi-framework, nifi-extensions | Will be the new home for the nifi-assembly that produces convenience binaries such as Eventually, this repository will also take over the current role of the nifi-minifi project/repository by providing the |
| nifi-toolkit | new - will be extracted from nifi | nifi-framework, nifi-registry, nifi-commons | Contains the nifi-toolkit source code and assembly |
| nifi-maven | existing | nifi-api | Contains the Maven plugin used for packaging NARs |
| nifi-fds | existing | NiFi Flow Design system - reusable front end components used to create consistent front ends, such as the nifi and nifi-registry web UIs | |
| nifi-registry | existing | nifi-api, nifi-fds, nifi-commons | A web service for centralized storage and versioning of NiFi flows and extensions |
| nifi-minifi-cpp | existing | A native implementation of libminifi and the MiNiFi agent | |
| nifi-commons | new - will be introduced later | Contains shared code / implementations, such as security provider implementations for authentication and authorization. Note that introducing a nifi-commons project/repository is a long-term goal as part of the project structure roadmap and not part of the initial restructuring. The initial decomposition of nifi into new projects will not include any code changes, and therefore extracting shared code out of nifi-framework and nifi-registry into nifi-commons will happen in a future phase. |
The following repositories are proposed to be archived. That is, frozen from farther changes once the new repositories are ready.
- nifi: will have been broken up into nifi-api, nifi-framework, nifi-extensions, nifi-release
- Alternatively: this repo could be reduced to a single pom module containing the
org.apache.nifi:nifipom that inherits fromorg.apache:apacheand can be a parent pom for other projects.
- Alternatively: this repo could be reduced to a single pom module containing the
- nifi-minifi: will become a new assembly of nifi-framework and nifi-extensions that lives in the nifi-release repo
Proposed Approach
Phase 1: nifi
Decompose nifi project/repo into:
- nifi-api
- nifi-framework
- nifi-extensions
- nifi-release
- nifi-toolkit
Assumptions
- The project version for new projects being extracted from the existing nifi repository/project will match the current version that is set on the nifi project at the time of restructuring.
- Every Maven module that currently exists in the nifi repository/project will be migrated to exactly one destination repository/project. By "exactly one", we mean that we do not anticipate the contents of any existing module will need to be split across multiple destinations.
- No maven artifact coordinates will change for existing modules as part of the restructuring. That is, an existing artifact's
group:artifact:versionshould not change even if it is in a new location, meaning modules that have dependencies on moved modules should not need to have their dependencies updates.
Steps
- Create a mapping of existing modules in the nifi repository/project into where they will live in the new structure
- Use a utility such as git subtree to move existing modules while retaining as much revision history as possible
- Update the pom files for migrated modules to contain their new parents, etc.
- Each new repository/project will get a new assembly module that just packages that artifact so that releases can be done from that project repository
- Write new assemblies in the nifi-release project/repository that produces something that closely matches the existing assembly
- Get all builds and tests working, although this should be minimal as no maven module coordinates are changing.
Process
- One or more community volunteers will manually attempt the above steps, taking detailed notes of commands they use and changes (such as pom files) that they need to make in order to get everything working.
- From those notes, a script that streamlines (automated as much as possible) restructuring the nifi repository into new repositories will be written. This will enable us to repeatedly perform the restructuring from any given revision of the master branch.
- The restructuring automation script(s) and instructions for use will be distributed on this mailing list to the nifi community. The NiFI community will be asked to verify them and the resulting output in a manner similar to an RC verification vote. The final output assembly of the restructured process should match the output assembly of the current nifi-assembly module from the 1.x line on master today.
- The automation process will go through as many reviews as needed until it passes a vote by the NiFi PMC.
- A specified date and time will be chosen for the restructuring. Any large branches from master should be merged before that date to avoid burdensome rebasing that will be necessary for any changes not merged before the restructuring.
- On the agreed upon date and time, with the assistance of ASF Infra, the existing nifi repository will be frozen from modification. The restructuring script will be run and the resulting repositories will be added as top level github repositories, again, with the help of ASF infra.
- Unmerged pull requests to the nifi repo that were under active development will need to be recreated as PRs to the new repositories. This will fall to the PR authors. PRs will stay open to the old nifi repository for a time as a historical record, but only those that are recreated by their authors will be considered to be merged to the new repositories.
- Contributions continue as normal, but from forks of the new repositories.
- The first release after the restructuring will be a release from all repositories/projects (in the order dictated by the dependency tree shown above). After that, releases will only need to be performed as needed from each repository/project.
- NOTE: There should be a community discussion on how versioning and releasing of the new, smaller projects should be handled. That is, should we always release everything so that versions stay aligned, even if there are not changes to a particular component such as nifi-api, which changes very infrequently, or should we follow semantic versioning for each new project and allow them to advance and be released independently of each other and only as needed.
Phase 2: nifi-commons
Introduce nifi-commons as a home for shared, top-level, reusable libraries of components across the new collection of projects
Steps
- A new nifi-commons project and repository is introduced.
- Similar code across nifi-framework and nifi-registry, and possibly other projects such as nifi-toolkit and nifi-extensions, is rewritten as generic, reusable libraries in sub-modules of nifi-commons, e.g., nifi-commons-security could contain shared authentication and authorization APIs and provider implementations.
- Code from nifi-framework, nif-registry, etc. is removed and replaced with library implementations from nifi-commons modules.
Phase 3: nifi-minifi
Migrate code modules from nifi-minifi into nifi-framework, nifi-extensions, and (possibly) nifi-commons. Migrate the nifi-minifi assembly into nifi-release.
Additional Potential Phases
Once the initial restructuring and decomposition of the nifi project/repo is complete, there may be opportunities for farther steps such as breaking nifi-extensions into smaller projects, or dropping support for some legacy extensions.