DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
- Faster build times for dev/testing/review/release
- Smaller release artifacts
- Maintain or increase the productivity of dev community (i.e., the new project structure should not be too burdensome for existing and future contributors to adopt)
- Facilitate code reuse across projects, such as nifi and nifi-registry, e.g., shared nifi security provider implementations for authentication and authorization.
- Position the NiFi project to more easily support and take advantage of future Java language features, such as modules.
- Better separation of concerns & better defined interfaces / touch points between project components
- Promote the idea that each module contains its own code and unit tests against public APIs such that it can be developed, tested, and released independently
- Cross-module integration tests can exist outside of each module, in their own module that tests the integration of two or more modules
...
The following repositories are proposed. Each repository would contain a top-level maven project that inherits from the org.apache:apache parent pom.
| Project / Repo Name | New? | Dependencies | Description | |
|---|---|---|---|---|
nifi-api | new - will be extracted from nifi | Contains any Java APIs that need to be agreed upon by multiple top-level projects, such as nifi-framework and nifi-extensions | ||
nifi-framework | new - | will beextracted from nifi | nifi-api, nifi-maven, nifi-fds, nifi-registry, nifi-commons | Contains the NiFi web server and front end without any unneeded flow extension bundles such as processors, controller services, and reporting tasks. |
nifi-extensions | new - will be extracted from nifi | nifi-api, nifi-maven, nifi-commons | Extension bundles for NiFi, such as processors, controller services, and reporting tasks. | |
nifi-release | new - | will beextracted from nifi | nifi-framework, nifi-extensions | Will be the new home for the nifi-assembly that produces convenience binaries such as Eventually, this repository will also take over the current role of the nifi-minifi project/repository by providing the |
nifi-toolkit | new - will be extracted from nifi | nifi-framework, nifi-registry, nifi-commons | Contains the nifi-toolkit source code and assembly | |
nifi-maven | existing | nifi-api | Contains the Maven plugin used for packaging NARs | |
nifi-fds | existing | NiFi Flow Design system - reusable front end components used to create consistent front ends, such as the nifi and nifi-registry web UIs | ||
nifi-registry | existing | nifi-api, nifi-fds, nifi-commons | A web service for centralized storage and versioning of NiFi flows and extensions | |
nifi-minifi-cpp | existing | A native implementation of libminifi and the MiNiFi agent | ||
nifi-commons | new - | will beintroduced later | Contains shared code / implementations, such as security provider implementations for authentication and authorization. Note that introducing a nifi-commons project/repository is a long-term goal as part of the project structure roadmap and not part of the initial restructuring. The initial decomposition of nifi into new projects will not include any code changes, and therefore extracting shared code out of nifi-framework and nifi-registry into nifi-commons will happen in a future phase. |
The following repositories are proposed to be archived. That is, frozen from farther changes once the new repositories are ready.
nifi: will have been broken up into nifi-api, nifi-framework, nifi-extensions, nifi-release- Alternatively: this repo could be reduced to a single pom module containing the
org.apache.nifi:nifipom that inherits fromorg.apache:apacheand can be a parent pom for other projects.
- Alternatively: this repo could be reduced to a single pom module containing the
nifi-minifi: will become a new assembly of nifi-framework and nifi-extensions that lives in the nifi-release repo
...
- One or more community volunteers will manually attempt the above steps, taking detailed notes of commands they use and changes (such as pom files) that they need to make in order to get everything working, such as modifications to pom files.
- From those notes, a script will be made that streamlines (automated as much as possible) ideally, fully automated) the restructuring the nifi repository into new repositories will be written. This will enable us to repeatedly perform the restructuring from any given revision of the master branch.
- The restructuring automation script(s) and instructions for use will be distributed on this the Apache NiFi dev@ mailing list to the nifi community. The NiFI NiFi dev community will be asked to verify them the procedure and the resulting output in a manner similar to an a RC verification vote. The final output assembly of the restructured process should match the output assembly of the current nifi-assembly module from the 1.x line on master today.
- The automation process will go through as many reviews as needed until it passes a vote by the NiFi PMC.
- A specified date and time will be chosen for the restructuring. Any large branches from master should be merged before that date to avoid burdensome rebasing that will be necessary for any changes not merged before the restructuring.
- On the agreed upon date and time, with the assistance of ASF Infra, the existing nifi repository will be frozen from modification. The restructuring script will be run and the resulting repositories will be added as top level github Github repositories, again, with the help of ASF infraInfra.
- Unmerged pull requests to the nifi repo that were under active development will need to be recreated as PRs to the new repositories. This responsibility will fall to the author of each PR authors. PRs will stay open to the old nifi repository for a time as a historical record, but only those that are recreated against the new repositories by their authors will be considered to be merged to the new repositories.
- Contributions continue as normal, but from forks of the new repositories.
- The first release after the restructuring will be a release from all repositories/projects ( in the order dictated by the dependency tree graph shown above). After that, releases will only need to be performed as needed from each repository/project.
- NOTE: There should be a community discussion on how versioning and releasing of the new, smaller projects should be handled. That is, should we always release everything so that versions stay aligned, even if there are not changes to a particular component such as nifi-api, which changes very infrequently, or should we follow semantic versioning for each new project and allow them to advance and be released independently of each other and only as needed.
...