View Source

Target release	Release name or number
Epic
Document status
Document owner	Unknown User (aldrin)
Designer	Lead designer
Developers	Lead developer
QA	Lead tester

Goals

Be more resource efficient for handling very large files (multi-gigabyte) to avoid unnecessarily duplication
To provide a framework level mapping to external content from within NiFi FlowFiles
Establish an API for source processors that introduce content/flowfiles into a dataflow to provide a dereferencable URI to content, creating a pass by reference for the entirety of dataflow.

Background and strategic fit

NiFi supports files of all sizes and formats. However, when considering bulk ingest and handling of large files, this can create suboptimal patterns on ingest of data. Consider the case of a multi-gigabyte file being transported/routed by NiFi to something like HDFS with minimal inspection or modification to the source data. In such scenarios, there is little value in introducing the footprint of this content into the NiFi repository itself for sheer routing decisions. What is prescribed is a means of providing an extended pass by reference to the source of the data itself avoiding duplication of data and, possibly, IO unless it is needed. This approach would deprecate/alter the mechanism by which the List/Fetch, Get and related source processors behave. Effectively, one could locate a series of files that are to be delivered to some consumer, performing the intermediary routing and then streaming the content to its destination(s) as needed without introducing a duplicate copy into NiFi's configured content repository. This would be an opt-in and configurable mechanism for those dataflow paths that deal with simple movement of large files while still benefitting from many of NiFi's core values like provenance and event level processing.

Apache NiFi > External FlowFile content > External File Diagram.png

Assumptions

Content modification to an external file would introduce changes into a new content claim in NiFi's internal repository
Source processors (those that introduce/create flow files) are the key point of this feature's incorporation into NiFi and would work in tandem with the framework to provide an appropriate URI to access the data

Requirements

#	Title	User Story	Importance	Notes
1	Transparency of External File Approach	Users need to be able to manage external files in flow in a homogeneous manner with "classic" FlowFiles/content		Additional considerations or noteworthy references (links, issues)
2	Extensible to support varied protocols	The largest files where simple replication and transport occurs could include varied sources such as local file, HDFS, S3, and SFTP.

User interaction and design

Extension configuration will need an additional property to enable the treatment of fetched/received files as external
After the point where a FlowFile with a reference to an external repository enters a flow, its handling should fall under the same mechanisms as traditional FlowFiles and should be relatively transparent

Other enhancements enabled with this feature

Depending on source and sink relationships for a dataflow, it might be possible to leverage/integrate other tools. As an example, consider the case where we are doing a bulk HDFS to HDFS transfer. Given this kind of agreement, it might be possible to delegate to a tool like distcp to provide movement of this information with NiFi orchestrating this arrangement. NiFi could manage the totality of the dataflow but provide an optimized transport mechanism.

Prior Art and Related Work

Filesystem in User Space (FUSE): https://en.wikipedia.org/wiki/Filesystem_in_Userspace

Questions

Below is a list of questions to be addressed as a result of this requirements document:

Question	Outcome
If a content is to be delivered to multiple endpoints, and we can determine this, are there optimizations available to avoid the issues of
How do we know when/handle changes to an external file?	Communicate the decision reached
How are reads handled (something like ExtractText)? Does this content then get introduced into a NiFi content repository at this time?
Can we apply our offset & length mechanism for splittling parts of a component (SplitText)?
What, if any, are the commonalities of this with the ideas of the High Availability Processing?

Not Doing

List the features discussed which are out of scope or might be revisited in a later release.