Recent commits have allowed us to create extensions with code and dependencies that can be included conditionally. This will allow MiNiFi C++ consumers to select the features that they desire, help us build a cohesive API within libminifi, and support patches/upgrades with shared objects. The first step is to move functional components into this paradigm.

 

The first component to move will be processors. There has always been a small selection of base processors within MiNiFi. Moving these to a separate extension will allow consumers to include processors conditionally. The design of this extension will be specialized to allow processors/or controller services to be selected in groups or individually. Further, these selected components will be built within the executable and as a shared object that can be versioned as a deliverable to MiNiFi agents. This will be integral to the update framework that will be discussed shortly.

 

If a consumer wishes to create a binary with only a GetFile processor and an RPG, they could specify the following flags (subject to change):

            cmake –DINCLUDE_ONLY=GetFile –DINCLUDE_ONLY=RemoteProcessGroup.


This approach will create a binary with only the GetFile processor and RemoteProcessGroup along with a shared object that can be used for patching or updates.  The addition of new base processors will not require the addition of any CMAKE code or code beyond the processor and test. The shared object will include a version that identifies the built source and enables agents that load that shared object to enforce update rules. While a binary limited to GetFile and an RPG may not be useful to many, including updates to either can allow us to update agents in real time through the transmission of a shared object. Realtime updates with non updates to system dependencies can be done in realtime. This can be done by versioning the class loading mechanism. While we don't aim to match the complexity of java class loading, we can certainly use the facilities that exist now to separate class instantiation. Once registry versioning is complete, we should be able to support object isolation.

 

The next focus will be components within MiNiFi such as C2. Creating build time conditionals for these components will facilitate the eventual goal of making libminifi more developer friendly.  The code is created in such a way that modularization is simple; however, we don’t have compile time capabilities to exclude integral components to an agent. Further, we don’t have an API that is intuitive. This proposal includes the foundation for that API. The goal will be to use CMAKE to more easily facilitate building either a library or agent.

 

While MiNiFi C++ can create bindings for virtually any language we recognize that not everyone wishes to use C++ or some component libraries that make bindings simpler. As a result, there will be C functions that facilitate the usage of LibMiNiFi. In doing so we can seamlessly create bindings for any language without the need for libraries that may not be desired across platforms. The current projection is that we facilitate compartmentalizing components. All functional components can and should be programmatically linked with the ability to have some or non. For example, if a consumer simply wants to transmit files across site to site. The following functions will facilitate this( subject to change):

 

int8_t create_flow_file(FlowFile** flow_file, char *content_location, size_t size)
int8_t transmit_flow_file(FlowFile *flow_file, char *nifi_host);
int8_t free_flow_file(FlowFile *flow_file);

 

These efforts will be separated into multiple tickets in a larger epic. More tickets will arise from the development of this proposal; however, items one and two can begin immediate development. 

  1. Separate base processors into a conditional component
  2. Separate C2 into a conditional compile time component
  3. Improve API to facilitate building custom agent
  4. Define policy for updating components to include better management of properties
    1. Enable dynamic properties
  5. Create C library function calls to utilize this API

Cons

  • The major negative to this has been one we've already experienced: increased cost in vtables lookup and class loader lookup 1. While we can't combat the former, we have done some work with template meta programming to fight the latter. More can be done; however, the cost will always exist, but is traditionally very minimal and paid at startup. Nonetheless, we already pay these costs so it will be difficult to argue that we pay the cost more. 
  • Design complexity may make development more difficult.
  • Debugging may become more difficult due to the depth of call graphs. 

 

 

1 https://stackoverflow.com/questions/18404988/performance-hit-of-vtable-lookup-in-c

  • No labels