THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
| Page properties |
|---|
Goals
- Allow user to indicate that data in a Connection should be spread across the cluster instead of just queuing up on the node
...
This Feature Proposal aims to simplify the configuration of the flow by allowing the user to configure a Connection and choose a Load Balancing Strategy: "Do Not Load Balance" (the default), "Round Robin" (which would automatically fail over if unable to communicate with a node), "Single Node" (all data goes to a single node in the cluster) or "Partition by Attribute" (more details below). By enabling the user to configure the flow this way, a user will be able to simply create a flow as ListSFTP → FetchSFTP and have the connection configured to use the "Round Robin" Load Balancing Strategy. This would result in automatically spreading the listing throughout the cluster and would require no Remote Process Groups or Ports be created. The flow is now much more self-contained and clear and concise.
Assumptions
Requirements
| # | Title | Importance | Notes |
|---|---|---|---|
| 1 | Provide protocol for distributing data between nodes in the cluster | MUST | |
| 2 | Support secure data transfer between nodes | MUST | Inter-node data transfer should be performed via SSL/TLS if a Keystore and Truststore are configured. |
| 3 | Expose configuration of load balancing properties in nifi.properties | MUST |
| 4 | Expose configuration in user interface for selecting a Load Balancing Strategy | MUST | |
| 5 | Update User Guide images to depict new Connection Configuration dialog | MUST | |
| 6 | Update Admin Guide to explain new nifi.properties properties | MUST | |
| 7 | Update User Guide to explain how the feature works | MUST | |
| 8 | Persist state about the nodes in the cluster and restore this state upon restart | SHOULD | When the cluster topology changes, data that is partitioned via attribute will have to be rebalanced. Upon a restart of the entire cluster, this can result in a lot of data being pushed around. To avoid this, we should persist the nodes in a cluster and restore this information upon restart. If the cluster is shutdown and one node is not intended to be brought back up, the user will be responsible from removing the node from the cluster via the "Cluster" screen in the UI. |
| 9 | Provide a visual indicator in the UI as to how many FlowFiles are awaiting transfer to another node | SHOULD | If the UI does not easily provide an indicator of how many FlowFiles are waiting to be transferred, users may well be confused about why the connection shows X number of FlowFiles but the destination is not processing them. |
| 10 | Allow user to configure whether or not compression is used | SHOULD | For some data, it will not make sense to compress the data during transfer, such as data that is known to be GZIP'd. For other data, compression can be very helpful. Additionally, even when the FlowFile content cannot be usefully compressed, the attributes likely can and so an option should be provided to compress Nothing, FlowFile Attributes Only, or FlowFiles Attributes and Content. This may well be implemented at a later time. |
User interaction and design
...
This will necessitate adding the following properties to nifi.properties:
| Proposed Property Name | Description | Default Value |
|---|---|---|
| nifi.cluster.load.balance.port | The port to listen on for incoming connections for load balancing data | 6342 |
| nifi.cluster.load.balance.host | The hostname to listen on for incoming connections for load balancing data | The same as the `nifi.cluster.node.address` property, if specified, else localhost |
| nifi.cluster.load.balance.connections.per.node | The number of TCP connections to make to each node in the cluster for load balancing | 4 |
| nifi.cluster.load.balance.max.threads | The max size of the thread pool to use for load balancing data across the cluster | 8 |
It will be important to avoid heap exhaustion if we are now maintaining multiple queues of FlowFiles. While the default is to swap out any FlowFiles over 20,000 in increments of 10,000 we cannot do this for each internal queue. If we did, then a cluster of 10 nodes would end up queuing 10 * 20,000 = 200,000 FlowFiles in each queue, which would quickly exhaust heap. As a result, we will swap out the data more aggressively in the queues that are responsible for distributing to other nodes - likely after 1,000 FlowFiles have been reached, in increments of 1,000. The "local partition," however, can continue to swap out at 20,000 / 10,000.
...
Below is a list of questions to be addressed as a result of this requirements document:
| Question | Outcome |
|---|---|
| What permissions should be required/checked when receiving data from a load balancing protocol? | Rather than relying on explicit permissions, we will ensure that data that is sent in a secure clustered is sent from a node whose certificate is trusted by the configured TrustStore and moreover that the certificate's Distinguished Name or one of its Subject Alternative Names maps to the Node identity of one of the nodes in the cluster. In this way, we ensure that the data comes from a node that is in fact part of the cluster, and this scales nicely even as we have more elastic clustering. |
| How should Disconnected nodes be handled? | A node that has been disconnected from the cluster should continue to Load Balance the data in its queue to all nodes in the cluster. However, the nodes that are connected should not send data to the disconnected node. This approach mimics how the Site-to-Site works today and lays the groundwork for later work to decommission a node in a cluster. If a node is disconnected, then it will not receive any new notifications about cluster topology updates. Therefore, it is possible that it will start sending the data to the wrong node if using the "Partition by Attribute" strategy. This needs to be acceptable, and the node that receives such data needs to be responsible for determining which node in the cluster should receive the data and distributing the data appropriately. |