Table of Contents |
---|
...
Node - individual instance of Ignite, either a server or a client.
...
The main goal of the discovery mechanism is to create a topology of Ignite nodes and build and maintain a consistent in-memory view of it on every node. For example, this view can may contain the number of nodes in the cluster and their orders.
Discovery mechanism is represented by the DiscoverySpi
interface, while TcpDiscoverySpi
being its default is its default implementation. Other Other implementations, like ZookeeperDiscoverySpi
, also exist but are out of the scope of this article.
Exact topology structure is defined by the particular DiscoverySpi
implementations, for example, TcpDiscoverySpi
defines a ring-shaped topology.
Info |
---|
When describing cluster topology we , we are talking about a logical arrangement that only exists on the "discovery" level. For example, when querying data residing in caches, a cluster might use a different topology than described here. |
...
TcpDiscoverySpi
organizes all server nodes of a cluster into a ring-shaped structure where each node can only send discovery messages to a single node (called a "neighbor"). Client nodes stay outside of the ring and are always connected to a single server. This logic is contained inside ServerImpl
and ClientImpl
classes for server and client nodes respectively.
Gliffy Diagram | ||||||
---|---|---|---|---|---|---|
|
...
When a new node starts, it tries to find an existing cluster by probing all addresses provided by the TcpDiscoveryIpFinder
. If all addresses are unavailable, the node considers itself as the only node, forms a cluster from itself and becomes the coordinator. Otherwise, it starts the join process.
Node join process consists of several phases:
TcpDiscoveryJoinRequestMessage
to a random node in the cluster, which gets forwarded to the coordinator.TcpDiscoveryNodeAddedMessage
across the ring.TcpDiscoveryNodeAddedMessage
has been received by all members of the cluster, TcpDiscoveryNodeAddFinishedMessage
is sent to finalize the changes....
When the coordinator receives the request it validates the message and generates the TcpDiscoveryNodeAddedMessage,
if validation a validation was successful (see ServerImpl.RingMessageWorker#processJoinRequestMessage
). This message is then sent across the ring.
...
When handling the TcpDiscoveryNodeAddedMessage,
every node in the cluster applies joining node discovery data to components, collects its local discovery data, and adds it to the message (see ServerImpl.RingMessageWorker#processNodeAddedMessage
for details). The message is then propagated further across the ring by calling ServerImpl.RingMessageWorker#sendMessageAcrossRing
.
...
TcpDiscoveryNodeAddFinishedMessage
finishes the process of node join. When receiving this message, each node fires the NODE_JOINED
event to notify the discovery manager about the new joined node.
Info | ||
---|---|---|
| ||
The joining node will send an additional join request if it doesn't receive |
...
Detecting failed nodes is the responsibility of each server node in a cluster. However, detecting failed servers and clients works slightly differently.
...
As servers are organized into a ring each node can easily detect failure of its next neighbor node if sending of a discovery message fails.
Gliffy Diagram | ||||
---|---|---|---|---|
|
However, the process of removing a failed node from the ring is not that simple and has to be managed by the coordinator. It contains two steps:
...
During the first step, the server that detected that its neighbor has failed adds the broken node's address to a special in-memory structure called failedNodes
(see TcpDiscoveryAbstractMessage#failedNodes
). This structure is then used to filter out the failed nodes from the list of all nodes in the ring until the second step is complete.
After that, the server tries to establish a connection to the next node in the ring. If the connection is established, this node becomes the new neighbor, otherwise the process gets repeated until a working node is found.
...
The second step starts on the coordinator. When the coordinator receives a TcpDiscoveryNodeFailedMessage
it verifies it and sends it across the ring. All servers (including the coordinator) update their local topology versions, remove remove the failed node(s) from the ring structure and failedNodes
internal structures (see ServerImpl$RingMessageWorker#processNodeFailedMessage
).
...
Section patiently waits for someone to contribute contentthe content
...
Algorithm of restoring ring without failing nodes (that one involving CrossRingMessageSendState and other tricky things and special timeouts) should be covered here in great details (including special timeouts)
...
...
...
...
Section patiently waits for someone to contribute contentthe content
...
...
Section patiently waits for someone to contribute the content
...
Section patiently waits for someone to contribute the content