...

Motivation

...

The following system workers are critical and ignite node will be inoperative in case of termination one of this worker:

• disco-event-workertcp-disco-sock-reader
• tcp-disco-srvr
• tcp-disco-msg-worker
• tcp-comm-worker
• grid-nio-worker-tcp-comm
• exchange-worker
• sys-stripe
• grid-timeout-worker
• db-checkpoint-thread
• wal-file-archiver
• wal-write-worker
• wal-file-decompressor
• ttl-cleanup-worker
• nio-acceptor
  
  

...

interface FailureHandler {
   boolean onFailure(Ignite ignite, FailureContext failureCtx);
}

class FailureContext {
   FailureType type;
   Throwable error;
}

enum FailureType {
   SEGMENTATION,
   SYSTEM_WORKER_TERMINATION,
   CRITICAL_ERROR
}

...

Default failure handler is StopNodeOrHaltFailureProcessor where tryStop value is false.

Critical system worker must catch all exceptions ( Throwable and derived classes) in high-level try-catch block and take into account that thread could be terminated due to an programmatic mistake that leads to unintentional worker termination. So basic template should looks like the following code snippet:

@Override
public void run() {
    Throwable err = null;

	try {
      // Critical worker's code.
    }
    catch(Throwable e) {
      err = e;
    }
    finally {
      // Call failure handler.
      FailureContext failureCtx = new FaulureCtx(FailureType.SYSTEM_WORKER_TERMINATION, err);

      ctx.failure().process(failureCtx);  // Handle failure. Where ctx - kernal context.
    }
}

Example of using FailureHandler in IgniteConfiguration via Spring XML:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    <property name="failureHandler">
        <bean class="org.apache.ignite.failure.StopNodeFailureHandler"/>
    </property>
</bean>

Risks and Assumptions

Discussion Links

...