Motivation

Apache Ignite should have some general engine approach to handle critical failures.

Description

List of The following failures should be covered by this engine:

• Critical Errors
• Critical system workers crashes
• Segmentation

treated as critical:

• System critical errors (e.g. OutOfMemoryError);
• Unintentional system worker termination (e.g. due to an unhandled exception);
• Cluster node segmentation.

User should have an ability to define node behavior in case of this failures.

System critical error - error which leads to the system's inoperability.

The following system critical errors should be handled with proposed approach:

• File IO errors. Usually IOException's threw by read/write operations on file system. The following subsystems should be considered as critical:
  
  • WAL
  • Page store
  • Meta store
  • Binary meta store
• IgniteOutOfMemoryException
• OutOfMemoryError (we should have some memory reserved for this case at node startup to increase chances to handle OOM).
  
  

The following system workers are critical and ignite node will be inoperative in case of termination one of this workerList of system workers should be covered by this engine:

• disco-event-workertcp-disco-sock-reader
• tcp-disco-srvr
• tcp-disco-msg-worker
• tcp-comm-worker
• grid-nio-worker-tcp-comm
• exchange-worker
• sys-stripe
• grid-timeout-worker
• db-checkpoint-thread
• wal-file-archiver
• wal-write-worker
• wal-file-decompressor
• ttl-cleanup-worker
• nio-acceptor
  

List of errors to be handled 

• Persistence errors
• IOOM errors (part of persistence errors?)
• IO errors (list to be provided)
• OOM (we should have some memory reserved for this case at node startup to increase chances to handle OOM)
• Assertion errors (we should handle assertions as failures in case -ea flag set) (should be covered at Throwable catch for every system worker as well)

Initial design

Important notes:

• More than one Ignite node could be started in one JVM process.
• Different nodes in one JVM process could belong to different clusters.

Initial design

IgniteConfiguration should be extended with methods:IgniteConfiguration have to be extended with methods

public IgniteConfiguration setIgniteFailureHandlersetFailureHandler(IgniteFailureHandlerFailureHandler igniteFailureHndhnd);

public IgniteFailureHandlerFailureHandler getIgniteFailureHandlergetFailureHandler();

Where :

interface IgniteFailureHandlerFailureHandler {
   IgniteFailureActionboolean onFailure(IgniteFailureContextIgnite ignite, FailureContext failureCtx);
}

class IgniteFailureContextFailureContext {
   IgniteFailureTypeFailureType type;
   Throwable causeerror;
}

enum IgniteFailureActionFailureType {
   SEGMENTATION,
   RESTARTSYSTEM_WORKER_JVMTERMINATION,
   STOP,
   NOOP;
}

enum IgniteFailureType {
   SEGMENTATION,
   SYSTEM_WORKER_CRASHED,
   CRITICAL_ERROR
}

So, provided by user subclass of IgniteFailureHandler able to decide what to do (see. IgniteFailureAction) on each registered failure (see. IgniteFailureContext).

Risks and Assumptions

// Describe project risks, such as API or binary compatibility issues, major protocol changes, etc.

Discussion Links

http://apache-ignite-developers.2346864.n4.nabble.com/Internal-problems-requiring-graceful-node-shutdown-reboot-etc-td24856.html

Reference Links

CRITICAL_ERROR
}

FailureHandler implementation will be able to handle Ignite critical failures accordingly to strategy provided by user.

The following implementations should be provided out of the box:

• NoOpFailureHandler - Just ignores any failure. It's useful for tests and debugging.
• RestartProcessFailureHandler - Specific implementation that could be used only with ignite.(sh|bat). Process must be terminated using Ignition.restart(true) call.
• StopNodeFailureHandler - This implementation will stop Ignite node in case of critical error using Ignition.stop(true) or Ignition.stop(nodeName, true) call.
• StopNodeOrHaltFailureHandler(boolean tryStop, long timeout) - This implementation will try to stop node if tryStop value is true. If node can't be stopped during provided timeout or tryStop value is false then JVM process will be terminated forcibly ( Runtime.halt() ).

Default failure handler is StopNodeOrHaltFailureProcessor where tryStop value is false.

Critical system worker must catch all exceptions ( Throwable and derived classes) in high-level try-catch block and take into account that thread could be terminated due to an programmatic mistake that leads to unintentional worker termination. So basic template should looks like the following code snippet:

@Override
public void run() {
    Throwable err = null;

	try {
      // Critical worker's code.
    }
    catch(Throwable e) {
      err = e;
    }
    finally {
      // Call failure handler.
      FailureContext failureCtx = new FaulureCtx(FailureType.SYSTEM_WORKER_TERMINATION, err);

      ctx.failure().process(failureCtx);  // Handle failure. Where ctx - kernal context.
    }
}

Example of using FailureHandler in IgniteConfiguration via Spring XML:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    <property name="failureHandler">
        <bean class="org.apache.ignite.failure.StopNodeFailureHandler"/>
    </property>
</bean>

Risks and Assumptions

Discussion Links

1. Internal problems requiring graceful node shutdown, reboot, etc.
2. IEP-14: Ignite failures handling (Discussion)

Reference Links

1. Apache Ignite documentation: Ignite life cycle
2. Apache Ignite documentation: Start from command line

...

Tickets