Motivation

In several cases we need a special state of Ignite node when it accepts incoming commands via control scripts or JMX API but doesn't join the cluster.

In other words, we need Ignite node to be able to enter Maintenance Mode when maintenance actions could be applied.

Examples of use cases: IEP-47: Native persistence defragmentation,  Unable to render Jira issues macro, execution error.

Description

Summary

• In Maintenance Mode (later MM for short) node doesn't join the cluster but accepts user commands and/or executes other actions needed for maintenance.
• MM is applicable only when persistence is enabled and storage is available.
• To enter or leave MM node has to be restarted.

Suggested design

• Node decides to switch to maintenance only on startup if a special maintenance marker (or several markers) called Maintenance Task (later MT for short) is presented on disk. Maintenance Task consists of unique ID, user-readable description and (if necessary) arguments to complete maintenance action.
• Maintenance Tasks could be created both automatically (when an erroneous situation is found like in Unable to render Jira issues macro, execution error. ) or by user request (e.g. when user requests performing node's data files defragmentation). Maintenance Registry is responsible for managing MTs and provides API for such management.
• When maintenance action for a particular MT is completed, the task could be removed from registry. If no tasks are left node will return to normal operations on next restart. Otherwise (at least one MT is still presented on disk) node will again enter MM.

Use Case 1 - Cleaning up potentially corrupted PDS

This case is described in ticket  Unable to render Jira issues macro, execution error.  where draft implementation of MM is ready for review. This case involves automatic creation of MT but requires manual actions to complete the maintenance. Case works as follows:

1. Node fails in the middle of checkpoint when WAL is disabled for one or several caches.
2. On next restart node detects that data files of that caches could be corrupted, creates an MT and shuts down.
3. On next restart node enters MM and waits for user to fix the problem (instead of failing again). In managed environments like Kubernetes it means that node won't be automatically restarted and user will be able to find possibly corrupted files and remove them.
4. When the files are removed (manual action) user removes MT from the registry and restarts the node.
5. Node starts up in normal mode and joins the cluster in a regular way.

Use Case 2 - Native Persistence Defragmentation

IEP-47: Native persistence defragmentation is dedicated to implement this rather big piece of functionality. But in a nutshell this case again involves MM in an opposite form: Maintenance Task is created manually but action is completed automatically. Case main steps:

1. User with control.{sh|bat} script or via other APIs requests creating MT for defragmenting all native persistence on the node or particular caches. MT is created and saved on disk.
2. User restarts the node, it enters Maintenance Mode finds MT about defragmentation and starts working on the task.
3. When defragmentation is done, MT is automatically deleted. On next restart node with defragmented PDS enters normal operations.

Maintenance Action and Maintenance Workflow

Although MM supposes manual user intervention to fix the reason for maintenance, it can also be true that component requested MM knows how to fix the issues and can execute necessary actions automatically.
The only thing it may need is user command to execute these actions.

In case of PDS defragmentation also covered by MM all actions are executed automatically from the very beginning.

To cover both cases additional entity is suggested: MaintenanceAction. It is just an interface that could be called by Maintenance component when user requests its execution or when Maintenance component decides it is time to start automatic actions.

Workflow with MaintenanceAction may look like this:

1. Maintenance Registry starts among first and reads from disk information about MaintenanceTasks registered earlier.
2. Other components start after Maintenance and check MM if they should function differently in this mode. If node in MM they register special callback within Maintenance Registry that provides Maintenance Actions to the registry.
3. After all components are started Maintenance Registry prepares maintenance: checks if user has already fixed issues manually during shut down, prints information about this to log and modifies/deletes Maintenance Tasks if needed.
4. When Maintenance is prepared and there are still unresolved MaintenanceTasks Maintenance component starts automatic actions like PDS defragmentation or exposes list of user-triggered actions through CLI/JMX APIs and waits for user commands.

Implementation improvements

1. With fixed structure of workflow (register tasks → restart → prepare maintenance → execute maintenance) it is not possible to request new actions when node is in maintenance mode. We could improve this by exploiting InternalSubscriptionProcessor and registering components as listeners for events like "Maintenance Task with ID registered..."
2. Current implementation lacks validation of Maintenance Tasks except of basic validation of UUID structure. The problem here is that validation is component-specific and should be implemented by components themselves. If we employ InternalSubscriptionProcessor we could potentially implement validation of MaintenanceTasks in an elegant way: when component is notified about new task it knows how to validate, it validates it and sets specific flag to the task. When MaintenanceProcessor sees that no components have validated next task, it skips it as invalid.

Risks and Assumptions

1. It is assumed that no major changes are needed in CommandHandler to enable connecting to a particular node (the one in MM).

Discussion Links

Discussion on dev-list

Tickets