Table of Contents |
---|
Status
Current state: Under Discussion
...
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Cassandra makes use of the JMX to expose the management commands, such as taking a snapshot operation of specified keyspaces, with local JMX access enabled by default and remote access enabled by configuration when it's needed. JMX is used as the transport layer for command execution by all of the management tools, such as the built-in command line tool, the `nodetool` that ships with the Cassandra itself, or the Cassandra Sidecar a standalone JVM process that runs alongside the Cassandra server daemon and is used for the configuration management and/or metrics exposure.
...
There is no intention to fix everything in an initial pull request, for reasons of backwards compatibility. However, once the necessary steps have been taken to support command execution via CQL will be completed, the adopted deprecation policy will be applied.
Audience
- Cassandra operators who want to configure the database and/or run adopted cluster commands;
- Cassandra developers who are involved across the ecosystem building tools for cluster management;
Goals
- To provide a way to run predefined management commands via CQL;
- To provide a mechanism for retrieving command definitions and metadata via CQL;
- To provide information on all available management commands via virtual tables;
- To provide a registry that stores all C* commands and their metadata accordingly;
- To internal instrumentation and a reasonable plan for migrating cluster management from JMX to CQL, taking into account backward compatibility and adopted deprecation policies;
Non-Goals
Considering how much of the world's infrastructure relies on our current JMX interfaces to orchestrate C* clusters, there is no appetite to update or remove existing MBeans as well as CLI tools currently in use. This proposal doesn't also imply that JMX will be deprecated once the design is fully implemented, but rather it moves the state of the project toward the point where such deprecation might be possible.
Proposed Changes
The proposed design describes all the changes that are needed to support command execution via CQL from a high-level component perspective, changes that are necessary to understand the solution design, and only incorporates public API changes, leaving internal classes and interfaces undescribed to allow room for manoeuvre during the implementation.
Basic Requirements
Requirements that are necessary to support the execution of management commands through CQL:
- Asynchronous execution, preferably with a UUID for each execution to track the execution result;
- A single command registry, to provide generic command metadata to the API consumers such as CLI tools, or other ecosystem products. This concept is borrowed from the Dropwizard metrics registry and shares the same idea;
- A dedicated admin port with the native protocol behind it, allowing only admin commands, to address the concerns when the native protocol is disabled in certain circumstances e.g. the disablebinary command is executed;
- Management via CQL must be developed alongside existing C* management API such as statically registered JMX MBeans, to ensure backward compatibility for ecosystem products that rely on it. However, this doesn't mean that new commands have to be implemented twice when they are required;
- Reduce the cost of implementing a new command, meaning that once a new command is registered in the command registry, it is automatically available through all of the public interfaces we support e.g. JMX, CQL. There is no need to implement a new command for the CLI tools, such as the nodetool, or the corresponding MBeans, a new command must be available there out of the box.
Other Vendor Solutions
How other vendors have solved the same problem:
- Apache Ignite offers a variety of management interfaces, among which the REST API as a database plugin and dynamically generated JMX MBeans. However, the preferred tool for many is the CLI tool, which acts as a wrapper over a standard thin client. This CLI connects to the cluster through a special admin port, typically set to 11211, to perform its management tasks, better known as compute tasks, across the nodes in the cluster. These compute tasks are part of the standard Ignite API for users [1].
- Apache HBase uses Google’s protobuf to describe the RPC interfaces they expose to clients, for example, the Admin and Connection interfaces. The Admin and Connection interfaces in HBase don’t require a dedicated admin port to initiate a connection for cluster management. These classes utilize the client API to communicate with the HBase cluster over the same ports that are used for regular client access to HBase services [2].
Command Specifications
At present, Apache Cassandra doesn't advocate a unified approach to the specification of management commands that are promoted directly by the C* node itself. The heart of these management commands lies in the JMX MBeans. While each MBean provides operations on the same keyspaces and tables, they differ subtly in aspects such as command parameter names and their order. Additionally, MBeans do not provide fine-grained command metadata to a user. This limitation causes command-line interfaces and other tools that depend on the JMX API to create their layer of organizing MBean operations, adding an extra layer of grouping to the process.
...
To solve this problem we can use the same design approach that has been adopted by the Dropwizard Metrics library for metrics [3], choosing the right granularity when creating command metadata. This means that all of the commands that are executed across the same keyspaces must have a common denominator of the command arguments and the order of the arguments.
Command Registry
The starting point for management operations is the CommandRegistry (or OperationManager), which is a collection of all commands or subcommands for C* management commands.
Command Registry Adapters
Once the CommandRegistry is available, some out-of-the-box adapters must be developed to achieve the goals of the proposal and make the commands available through JMX and CQL:
- CQL Command Adapter - The CQL invoker validates the given arguments based on the command metadata from CommandRegistry and invokes the corresponding command;
- Dynamic Command MBean Adapter - New dynamic JMX MBeans for management operations are generated and exposed to public API based on available command metadata for use by the nodetool. The JMX MBeans that are now statically registered are still supported. However, they are deprecated in favour of new ones;
Commands Virtual Table
All the commands and command definitions are available by querying a corresponding new virtual table based on metadata provided by the CommandRegistry.
CQL Command Syntax
The commands below become valid. While many commands in Apache Cassandra are typically run on keyspaces and tables, and adopting a CQL syntax like `EXECUTE COMMAND rebuild ON keyspace.table` seems logical, it's more practical for the initial implementation to concentrate on accepting command arguments as straightforward key-value pairs or as a JSON string. This approach simplifies the early stages of development. Subsequently, the more intuitive CQL syntax can be introduced as an alias for these commands, enhancing usability and aligning with familiar patterns.
Execute Command
JSON Syntax
Code Block | ||
---|---|---|
| ||
EXECUTE COMMAND forcecompact WITH JSON '{ "keyspace" : "distributed_test_keyspace", "table" : “tbl”, "keys" : ["k4", "k2", "k7"] }'; |
...
Code Block | ||
---|---|---|
| ||
EXECUTE COMMAND rebuild WITH JSON '{ "keyspace" : “distributed_test_keyspace”, "sourceDataCenterName" : "datacenter1", "tokens" : null, "specificSources" : null, "excludeLocalDatacenterNodes" : true }'; |
Standard Syntax
Code Block | ||
---|---|---|
| ||
EXECUTE COMMAND [command_name]forcecompact WITH keyspace=distributed_test_keyspace AND table=tbl AND keys=["k4", "k2", "k7"]; |
...
Code Block | ||
---|---|---|
| ||
EXECUTE COMMAND setconcurrentcompactors WITH concurrent_compactors=5; |
Describe Command
Code Block | ||
---|---|---|
| ||
DESCRIBE COMMANDS | COMMAND command_name; |
...
- List of all commands in a cluster;
- Executable CQL information for a single command, including command attributes and command description;
Nodetool Compatibility
Once the CommandRegistry is in place, our goal is to streamline the process so that simply adding a command to the registry makes it available to the CLI. This leads to a decision point: we can either adopt a new CQL syntax for executing commands or continue utilizing JMX MBeans for managing node states. Opting for the new CQL syntax would necessitate reconfiguring the nodetool to communicate through a new administrative port. However, this change risks breaking backward compatibility for numerous existing deployments. We must consider various deployment complexities, such as potential firewall restrictions blocking the new port, particularly in scenarios involving remote node management.
...
- New dynamic JMX MBeans must be created to expose the available commands to the public API in a way that matches the corresponding CQL queries. The API that is provided by static MBeans and the CLI are too far apart.
- The nodetool uses a newly created dynamic MBean to achieve both API alignment and API backward compatibility goals;
- The nodetool parses the input arguments based on the command metadata it receives from the CommandRegistry;
MVP
Although the scope has been described quite broadly, the minimum viable product includes the following changes:
- The CommandRegistry is available;
- Some commands from the nodetool are migrated to the new CommandRegistry;
- New dynamic MBeans representing the registered commands are available;
- Added support for a new CQL syntax that accepts JSON for command execution;
- Added to the nodetool a new command invoker, which is aware of new commands via the CommandRegistry;
New or Changed Public Interfaces
The new public interfaces are available:
- The native protocol specification is updated and new CQL syntax is supported;
- New dynamic JMX MBeans are exposed that are aligned with the CLI and CQL;
- The `nodetool` commands are adapted based on the metadata available in the CommandRegistry, while the old arguments and argument order are still supported;
Compatibility, Deprecation, and Migration Plan
It's not expected that the proposed changes will be finalized in a single JIRA issue and pull request unless we're focusing on a minimum-viable product. It is planned to implement a deprecation policy for the statically registered MBeans over several major releases. For example, `nodetool` commands will be phased into new JMX MBeans, with each command migrated individually. Throughout this process, we'll ensure that full backward compatibility is maintained for users and the distributed tests as well.
Dynamic JMX MBeans
New dynamic MBeans are available in the public API that match the migrated `nodetool` commands and the way the same commands can be executed via CQL. These endpoints are fully compatible as the statically registered MBeans remain intact.
CQL Syntax
The proposed changes to CQL should be fully backwards compatible since it doesn’t introduce any regressions to current CQL capabilities.
Internode Messaging
Command execution via CQL using the newly introduced syntax must be rejected until all nodes in the cluster are upgraded to the new version that supports the new syntax.
Test Plan
The nodetool commands are widely used across the distributed tests to perform operations on the node, which in turn means that running the full CI is sufficient to verify that these executions produce the same expected results.
For the CQL commands a new custom test adapter is implemented that translates an MBean method call into the appropriate CQL syntax for the command that is being executed, so everything is tested via CQL on the same set of `nodetool` tests.
Rejected Alternatives
There are not many alternatives in this case, since the CQL is the Cassandra-specific protocol, meaning that all other possible implementations would lead to implementations that are overwhelmed by the unrelated code and would also increase the amount of change and maintenance enormously.
Since management via CQL will be available after the changes have been implemented, the question remains whether nodetool should still use the JMX client to trigger the management operation, or whether it should be migrated to the CQL instead. The former seems to be a smoother transition in terms of usage in C*s already in useIf there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.