THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Status
Current state: Under discussion
Discussion thread:
JIRA:
Released: 5.x
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
When you are looking at Cassandra from the point of view of an operator digging into the cluster monitoring system, the operator may prefer to first take a look at the documentation for those APIs with which he is familiar and has the most personal experience, before looking at others. The most expected thing here is that the operator must see the same view of Cassandra's internal structures/processes, regardless of which type of API he intends to use or prefers. All of the internal structures and processes must be exposed to different APIs (JMX, Virtual Tables, REST, Logging) with the same order of attributes, the same attributes naming, the exact value representation and so on. Everything above must also be fair enough for cluster management operations, such as changing configuration properties at runtime.
The CEP-1: Apache Cassandra Management Process(es) already shed some light on Cassandra's usability for DevOps / Operators perspective but this document is an intention of the internal structures and management commands to get aligned for all of the existing public API of management and monitoring, which might also facilitate the writing of another type of integration as well as Cassandra's plugins.
Monitoring
Let's look at examples for each subsystem in terms of how they are represented in the most valuable of Cassandra's APIs: JMX, and Virtual Tables. However, the same should apply to others as well.
- Fragile balance. The most representative example here of how the same internal structure can look different in JMX and Virtual Tables is the thread pool metrics: ThreadPoolMetrics and ThreadPoolsTable. We should also be aware of all the APIs involved and modify them accordingly if we want to add or extend thread pool indicators, which in turn is very fragile in the face of error.
- Boilerplate implementations. Every time we add a new metric or set of metrics, we have to write additional code to manually expose them to other APIs. The example here is the CompactionMetrics, which doesn't yet have its representation in the Virtuals Table API, or another example the SettingsTable virtual table, which doesn't have its representation in the JMX API. The complexity of implementations also increases even more with the number of APIs we intend to support.
- Dropwizard and Cassandra's JMX. There are a number of useful node metrics that do not yet appear in the Dropwizard metrics set. Considering that JMX is usually not used and disabled in production environments for various performance and security reasons, the operator may not see the same picture from various of Dropwizard's metrics exporters and integrations as Cassandra's JMX metrics provide [1][2]. The example here is the CommitLog#getActiveSegmentCompressionRatios metrics.
Configuration Management
The "management" here is more about managing the running cluster configuration for the attributes we are able to change and/or we are able to load, rather than scheduling or running some bulk operation or complex task with the CLI. This part of cluster management is covered by the CEP-1: Apache Cassandra Management Process(es).
- Unsafe runtime settings change. For some of the cases, it may be possible to extend or implement the JMX MBean in such a way that its state is not reflected in the appropriate cluster configuration property. In this case, the operator cannot see the true state of the cluster and the whole approach looks fragile. For an example of such a problem, see:
- Lack of property change subscriptions. Cassandra doesn't have a mechanism for the management of subscriptions with configuration property changes, which might be changed through public API. For production environments, this is crucial for having plugins for audit logging in a financial company's internal security system, and can also help with having custom property change events in plugins.
- Scattered configurations. Currently, Cassandra provides a few property sources for the cluster configuration: Environment, System Properties, and Config class. Each of these configuration sources provides its own property naming convention and the range of influence it has on the behaviour of the cluster. From an operator's point of view, it is highly desirable for a cluster configuration property to have the same name from different property sources (and the same effect on the cluster), but to be set equally well from an environment variable, system property or configuration file.
- Metrics and Management in the same MBean. In some cases, Cassandra shares the same JMX MBean for exposing metrics and managing cluster configuration parameters. For example, in a pre-production environment, we might want to look at metric X or expose JMX access, but we don't want to expose the management commands like repair/decommission etc. Currently, we are unable to achieve this goal.
- Configuration exporters. There is no way to export the running cluster configuration, as some of the configuration properties may be changed at runtime, and the cluster may be bootstrapped with different system properties or environment variables. Without a single source of configuration property behaviour, this goal is not easy to achieve. As an example, see:
Audience
From the user and the operator perspective, these changes will help to:
- see the same metric attribute names, and metric attribute order throughout the Cassandra API;
- use any public API they like, with the guarantee that everything they can see in it is also available in other public APIs;
- make their release rollout pipelines more transparent across a variety of environments and different cluster configurations (including system properties, environment variables etc);
From the developer's perspective, these changes will help to:
- focus on the metrics, metric sets, and internal structures the developer would like to expose to the API rather than on the implementation in particular API itself;
- remove a lot of boilerplate source code from virtual table implementations;
- make the usage of internal configuration properties more robust and safe;
Goals
The following objectives have been set to address the issues described above:
- provide a framework for creating simple non-public views over internal metadata collections (property sets, running processes, metric sets), these views, in turn, can be easily exposed to all available public APIs like JMX, Virtual Tables etc. we intend to support;
- provide an internal property registry - the single internal API point for managing cluster configuration, system properties and environment variables in the same way, respecting general property source behaviour and naming conventions;
- provide a robust way to manage configuration properties through various APIs such as JMX, Virtual Tables;
Non-Goals
This proposal is not about:
- changing the structure of configuration properties, e.g. grouping them by subsystem they are related to, etc;
- changing the names of configuration properties, just the way they appear in the APIs;
- creating, and modifying cluster management tools or plugins;
Proposed Changes
Add System View Abstraction
Currently, each subsystem such as JMX or Virtual Tables implements its own internal data collection exposing machinery. To solve the problems described above, I propose to generify JMX MBeans and Virtual Tables creation and provide for this purpose a single point of API - SystemView, which technically looks like a wrapper for internal data collections. As an advantage of the generic approach, the extensions and plugins could also register their own system views in the registry and be automatically exposed in the same way as the base system views.
System View Classes Kit
SystemViewRegistry – a collection of all system views registered by Cassandra subsystems, with support for a subscription mechanism for system view exposure;
SystemView<? extends ViewRow> – a non-public set of some internal data we'd like to expose to supported data APIs, can be thought of as a wrapper over the internal data collection;
ViewRow – a reflection of an internal data collection row as a java plain data object format;
SystemViewWalker<? extends ViewRow> – given that each row of the internal data collection is transformed into its plain representation, we still need to do it for each row over the data set and collect metadata; this walker is created for this purpose;
System View Exporters
The initial implementation must be shipped with the following system view exporters:
- JMX – a new MXBean can be exposed by using a DynamicMBean generated on the top of the ViewRow metadata;
- Cassandra's VirtualTable – the corresponding virtual table metadata can be pulled from the system view by the walker, and data collection is iterated using the appropriate system view iterator;
Add Configuration Property Registry
The cluster configuration is loaded into the Config class during a node bootstrap. Some of the attributes of the Config class can be modified at runtime leaving the initial configuration file unchanged. I would like to keep this behaviour as it is, but make the way of changing the property value generic.
Property Registry
ConfigPropertyRegistry – a runtime collection of all cluster configuration properties and their metadata built from various property sources (e.g. system properties, environment variables, class Config attributes) with support for a subscription mechanism to handle property changes.
The registry has the following crucial characteristics:
- transparent naming of cluster configuration properties across all environments - a given property must have the same name (respecting the naming convention in each environment) across environment variables, system properties and the Config class attributes;
- a given cluster configuration property may be overridden and loaded in the following order: Command line arguments → Java system properties → OS environment variables → Config class (loaded from the
cassandra.yaml). As an example how can a particular property may look like, which threat the same property for the ConfigPropertyRegistry: Java Class Attribute –Config.cdc_block_writes, Java System Property –cassandra.cdc_block_writes, Environment Variable –CASSANDRA_CDC_BLOCK_WRITES; - each of the internal subsystems (CommitLog, Compaction) can register a property setter thus the same handler will be used by various public APIs like JMX MBeans or an update statement over Virtual Table with configuration properties;
- support subscriptions on property change events;
Configuration Exporters
It is anticipated that custom exporters from the config property registry to YAML and/or the properties file format will eventually be implemented.
New or Changed Public Interfaces
Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.
A public interface is any change to the following:
- native protocol (and CQL)
- gossip and the messaging service
- pluggable components (SPIs) like authorisation, triggers, ..?
- commitlog, hintlog, cache files
- sstables components
- configuration
- jmx mbeans (including metrics)
- monitoring
- client tool classes
- command line tools and arguments
- operational routines
- Anything else that will likely break existing users in some way when they upgrade
Compatibility, Deprecation, and Migration Plan
In order to align the attribute names of the exposed internal data collection in different Cassandra APIs, it is necessary to slightly change the representation of these names in JMX MBeans and Virtual Tables, which in turn may lead to compatibility issues. To make the proposed changes backwards compatible with the previous version of Cassandra, all MBeans and Virtual Tables we already have will remain unchanged.
However, to achieve usability and transparency, each new system view that is added will:
- a newly automatically exposed MBean is placed in the new group;
- a newly automatically exposed Virtual Table is placed in the new keyspace;
Those JMX MBeans, Virtual Tables that have already been moved to the new framework will be marked as deprecated with an appropriate warning log message added each time they are accessed and will eventually be removed in future releases.
Implementation Plan
Phase 1
TBD
Phase 2
Test Plan
Describe in few sentences how the CIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?
Rejected Alternatives
The Apache Commons Configuration [4] provides a generic configuration interface which enables to read configuration data from a variety of sources and at the first glance may address all the problems described in the configuration management section. At this point in the investigation adding an extra library dependency sound like a bad option, and need to be widely discussed with the community.
Related Issues
Monitoring
Configuration Management