Overview
Currently, the only way for a developer to modify cluster configuration is through internal methods or to manually edit the configuration file while the cluster is offline.
It can be expected that a developer will create commands or functions that require information in or modification of the cluster configuration.
Therefore, we should expose a public API for the cluster configuration, to promote correctness and ease of use.
Background
The cache-1.0.xsd
defines the permissible structure of the cluster config, currently represented in XML.
Developers can include their own cluster configuration elements in one of two places as defined in the cache-1.0.xsd
: at the cache level or at the region level.
These accept any XML element, so long as the namespace does not match Geode's (i.e., {{http://geode.apache.org/schema/cache}}).
Goals
- The API is intended to ONLY update the cluster or server-group's configuration saved by the locator(s). For example, when developer uses the api to add a region definition to the cache configuration, it should NOT be expected that the api will also create the regions for you on the existing servers.
- The API should not be tied with XML. We should be able to change how we save cluster configuration internally without changing the api.
- Expose a clean Java API for retrieval and modification of the cluster configuration.
Approach
- Define a no-op interfaces
CacheElement
andRegionElement
to identify classes that may be saved to the cache and region XML entities, respectively. - Leverage JAXB to generate an initial set of configuration objects.
- Expose
ClusterConfigurationService
via the cache, provided a locator is managing that cache.
Proposal
As a minimum viable product, we would require only two methods: a getter and an updater.
public interface ClusterConfigurationService { /** * @param group The group to which the cluster configuration applies. * For configuration that applies to every member, use the group "cluster". * @param additionalBindClass These classes will allow the returned CacheConfig to include objects * if the bound types, rather than defaulting to {@link com.sun.org.apache.xerces.internal.dom.ElementNSImpl} * to represent the configuration elements XML. * @return The cluster configuration for the specified group. */ CacheConfig getCacheConfig(String group, Class... additionalBindClass); /** * @param group The group to which the cluster configuration applies. * For configuration that applies to every member, use the group "cluster". * @param mutator Specification of how the cluster configuration should be altered. * @param additionalBindClass These classes will allow the returned CacheConfig to include objects * if the bound types, rather than defaulting to {@link com.sun.org.apache.xerces.internal.dom.ElementNSImpl} * to represent the configuration elements XML. */ void updateCacheConfig(String group, UnaryOperator<CacheConfig> mutator, Class... additionalBindClass); }
We accept the UnaryOperator
to mutate the existing cluster configuration rather than accepting an explicit CacheConfig
to overwrite the existing cluster configuration in the interest of safe concurrent access.
For additional utility, each method could assume group = "cluster"
when not provided.
The ClusterConfigurationService
will be made available from the Cache
, provided the Cache
is managed by a Locator
. The service is unavailable for other members.
Intended Use:
Given an interface as the above, we would be able to implement, for instance, the CreateIndexCommand
instead as
public class CreateIndexCommand implements GfshCommand { private static final CreateIndexFunction createIndexFunction = new CreateIndexFunction(); @CliCommand(value = CliStrings.CREATE_INDEX, help = CliStrings.CREATE_INDEX__HELP) @CliMetaData(relatedTopic = {CliStrings.TOPIC_GEODE_REGION, CliStrings.TOPIC_GEODE_DATA}) @ResourceOperation(resource = ResourcePermission.Resource.CLUSTER, operation = ResourcePermission.Operation.MANAGE, target = ResourcePermission.Target.QUERY) public Result createIndex(...){ Result result; final Set<DistributedMember> targetMembers = findMembers(group, memberNameOrID); if (targetMembers.isEmpty()) { return ResultBuilder.createUserErrorResult(CliStrings.NO_MEMBERS_FOUND_MESSAGE); } // Objective: create this Index object. // This class was generated by JAXB RegionConfig.Index index = new RegionConfig.Index(); index.setName(indexName); index.setExpression(indexedExpression); index.setType(indexType.name()); index.setFromClause(regionPath); // Pass the function to each member, where the index will be instantiated. List<CliFunctionResult> functionResults = executeAndGetFunctionResult(createIndexFunction, index, targetMembers); result = ResultBuilder.buildResult(functionResults); ClusterConfigurationService service = getSharedConfiguration(); if (result.getStatus().equals(Result.Status.OK) && service != null) { // If the function executes successfully and the ClusterConfigurationService exists // (i.e., the member executing this command is a Locator), then update the cluster config. UnaryOperator<CacheConfig> mutator = cc -> { RegionConfig regionConfig = cc.getRegion().stream().filter(x -> x.getName().equals(regionPath)).findFirst() .orElse(null); regionConfig.getIndex().add(index); return cc; }; service.updateCacheConfig("cluster", mutator); } return result; }
Here, we declaratively build the RegionConfig.Index
object we would like to add to the configuration.
On successfull creation, we search for the RegionConfig
object to which the index configuration should belong and add this index to that array.
With the mutator defined, we execute (a concurrency-safe) update to the cluster configuration.
Comments
Common operations, such as the above "find region config", could also be considered for initial inclusion to the public API.
The use of UnaryOperator<CacheConfig>
could be replaced by a direct assignment, i.e., updateCacheConfig("cluster", myNewCacheConfig).
While this might present a cleaner API, it would also allow for race conditions between concurrent modifications of the configuration.
Long-term Goals
- Reduce duplication of effort by replacing "creation" classes (i.e.,
CacheCreation
,RegionCreation
,BindingCreation
, et al) with JAXB-generated classes. - Remove requirement of JAXB / XML annotations on configuration element classes.
- This api is meant to be used by commands that would modify the Cache configuration, e.g. create region, create gateway receiver etc. It's only available where these commands are executed. After this, we might expose another set of java API that would be available in other nodes like client or servers. That api would allow developer to create a region as well as modifying the cluster configuration at the same time.
Additional Concerns
Any exposure or modification of cluster configuration should immediately raise concerns for both security and concurrency.
Regarding security:
The onus of responsible and authorized use of this interface rests on the developer using this interface.
Any command or function reporting or modifying cluster configuration should require appropriate permissions.
Within Geode itself, these permissions would be CLUSTER:READ
and CLUSTER:WRITE
respectively.
Outside Geode, these permissions would correspond to whatever permissions are expected by the application and implementation of SecurityManager
.
Remember that permissions may be set for classes implementing CommandMarker
(i.e., gfsh commands) via the @ResourceOperation
annotation.
Permissions may be set for classes implementing org.apache.geode.cache.execute.Function
interface via the interface's method getRequiredPermissions
.
Regarding concurrency:
While we are motivated by thread safety to accept a UnaryOperator
to mutate the cluster configuration rather than expose an explicit setter, race conditions can still easily exist.
The onus of responsible and safe concurrent use rests again on the developer using this interface.
Internally, we will guarantee safe concurrent writing to avoid data corruption.
However, potential inconsistency due to the order in which commands arrive is beyond our ability or responsibility to control.
Developers are strongly encouraged to use distributed locks to safeguard configuration information between acquisition and update, as well as any context which requires the most up-to-date cluster configuration information.
Resources
- JAXB generated CacheConfig class: CacheConfig.java
- An example of an extension's CacheElement: ConnectorService.java