Page History

...

CoreContainer creates and initializes a single instance of ResourceManager in its load() method. This instance is configured using a new section file in /resourceMgr/clusterpropsmanagerConfig.json/poolConfigs. Several default pools are always created (at the moment they are all related to SolrIndexSearcher caches) but their parameters can be customized using the clusterprops /resourceMgr/poolConfigs.json.

SolrIndexSearcher.register() now also registers all its caches in their respective pools and unregisters them on close().

...

Pool configurations are kept in ZK in /resourceMgr/poolConfigs.json. Changes made to this file via API are watched by all live nodes, and upon change each node refreshes its internal copy of the config and re-configures local pools to match the config.

Currently the PR doesn't use other configuration files or system properties.

Remote API

The content of the pool configurations file is a serialization of `ResourceManagerAPI.ResourcePoolConfigs`, which is basically a map of pool names to their configurations. Each pool configuration consists of the following:

• name (required) - unique pool name
• type (required) - one of the supported pool types (currently only "cache" is supported)
• poolParams (optional) - a map of arbitrary key-value pairs containing runtime parameters of the pool. Currently supported parameters:
  • scheduleDelaysSeconds - how often the resource manager will invoke the pool's manage() method, which checks and controls the resource usage of its components.
• poolLimits (optional) - a map of arbitrary key-value pairs containing total resource limits for the pool. Eg. for "cache" type pools these are currently:
  • maxSize - defines the total maximum number of elements in all cache instances in the pool
  • maRamMB - defined the total maximum memory use of all cache instances in the pool

There are several pre-defined pools, which can be listed using the /cluster/resource API.

Example configuration in /resourceMgr/poolConfigs.json:

{
 'searcherDocumentCache' :
 {
  'maxRamMB' : 100
 }
}

Currently the PR doesn't use other configuration files or system properties.

Remote API

There's a new v2 ResourceManagerAPI accessible at /cluster/resources for managing cluster-level aspects of the framework (such as pool configurations, their There's a new v2 ResourceManagerAPI accessible at /cluster/resources for managing cluster-level aspects of the framework (such as pool configurations, their limits and parametersrs) and /node/resource for managing node-level parameters (such as directly modifying individual component's limits).

Changes to pool configurations are persisted in ZK . Also, each node watches the changes in this file and upon change it reloads the config and re-configures local pools to match the config - this may include removing, adding pools, changing their limits and parameters.Per-node (component) operations that select named items all treat the name as a prefix, ie. the selected items are those that match the prefix provided as the name parameter. This is required because of the quickly changing identifiers of the components.pools to match the config - this may include removing, adding pools, changing their limits and parameters.

Per-node (component) operations that select named items all treat the name as a prefix, ie. the selected items are those that match the prefix provided as the name parameter. This is required because of the quickly changing identifiers of the components.

Update operations that use maps of key-value pairs as payload all use the same "partial update" semantics: new or existing values with the same keys are created/updated, null values cause existing keys to be deleted, and all other existing KV pairs are unchanged.

The following operations are supported:

• Pool operations
  • LIST - lists selected pools and their limits and parameters.
  • CREATE - create a new pool, with the provided limits (limit.<name>=<value>) and parameters (param.<name>=<value>).
  • DELETE - delete an existing pool (and unregister its components)
  • SETLIMITS - set, modify or delete existing pool(s) limits
  • SETPARAMS - set, modify or delete existing pool(s) parameters
• Component operations
  • LIST - list components in specified pool(s) and their current resource limits
  • SETLIMITS - set the current limits of specified component(s). Payload is a map of key-value pairs defining updated limits.
  • DELETE - unregister specified components from the pool(s) 

...