Proposal for pluggable cache stores

Overview

Apache geode provided following two types of persistence models.

1. persisting on local disk (shared nothing architecture each member writes to it own disk).
2. persistence on HDFS (removed from current develop branch).

But in the current Apache Geode architecture there is no support for plugin any other custom storage for persistence of cold data and keep operational data in memory. 
There are no direct interfaces provided which you can implement for your store and that store will be used for persisting/retrieving data.
We are trying to create a pluggable storage framework wherein you can attach any kind of storage, write/read in any format based on your storage.
Consistency and availability is managed by attached store and not by Apache Geode. 

Proposal Details:

We are proposing a pluggable storage interface which  can be implemented by any application and they will be able to push and retrieve the data from that store.
Following major interfaces we are adding for pluggability.

1. CacheStoreManager : Manages all the cache stores. Creation/Removal/modification to the cache store is done by CacheStoreManager.

   CacheStoreManager

   /**
    * Manages all the cache stores
    */
   public interface CacheStoreManager {
     /**
      * Creates the cachestore using Cache Store Factory
      * @param cacheStoreFactory factory used for creating cache store
      * @param cacheStoreConfig configuration associated with cache store
      * @return the cachestore instance created
      */
     CacheStore createCacheStore(CacheStoreFactory cacheStoreFactory, CacheStoreConfig cacheStoreConfig);
   
     /**
      * Gets the cache store instance using the store name
      * @param storeName name of cachestore
      * @return
      */
     CacheStore getCacheStore(String storeName);
   
     /**
      * removes the cache store
      * @param storeName
      */
     void removeCacheStore(String storeName);
   
     /**
      * closes all the cache stores associated with this cache
      */
     void closeAllCacheStores();
   
     /**
      * gets all the cache stores associated with this cache
      * @return cachestore set
      */
     Set<CacheStore> getAllCacheStores();
   
     /**
      * Gets the statistics associated with cache store
      * @param storeName stats for cachestore
      * @return
      */
     CacheStoreStats getCacheStoreStats(String storeName);
   }
   
   

2. CacheStore:  

   CacheStore

   /**
    * Cache stores provide a means of persisting data on configured store, e,g HBase, Hive etc..
    * There can be multiple instance of Cache stores in a cluster.
    * The regions connected using a Cache store will share the same store persistence attributes.
    * <p/>
    */
   public interface CacheStore {
   
     /**
      * Permanently deletes all resources (tables, files etc) associated with this {@link CacheStore}.
      * This operation will fail if any region is still using this store for persistence.
      *
      * @throws IllegalStateException if any region using this Store still exists
      */
     void destroy(Region region) throws CacheStoreException;
   
     /*
      * initialize a cacheStore, this includes creating a corresponding entity (table incase of HBase) in a store associated with this region name.
      */
     void initialize(Region region) throws CacheStoreException;
   
     /**
      * Identifies attributes configured in {@link CacheStoreMutator} and applies
      * the new attribute values to this instance of {@link CacheStore} dynamically.
      * Any property which is not set in {@link CacheStoreMutator} remains
      * unaltered. In most cases altering the attributes does not cause existing
      * operations to terminate. The altered attributes are used in the next cycle
      * of the operation they impact.
      *
      * @return cacheStore reference representing the old {@link CacheStore}
      */
     CacheStore alter(CacheStoreMutator mutator);
   
     /**
      * Retrieves the cache store configurations
      *
      * @return Cache store config {@link CacheStoreConfig}
      */
     CacheStoreConfig getConfig();
   
    
     /**
      * Retrieves the cache store configurations
      *
      * @return Cache store config {@link CacheStoreStats}
      */
     CacheStoreStats getCacheStoreStats();
   
     /**
      * Save new KV pair as objects or if already exist in  store
      * @param region Region on which operation is done
      * @param key Row key
      * @param val Row value byte array
      * @param isUpdate update an existing row in store.
      * @exception CacheStoreException;
      */
     void persist(Region region, Object key, Object val, boolean isUpdate) throws CacheStoreException;
   
     /**
      * Remove existing KV from cache store
      * @param key value
      * @return Object; gives back the old value
      * @exception CacheStoreException;
      */
     Object remove(Region region, Object key) throws CacheStoreException;
   
   
     /**
      * Gives back the value from the cache store
      * @return Object val;
      * @exception CacheStoreException;
      */
     Object get(Region region,Object key) throws CacheStoreException;
   
   
     /**
      * Clears the complete CacheStore
      * @exception CacheStoreException;
      */
     void clear(Region region) throws CacheStoreException;
   
     /**
      * Close cache Store
      */
     void close();
   
     /**
      * destroy CacheStore
      */
     void destroyCacheStore();
   
   }
   
   

3. CacheStoreFactory : Used for creating cache store

   CacheStoreFactory

   /**
    * Factory for creating cache store
    */
   public interface CacheStoreFactory implements Serializable{
     /**
      * Create cache store
      * @param cache geode cache 
      * @param cacheStoreConfig configurations associated with cachestore
      * @return
      */
       CacheStore create(Cache cache, CacheStoreConfig cacheStoreConfig);
   }
   
   

Usage:
CacheStore creation and attaching the cache store to the region is done as follows

//create the cache
CacheFactory cacheFactory = new CacheFactory();
Cache cache = cacheFactory.create();

//create cachestore factory and cache store configurations
CacheStoreFactory cacheStoreFactory = new CacheStoreFactory();
CacheStoreConfig cacheStoreConfig = CacheStoreConfig.create();

//create the cache store
CacheStoreManager cacheStoreManager = cache.getCacheStoreManager();
CacheStore cacheStore = cacheStoreManager.createCacheStore(cacheStoreFactory, cacheStoreConfig);

//Attach cachestore to region
RegionFactory<Object, Object> regionFactory = cache.createRegionFactory(RegionShortcut.PARTITION);
regionFactory.setCacheStore(cacheStore);
Region<Object, Object> cacheStoreregion = regionFactory.create("cacheStoreregion");

Implementation details:

CacheStore pluggabilitry is achieved using readthrough and writebehind concepts in geode. 
In geode we can create a asynceventlistener and attach that to the region. We are using the same asyncevent queue framework for persistence of data in cache store.
For retrieval of data, if there is miss in the cache then we are retrieving it from cache store.

Sequence diagram for put/get with attached cachestore

Limitations:

1. CacheStores are supported only for partitioned regions.