...
ID | IEP-97 | ||||||||
Author | Anton Vinogradov | ||||||||
Sponsor | |||||||||
Created |
| ||||||||
Status |
|
Table of Contents |
---|
Customers may want to
...
Ignite supports Disk Compression and Transparent Data Encryption, but they are able to transform the data at the persistent layer only.
To cover both layers (network and memory) and make the feature compatible with the existing data, it is proposed to transform/restore CacheObject's bytes on the fly.
A possible solution is to transform the byte arrays they provided during the marshaling/unmarshalling phase. This will cover both layers, messaging (network) and storage (in-memory + persist).
GridBinaryMarshaller already transforms objects to bytes.
...
We need to cover all CacheObjects.
...
Code Block | ||||
---|---|---|---|---|
| ||||
protected byte[] valueBytesFromValue(CacheObjectValueContext ctx) throws IgniteCheckedException { byte[] bytes = ctx.kernalContext().cacheObjects().marshal(ctx, val); return CacheObjectTransformerUtils.transformIfNecessary(bytes, ctx); } protected Object valueFromValueBytes(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException { byte[] bytes = CacheObjectTransformerUtils.restoreIfNecessary(valBytes, ctx); return ctx.kernalContext().cacheObjects().unmarshal(ctx, bytes, ldr); } public void prepareMarshal(CacheObjectValueContext ctx) throws IgniteCheckedException { if (valBytes == null) valBytes = valueBytesFromValue(ctx); } public void finishUnmarshal(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException { if (val == null) val = valueFromValueBytes(ctx, ldr); } |
BinaryObject(Impl)s have different structures:
...
Code Block | ||||
---|---|---|---|---|
| ||||
private byte[] arrayFromValueBytes(CacheObjectValueContext ctx) { return CacheObjectTransformerUtils.restoreIfNecessary(valBytes, ctx); } private byte[] valueBytesFromArray(CacheObjectValueContext ctx) { return CacheObjectTransformerUtils.transformIfNecessary(arr, start, arr.length, ctx); } public void finishUnmarshal(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException { if (arr == null) arr = arrayFromValueBytes(ctx); } public void prepareMarshal(CacheObjectValueContext ctx) { if (valBytes == null) valBytes = valueBytesFromArray(ctx); } |
Some customers may want to encrypt the data, some to compress it, while some just keep it as is.
So, we must provide a simple way to append any transformation.
Code Block | ||||
---|---|---|---|---|
| ||||
public interface CacheObjectTransformerManager extends GridCacheSharedManager { /** * Transforms the data. * * @param original Original data. * @return Transformed data (started with {@link GridBinaryMarshaller#TRANSFORMED} when restorable) * or {@code null} when transformation is not possible/suitable. */ public @Nullable ByteBuffer transform(ByteBuffer original); /** * Restores the data. * * @param transformed Transformed data. * @return Restored data. */ public ByteBuffer restore(ByteBuffer transformed); } |
...
Code Block | ||||
---|---|---|---|---|
| ||||
IgniteConfiguration getConfiguration() { IgniteConfiguration cfg = ... cfg.setPluginProviders(new XXXPluginProvider()); // Which provides some XXXCacheObjectTransformerManager() return cfg; } |
Code Block | ||||
---|---|---|---|---|
| ||||
class CompressionTransformer extends CacheObjectTransformerAdapter { protected ByteBuffer transform(ByteBuffer original) throws IgniteCheckedException { int overhead = 5; // Transformed flag + length. int origSize = original.remaining(); int lim = origSize - overhead; if (lim <= 0) return null; // Compression is not profitable. ByteBuffer compressed = byteBuffer(overhead + (int)Zstd.compressBound(origSize)); compressed.put(TRANSFORMED); compressed.putInt(origSize); int size = Zstd.compress(compressed, original, 1); if (size >= lim) return null; // Compression is not profitable. compressed.flip(); return compressed; } protected ByteBuffer restore(ByteBuffer transformed) { ByteBuffer restored = byteBuffer(transformed.getInt()); Zstd.decompress(restored, transformed); restored.flip(); return restored; } } |
Code Block | ||||
---|---|---|---|---|
| ||||
class EncryptionTransformer extends CacheObjectTransformerAdapter { private static final int SHIFT = 42; // Secret! protected ByteBuffer transform(ByteBuffer original) throws IgniteCheckedException { ByteBuffer transformed = byteBuffer(original.remaining() + 1); // Same capacity is required. transformed.put(TRANSFORMED); while (original.hasRemaining()) transformed.put((byte)(original.get() + SHIFT)); transformed.flip(); return transformed; } protected ByteBuffer restore(ByteBuffer transformed, int length) { ByteBuffer restored = byteBuffer(transformed.remaining()); // Same size. while (transformed.hasRemaining()) restored.put((byte)(transformed.get() - SHIFT)); restored.flip(); return restored; } } |
Transformation requires additional memory allocation and subsequent GC work.
Transformation requires additional CPU utilization.
// Links to discussions on the devlist, if applicable.
// Links to various reference documents, if applicable.
Jira | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
|