...

Motivation

...

A possible solution is to transform the byte arrays they provided during the marshaling/unmarshalling phase. This will cover both layers, messaging (network) and storage (in-memory + persist).

Transformation

All we need is to cover all CacheObjects.

CacheObjects

GridBinaryMarshaller already transforms objects to bytes. 

And, all we need is to transform and wrap these bytes.

For example,

• 42 will be transformed to [3, 42, 0, 0, 0], where 3 is a GridBinaryMarshaller#INT
• "Test string" will be transformed to [9, 11, 0, 0, 0, 84, 101, 115, 116, 32, 115, 116, 114, 105, 110, 103], where 9 is a GridBinaryMarshaller#STRING

and the idea is just to transform the given array somehow and add a special prefix GridBinaryMarshaller#TRANSFORMED == -3 at the beginning to make it distinguishable from untransformed data.

For example,

• No-Op transformer will produce [-3, 3, 42, 0, 0, 0] or [-3, 9, 11, 0, 0, 0, 84, 101, 115, 116, 32, 115, 116, 114, 105, 110, 103].
• Pseudo-Crypto transformer, which adds 1 to every original byte, will produce [-3, 4, 43, 1, 1, 1] or [-3, 10, 12, 1, 1, 1, 85, 102, 116, 117, 33, 116, 117, 115, 106, 111, 104]
• Magic-Compressor will produce [-3, 7] or [-3, 17], where 7 and 17 are the result of a magic compression.

CacheObjects

We need to cover all CacheObjects.

Most of them have Most of them has the following structure:

 protected Object val; // Unmarshalled value.
 protected byte[] valBytes; // Marshalled value bytes.

...

protected byte[] valueBytesFromValue(CacheObjectValueContext ctx) throws IgniteCheckedException {
    byte[] bytes = ctx.kernalContext().cacheObjects().marshal(ctx, val);

    return CacheObjectsTransformerCacheObjectTransformerUtils.transformIfNecessary(bytes, ctx);
}


protected Object valueFromValueBytes(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException {
    byte[] bytes = CacheObjectsTransformerCacheObjectTransformerUtils.restoreIfNecessary(valBytes, ctx);

    return ctx.kernalContext().cacheObjects().unmarshal(ctx, bytes, ldr);
}

... 

public void prepareMarshal(CacheObjectValueContext ctx) throws IgniteCheckedException {
	if (valBytes == null)
		valBytes = valueBytesFromValue(ctx);
}

...

public void finishUnmarshal(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException { 
	if (val == null) 
    	val = valueFromValueBytes(ctx, ldr);
}

BinaryObjects

BinaryObject(Impl)s have the different structurestructures:

private Object obj; // Deserialized value. Value converted to the Java class instance.
private byte[] arr; // Serialized bytes. Value!

(De)serialization is a simmilar similar to (un)marshalling, it's a process to gain java a Java class instance from bytes and or vice versa, but it happen happens at different time times and code layerlayers.

(Un)marshalling happens on putting/getting an object to/from the cache, but (de)serialization happens on building/deserializing of a binary object detached from any cache.

A In a lucky circumstance, BinaryObjectImpl require requires no marshalling, serialization already generates byte which bytes that can be used as marshalled bytes.

But, if we're going to transform the data during the marshaling/unmarshalling phase we need to add an additional data layer to the BinaryObjectImpl:

private Object obj; // Deserialized value. Value converted to the Java class instance.
private byte[] arr; // Serialized bytes. Value!
private byte[] valBytes; // Marshalled value bytes.

Where valBytes == arr when the transformation is disabled.

It's not possible to just replace arr with valBytes because, unlike , for example, from CacheObjectImpl arr is not just a mashalled bytes, it's a an object's value requredthat is required, for example, to provide hashCode/schemaId/typeId/objectField, and we must keep it as is.

So, BinaryObjectImpl requres requires valBytes to/from arr conversion:

private byte[] arrayFromValueBytes(CacheObjectValueContext ctx) {
    return CacheObjectsTransformerCacheObjectTransformerUtils.restoreIfNecessary(valBytes, ctx);
}

private byte[] valueBytesFromArray(CacheObjectValueContext ctx) {
    return CacheObjectsTransformerCacheObjectTransformerUtils.transformIfNecessary(arr, start, arr.length, ctx);
}

...

public void finishUnmarshal(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException {
	if (arr == null)
		arr = arrayFromValueBytes(ctx);
}

...

public void prepareMarshal(CacheObjectValueContext ctx) {
	if (valBytes == null)
		valBytes = valueBytesFromArray(ctx);
}

...

Transformer

Some customets customers may want to encrypt the data, some to compress it, while some just keep it as is.

So, we must provide the a simple way to append any transformation.

API

...

Code Block
language java title SPIInterface
public interface CacheObjectTransformerSpiCacheObjectTransformerManager extends IgniteSpiGridCacheSharedManager { /** Additional space required to* storeTransforms the transformed data. */ public* int@param OVERHEADoriginal =Original 6;data. /*** @return Transformed data (started with {@link GridBinaryMarshaller#TRANSFORMED} when restorable) * Transforms the data or {@code null} when transformation is not possible/suitable. */ public @Nullable ByteBuffer transform(ByteBuffer original); /** @param bytes Byte array* containsRestores the data. * @param offset Data offset. * @param lengthtransformed DataTransformed lengthdata. * @return Byte array contains the transformed data started with non-filled area with {@link #OVERHEAD} size. * @throws IgniteCheckedException when transformation is not possible/suitable. */ public byte[] transform(byte[] bytes, int offset, int length) throws IgniteCheckedException; /** * Restores the data. * * @param bytes Byte array ending with the transformed data. * @param offset Transformed data offset. * @param length Original data length. * @return Byte array contains the restored data. */ public byte[] restore(byte[] bytes, int offset, int length); }

This API known about overhead used to store transformed data and allowns to work with byte arrays with custom offsets, which is necessary to guarantee the performance.

Every customey may impelent this interface in proper way if necessary.

Simplified API

But, most customers just want to thansform the data, so, they may extend the adapter with the simple API.

public abstract class CacheObjectTransformerSpiAdapter extends IgniteSpiAdapter implements CacheObjectTransformerSpi {
...
    /**
     * Transforms the data.
     *
     * @param original Original data.
     * @return Transformed data.
     * @throws IgniteCheckedException when transformation is not possible/suitable.
     */
    protected abstract ByteBuffer transform(ByteBuffer original) throws IgniteCheckedException;

    /**
     * Restores the data.
     *
     * @param transformed Transformed data.
     * @param length Original data length.
     * @return Restored data.
     */
    protected abstract ByteBuffer restore(ByteBuffer transformed, int length);
}

...

Restored data.
     */
    public ByteBuffer restore(ByteBuffer transformed);
}

Every customer may implement this interface in a proper way if necessary and specify it via plugin configuration:

IgniteConfiguration getConfiguration() {
	IgniteConfiguration cfg = ...

	cfg.setPluginProviders(new XXXPluginProvider()); // Which provides some XXXCacheObjectTransformerManager()

    return cfg;
}

Examples

Compression example

class CompressionTransformer extends CacheObjectTransformerAdapter {
	protected ByteBuffer transform(ByteBuffer original) throws IgniteCheckedException {                     
		int overhead = 5; // Transformed flag + length.

        int origSize = original.remaining();
        int lim = origSize - overhead;              

		if (lim <= 0)             
        	return null; // Compression is not profitable.

        ByteBuffer compressed = byteBuffer(overhead + (int)Zstd.compressBound(origSize));    

		compressed.put(TRANSFORMED);
		compressed.putInt(origSize);    

		int size = Zstd.compress(compressed, original, 1);

 		if (size >= lim)
        	return null; // Compression is not profitable.          

		compressed.flip();          

        return compressed;
    }

    protected ByteBuffer restore(ByteBuffer transformed) {
        ByteBuffer restored = byteBuffer(transformed.getInt());

        Zstd.decompress(restored, transformed);

        restored.flip();
              
        return restored;
    }
}

Encryption example

protected static class CompressionTransformerSpiEncryptionTransformer extends CacheObjectTransformerSpiAdapterCacheObjectTransformerAdapter {
    private static final LZ4Factoryint lz4FactorySHIFT = LZ4Factory.fastestInstance();42; // Secret!

    protected ByteBuffer transform(ByteBuffer original) throws IgniteCheckedException {
        intByteBuffer limtransformed = byteBuffer(original.remaining() - CacheObjectTransformerSpi.OVERHEAD;

        if (lim <= 0)+ 1); // Same capacity is required.

		transformed.put(TRANSFORMED);

            throw new IgniteCheckedException("Compression is not profitable.");

while (original.hasRemaining())
         ByteBuffer compressed = byteBuffer(lim);

        Zstd.compress(compressed, original, 1transformed.put((byte)(original.get() + SHIFT));

        compressedtransformed.flip();             

        return compressedtransformed;
    }

    protected ByteBuffer restore(ByteBuffer transformed, int length) {
        ByteBuffer restored = byteBuffer(lengthtransformed.remaining()); // Same size.
		
		while (transformed.hasRemaining())
       Zstd.decompress(restored, transformed     restored.put((byte)(transformed.get() - SHIFT));

        restored.flip();
              
        return restored;
    }
}

Encryption example

Risks and Assumptions

Transformation requires additional memory allocation and subsequent GC work.

...