Page History

...

...

GridBinaryMarshaller already transforms objects to the bytes. 

And, all we need is to transform and wrap these bytes.

For example,

• int=42 will be transformed to [3, 42, 0, 0, 0], where 3 is a GridBinaryMarshaller#INT
• "Test string" will be transformed to [9, 11, 0, 0, 0, 84, 101, 115, 116, 32, 115, 116, 114, 105, 110, 103], where 9 is a GridBinaryMarshaller#STRING

and the idea is just to transform the given array somehow and add a special prefix GridBinaryMarshaller#TRANSFORMED == -1 at 3 at the beginning to make it distinguishable from untransformed data.

For example,

• No-op Op transformer will produce [-1, 0, 0, 0, 0, 5, 3, 3, 42, 0, 0, 0] or [-1, 0, 0, 0, 0, 16, 3, 9, 11, 0, 0, 0, 84, 101, 115, 116, 32, 115, 116, 114, 105, 110, 103], where [0, 0, 0, 0, 5] and [0, 0, 0, 0, 16] is an original array size.
• Pseudo-Crypto transformer, which adds 1 to every original byte, will produce will produce [-1, 0, 0, 0, 0, 6, 3, 4, 43, 1, 1, 1] or [-1, 0, 0, 0, 0, 163, 10, 12, 1, 1, 1, 85, 102, 116, 117, 33, 116, 117, 115, 106, 111, 104]
• Magic-Compressor will produce [-1, 0, 0, 0, 0, 53, 7] or [-1, 0, 0, 0, 0, 163, 17], where 7 and 17 is a are the result of a magic compression.

CacheObjects

We need is to cover all CacheObjects.

...

protected byte[] valueBytesFromValue(CacheObjectValueContext ctx) throws IgniteCheckedException {
    byte[] bytes = ctx.kernalContext().cacheObjects().marshal(ctx, val);

    return CacheObjectTransformerCacheObjectTransformerUtils.transformIfNecessary(bytes, ctx);
}


protected Object valueFromValueBytes(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException {
    byte[] bytes = CacheObjectTransformerCacheObjectTransformerUtils.restoreIfNecessary(valBytes, ctx);

    return ctx.kernalContext().cacheObjects().unmarshal(ctx, bytes, ldr);
}


public void prepareMarshal(CacheObjectValueContext ctx) throws IgniteCheckedException {
	if (valBytes == null)
		valBytes = valueBytesFromValue(ctx);
}


public void finishUnmarshal(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException { 
	if (val == null) 
    	val = valueFromValueBytes(ctx, ldr);
}

...

It's not possible to just replace arr with valBytes because, unlike , for example, from CacheObjectImpl arr is not just a mashalled bytes, it's an object's value that is required, for example, to provide hashCode/schemaId/typeId/objectField, and we must keep it as is.

...

private byte[] arrayFromValueBytes(CacheObjectValueContext ctx) {
    return CacheObjectTransformerCacheObjectTransformerUtils.restoreIfNecessary(valBytes, ctx);
}

private byte[] valueBytesFromArray(CacheObjectValueContext ctx) {
    return CacheObjectTransformerCacheObjectTransformerUtils.transformIfNecessary(arr, start, arr.length, ctx);
}


public void finishUnmarshal(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException {
	if (arr == null)
		arr = arrayFromValueBytes(ctx);
}


public void prepareMarshal(CacheObjectValueContext ctx) {
	if (valBytes == null)
		valBytes = valueBytesFromArray(ctx);
}

...

Transformer

Some customers may want to encrypt the data, some to compress it, while some just keep it as is.

So, we must provide a simple way to append any transformation.

API

...

Code Block
language java title SPIInterface
public interface CacheObjectTransformerSpiCacheObjectTransformerManager extends IgniteSpi { /** Additional space required to store the transformed data. */ public int OVERHEAD = 6; GridCacheSharedManager { /** * Transforms the data. * * @param bytes Byte array contains the original Original data. * @param offset Data offset. * @param length Data length. * @return Byte array contains the transformed@return Transformed data started with non-filled area(started with {@link #OVERHEADGridBinaryMarshaller#TRANSFORMED} when size.restorable) * or @throws{@code IgniteCheckedExceptionnull} when transformation is not possible/suitable. */ public @Nullable byte[]ByteBuffer transform(byte[] bytes, int offset, int length) throws IgniteCheckedExceptionByteBuffer original); /** * Restores the data. * * @param bytes Byte array ending with the transformed data. * @param offset Transformed data offset. * @param length Original data length. * @return Byte array contains the restored @return Restored data. */ public byte[]ByteBuffer restore(byte[] bytes, int offset, int lengthByteBuffer transformed); }

This API is known for the overhead used to store transformed data and is able to work with byte arrays with custom offsets, which is necessary to guarantee performance.

Every customer may implement this interface in a proper way if necessary and specify it in the via plugin configuration:

Code Block
language java title Custom SPItransformer
IgniteConfiguration getConfiguration() { IgniteConfiguration cfg = ... cfg.setCacheObjectTransformerSpisetPluginProviders(new XXXTransformerSpiXXXPluginProvider()); return // Which provides some XXXCacheObjectTransformerManager()    return cfg; }

Simplified API

...

Examples

Compression example

public abstract class CacheObjectTransformerSpiAdapterCompressionTransformer extends IgniteSpiAdapter implements CacheObjectTransformerSpiCacheObjectTransformerAdapter {
...
    /**
     * Transforms the data.
 	protected ByteBuffer transform(ByteBuffer original) throws IgniteCheckedException {            *
     * @param original Original data.
		int overhead = 5; // *Transformed @returnflag Transformed+ datalength.

     * @throws IgniteCheckedException when transformationint isorigSize not= possible/suitable.original.remaining();
     */
   int protectedlim abstract= ByteBufferorigSize transform(ByteBuffer original) throws IgniteCheckedException;

    /**
     * Restores the data.
     *- overhead;              

		if (lim <= 0)             
     * @param transformed Transformed data.
     * @param length Original data length.
     * @return Restored data.	return null; // Compression is not profitable.

     */
   ByteBuffer protectedcompressed abstract= ByteBuffer restorebyteBuffer(ByteBufferoverhead transformed,+ int length);
}

Compression example

class CompressionTransformerSpi extends CacheObjectTransformerSpiAdapter {
    private static final LZ4Factory lz4Factory = LZ4Factory.fastestInstance();

    protected ByteBuffer transform(ByteBuffer original) throws IgniteCheckedException {
        int lim = original.remaining() - CacheObjectTransformerSpi.OVERHEAD;

        if (lim <= 0(int)Zstd.compressBound(origSize));    

		compressed.put(TRANSFORMED);
		compressed.putInt(origSize);    

		int size = Zstd.compress(compressed, original, 1);

 		if (size >= lim)
        	return null;   throw new IgniteCheckedException("// Compression is not profitable.");

           ByteBuffer compressed = byteBuffer(lim);

        Zstd.compress(compressed, original, 1);

           

		compressed.flip();             

        return compressed;
    }

    protected ByteBuffer restore(ByteBuffer transformed, int length) {
        ByteBuffer restored = byteBuffer(lengthtransformed.getInt());

        Zstd.decompress(restored, transformed);

        restored.flip();
              
        return restored;
    }
}

...

class EncryptionTransformerSpiEncryptionTransformer extends CacheObjectTransformerSpiAdapterCacheObjectTransformerAdapter {
    private static final int SHIFT = 42; // Secret!

    protected ByteBuffer transform(ByteBuffer original) throws IgniteCheckedException {
        ByteBuffer transformed = byteBuffer(original.remaining() + 1); // Same capacity is required.

		transformed.put(TRANSFORMED);

        while (original.hasRemaining())
            transformed.put((byte)(original.get() + SHIFT));

        transformed.flip();

        return transformed;
    }

    protected ByteBuffer restore(ByteBuffer transformed, int length) {
        ByteBuffer restored = byteBuffer(lengthtransformed.remaining());

 //       Same size.
		
		while (transformed.hasRemaining())
            restored.put((byte)(transformed.get() - SHIFT));

        restored.flip();

        return restored;
    }
}

...