Page History

...

...

A possible solution is to transform the byte arrays they provided during the marshaling/unmarshalling phase. This will cover both layers, messaging (network) and storage (in-memory + persist).

Transformation

All we need is to cover all CacheObjects.

GridBinaryMarshaller already transforms objects to bytes. 

And, all we need is to transform and wrap these bytes.

For example,

• 42 will be transformed to [3, 42, 0, 0, 0], where 3 is a GridBinaryMarshaller#INT
• "Test string" will be transformed to [9, 11, 0, 0, 0, 84, 101, 115, 116, 32, 115, 116, 114, 105, 110, 103], where 9 is a GridBinaryMarshaller#STRING

and the idea is just to transform the given array somehow and add a special prefix GridBinaryMarshaller#TRANSFORMED == -3 at the beginning to make it distinguishable from untransformed data.

For example,

• No-Op transformer will produce [-3, 3, 42, 0, 0, 0] or [-3, 9, 11, 0, 0, 0, 84, 101, 115, 116, 32, 115, 116, 114, 105, 110, 103].
• Pseudo-Crypto transformer, which adds 1 to every original byte, will produce [-3, 4, 43, 1, 1, 1] or [-3, 10, 12, 1, 1, 1, 85, 102, 116, 117, 33, 116, 117, 115, 106, 111, 104]
• Magic-Compressor will produce [-3, 7] or [-3, 17], where 7 and 17 are the result of a magic compression.

CacheObjects

We need to cover all CacheObjects.

...

Most of them have the following structure:

...

protected byte[] valueBytesFromValue(CacheObjectValueContext ctx) throws IgniteCheckedException {
    byte[] bytes = ctx.kernalContext().cacheObjects().marshal(ctx, val);

    return CacheObjectTransformerCacheObjectTransformerUtils.transformIfNecessary(bytes, ctx);
}


protected Object valueFromValueBytes(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException {
    byte[] bytes = CacheObjectTransformerCacheObjectTransformerUtils.restoreIfNecessary(valBytes, ctx);

    return ctx.kernalContext().cacheObjects().unmarshal(ctx, bytes, ldr);
}


public void prepareMarshal(CacheObjectValueContext ctx) throws IgniteCheckedException {
	if (valBytes == null)
		valBytes = valueBytesFromValue(ctx);
}


public void finishUnmarshal(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException { 
	if (val == null) 
    	val = valueFromValueBytes(ctx, ldr);
}

...

private Object obj; // Deserialized value. Value converted to the Java class instance.
private byte[] arr; // Serialized bytes. Value!

(De)serialization is similar to (un)marshalling, it's a process to gain java a Java class instance from bytes and or vice versa, but it happens at different times and code layers.

...

In a lucky circumstance, BinaryObjectImpl requires no marshalling, serialization already generates byte which bytes that can be used as marshalled bytes.

...

private Object obj; // Deserialized value. Value converted to the Java class instance.
private byte[] arr; // Serialized bytes. Value!
private byte[] valBytes; // Marshalled value bytes.

...

It's not possible to just replace arr with valBytes because, unlike , for example, from CacheObjectImpl arr is not just a mashalled bytes, it's an object's value that is required, for example, to provide hashCode/schemaId/typeId/objectField, and we must keep it as is.

So, BinaryObjectImpl requires valBytes to/from arr conversion:

private byte[] arrayFromValueBytes(CacheObjectValueContext ctx) {
    return CacheObjectTransformerCacheObjectTransformerUtils.restoreIfNecessary(valBytes, ctx);
}

private byte[] valueBytesFromArray(CacheObjectValueContext ctx) {
    return CacheObjectTransformerCacheObjectTransformerUtils.transformIfNecessary(arr, start, arr.length, ctx);
}


public void finishUnmarshal(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException {
	if (arr == null)
		arr = arrayFromValueBytes(ctx);
}


public void prepareMarshal(CacheObjectValueContext ctx) {
	if (valBytes == null)
		valBytes = valueBytesFromArray(ctx);
}

...

Transformer

Some customers may want to encrypt the data, some to compress it, while some just keep it as is.

So, we must provide a simple way to append any transformation.

API

...

Code Block
language java title SPIInterface
public interface CacheObjectTransformerSpiCacheObjectTransformerManager extends IgniteSpi { /** Additional space required to store the transformed data. */ public int OVERHEAD = 6; GridCacheSharedManager { /** * Transforms the data. * * @param bytes Byte array contains the original Original data. * @param offset Data offset. * @param length Data length. * @return Byte array contains the transformed Transformed data (started with non-filled area with {@link #OVERHEADGridBinaryMarshaller#TRANSFORMED} size.when restorable) * @throws IgniteCheckedExceptionor {@code null} when transformation is not possible/suitable. */ public byte[]@Nullable ByteBuffer transform(byte[] bytes, int offset, int length) throws IgniteCheckedExceptionByteBuffer original); /** * Restores the data. * * @param bytes Byte array ending with the transformed data. * @param offset Transformed data offset. * @param length Original data length. * @return Byte array contains the restored Restored data. */ public byte[]ByteBuffer restore(byte[] bytes, int offset, int lengthByteBuffer transformed); }

...

Every customer may implement this interface in a proper way if necessary and specify it via plugin configuration:

Code Block
language java title Custom SPItransformer
IgniteConfiguration getConfiguration() { IgniteConfiguration cfg = ... cfg.setCacheObjectTransformerSpisetPluginProviders(new XXXTransformerSpiXXXPluginProvider()); return // Which provides some XXXCacheObjectTransformerManager()    return cfg; }

Simplified API

...

Examples

Compression example

public abstract class CacheObjectTransformerSpiAdapterCompressionTransformer extends IgniteSpiAdapterCacheObjectTransformerAdapter implements CacheObjectTransformerSpi {
...
    /**
     * Transforms the data.
	protected ByteBuffer transform(ByteBuffer original) throws IgniteCheckedException {               *
     * @param original Original data.
     * @return Transformed data.
		int overhead = 5; // Transformed flag + length.

     * @throws IgniteCheckedException whenint transformationorigSize is not possible/suitable.= original.remaining();
     */
   int protectedlim abstract= ByteBufferorigSize transform(ByteBuffer original) throws IgniteCheckedException;

    /**
     * Restores the data.
     *- overhead;              

		if (lim <= 0)             
     * @param transformed Transformed data.
     * @param length Original data length.
     * @return Restored data.	return null; // Compression is not profitable.

     */
   ByteBuffer protectedcompressed abstract= ByteBuffer restorebyteBuffer(ByteBufferoverhead transformed,+ int length);
}

Compression example

class CompressionTransformerSpi extends CacheObjectTransformerSpiAdapter {
    private static final LZ4Factory lz4Factory = LZ4Factory.fastestInstance();

    protected ByteBuffer transform(ByteBuffer original) throws IgniteCheckedException {
        int lim = original.remaining() - CacheObjectTransformerSpi.OVERHEAD;

        if (lim <= 0(int)Zstd.compressBound(origSize));    

		compressed.put(TRANSFORMED);
		compressed.putInt(origSize);    

		int size = Zstd.compress(compressed, original, 1);

 		if (size >= lim)
        	return null;   throw new IgniteCheckedException("// Compression is not profitable.");

           ByteBuffer compressed = byteBuffer(lim);

        Zstd.compress(compressed, original, 1);

        compressed   

		compressed.flip();             

        return compressed;
    }

    protected ByteBuffer restore(ByteBuffer transformed, int length) {
        ByteBuffer restored = byteBuffer(lengthtransformed.getInt());

        Zstd.decompress(restored, transformed);

        restored.flip();
              
        return restored;
    }
}

...

class EncryptionTransformerSpiEncryptionTransformer extends CacheObjectTransformerSpiAdapterCacheObjectTransformerAdapter {
    private static final int SHIFT = 42; // Secret!

    protected ByteBuffer transform(ByteBuffer original) throws IgniteCheckedException {
        ByteBuffer transformed = byteBuffer(original.remaining() + 1); // Same capacity is required.

		transformed.put(TRANSFORMED);

        while (original.hasRemaining())
            transformed.put((byte)(original.get() + SHIFT));

        transformed.flip();

        return transformed;
    }

    protected ByteBuffer restore(ByteBuffer transformed, int length) {
        ByteBuffer restored = byteBuffer(lengthtransformed.remaining());

 //       Same size.
		
		while (transformed.hasRemaining())
            restored.put((byte)(transformed.get() - SHIFT));

        restored.flip();

        return restored;
    }
}

...