Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

IDIEP-97
AuthorAnton Vinogradov 
Sponsor
Created

 

Created
Status

Status
colourGreen
titleactive

...

A possible solution is to transform the byte arrays they provided during the marshaling/unmarshalling phase. This will cover both layers, messaging (network) and storage (in-memory + persist).

Transformation

GridBinaryMarshaller already transforms objects to bytes. 

And, all we need is to transform and wrap these bytes.

For example,

  • 42 will be transformed to [3, 42, 0, 0, 0], where 3 is a GridBinaryMarshaller#INT
  • "Test string" will be transformed to [9, 11, 0, 0, 0, 84, 101, 115, 116, 32, 115, 116, 114, 105, 110, 103], where 9 is a GridBinaryMarshaller#STRING

and the idea is just to transform the given array somehow and add a special prefix GridBinaryMarshaller#TRANSFORMED == -3 at the beginning to make it distinguishable from untransformed data.

For example,

  • No-Op transformer will produce [-3, 3, 42, 0, 0, 0] or [-3, 9, 11, 0, 0, 0, 84, 101, 115, 116, 32, 115, 116, 114, 105, 110, 103].
  • Pseudo-Crypto transformer, which adds 1 to every original byte, will produce [-3, 4, 43, 1, 1, 1] or [-3, 10, 12, 1, 1, 1, 85, 102, 116, 117, 33, 116, 117, 115, 106, 111, 104]
  • Magic-Compressor will produce [-3, 7] or [-3, 17], where 7 and 17 are the result of a magic compression.

CacheObjects

We need All we need is to cover all CacheObjects.

...

Most of them has have the following structure:

Code Block
languagejava
titleXXX extends CacheObjectAdapter
 protected Object val; // Unmarshalled value.
 protected byte[] valBytes; // Marshalled value bytes.

...

Code Block
languagejava
titleCacheObjectAdapter transformation
protected byte[] valueBytesFromValue(CacheObjectValueContext ctx) throws IgniteCheckedException {
    byte[] bytes = ctx.kernalContext().cacheObjects().marshal(ctx, val);

    return CacheObjectsTransformerCacheObjectTransformerUtils.transformIfNecessary(bytes, ctx);
}


protected Object valueFromValueBytes(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException {
    byte[] bytes = CacheObjectsTransformerCacheObjectTransformerUtils.restoreIfNecessary(valBytes, ctx);

    return ctx.kernalContext().cacheObjects().unmarshal(ctx, bytes, ldr);
}

... 

public void prepareMarshal(CacheObjectValueContext ctx) throws IgniteCheckedException {
	if (valBytes == null)
		valBytes = valueBytesFromValue(ctx);
}

...

public void finishUnmarshal(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException { 
	if (val == null) 
    	val = valueFromValueBytes(ctx, ldr);
}

BinaryObjects

BinaryObject(Impl)s have the different structurestructures:

Code Block
languagejava
titleBinaryObjectImpl
private Object obj; // Deserialized value. Value converted to the Java class instance.
private byte[] arr; // Serialized bytes. Value!

(De)serialization is a simmilar similar to (un)marshalling, it's a process to gain java a Java class instance from bytes and or vice versa, but it happen happens at different time times and code layerlayers.

(Un)marshalling happens on putting/getting an object to/from the cache, but (de)serialization happens on building/deserializing of a binary object detached from any cache.

A In a lucky circumstance, BinaryObjectImpl require requires no marshalling, serialization already generates byte which bytes that can be used as marshalled bytes.

But, if we're going to transform the data during the marshaling/unmarshalling phase we need to add an additional data layer to the BinaryObjectImpl:

Code Block
languagejava
titleBinaryObjectImpl
private Object obj; // Deserialized value. Value converted to the Java class instance.
private byte[] arr; // Serialized bytes. Value!
private byte[] valBytes; // Marshalled value bytes.

Where valBytes == arr when the transformation is disabled.

It's not possible to just replace arr with valBytes because, unlike , for example, from CacheObjectImpl arr is not just a mashalled bytes, it's a an object's value requredthat is required, for example, to provide hashCode/schemaId/typeId/objectField, and we must keep it as is.

So, BinaryObjectImpl requres requires valBytes to/from arr conversion:

Code Block
languagejava
titleBinaryObjectImpl (un)marshalling
private byte[] arrayFromValueBytes(CacheObjectValueContext ctx) {
    return CacheObjectsTransformerCacheObjectTransformerUtils.restoreIfNecessary(valBytes, ctx);
}

private byte[] valueBytesFromArray(CacheObjectValueContext ctx) {
    return CacheObjectsTransformerCacheObjectTransformerUtils.transformIfNecessary(arr, start, arr.length, ctx);
}

...

public void finishUnmarshal(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException {
	if (arr == null)
		arr = arrayFromValueBytes(ctx);
}

...

public void prepareMarshal(CacheObjectValueContext ctx) {
	if (valBytes == null)
		valBytes = valueBytesFromArray(ctx);
}

...

Transformer

Some customets customers may want to encrypt the data, some to compress it, while some just keep it as is.

So, we must provide the a simple way to append any transformation.

API

Simplest way is to use Service Provider Interface (IgniteSpi):

Code Block
languagejava
titleSPIInterface
public interface CacheObjectTransformerSpiCacheObjectTransformerManager extends IgniteSpi {
    /** Additional space required to store the transformed data. */
    public int OVERHEAD = 6;

GridCacheSharedManager {
    /**
     * Transforms the data.
     *
     * @param bytes  Byte array contains the data.
     * @param offset Data offset.
     * @param length Data lengthoriginal Original data.
     * @return Byte array contains the transformedTransformed data (started with non-filled area with {@link #OVERHEADGridBinaryMarshaller#TRANSFORMED} size.when restorable)
     * @throws IgniteCheckedExceptionor {@code null} when transformation is not possible/suitable.
     */
    public byte[]@Nullable ByteBuffer transform(byte[] bytes, int offset, int length) throws IgniteCheckedExceptionByteBuffer original);

    /**
     * Restores the data.
     *
     * @param bytes  Byte array ending with the transformed data.
     * @param offset Transformed data offset.
     * @param length Original data length.
     * @return Byte array contains the restored@return Restored data.
     */
    public byte[]ByteBuffer restore(byte[] bytes, int offset, int lengthByteBuffer transformed);
}

This API known about overhead used to store transformed data and allowns to work with byte arrays with custom offsets, which is necessary to guarantee the performance.

Every customer may implement Every customey may impelent this interface in a proper way if necessary and specify it via plugin configuration:

Code Block
languagejava
titleCustom SPItransformer
IgniteConfiguration getConfiguration() {
	IgniteConfiguration cfg = ...

	cfg.setCacheObjectTransformSpisetPluginProviders(new XXXTransformerSpiXXXPluginProvider());

	return // Which provides some XXXCacheObjectTransformerManager()

    return cfg;
}

Simplified API

...

Examples

Compression example

Code Block
languagejava
titleCacheObjectTransformerSpiAdapterCompression
public abstract class CacheObjectTransformerSpiAdapterCompressionTransformer extends IgniteSpiAdapter implementsCacheObjectTransformerAdapter CacheObjectTransformerSpi {
...
    /**
     * Transforms the data.
	protected ByteBuffer transform(ByteBuffer original) throws IgniteCheckedException {             *
     * @param original Original data.
		int overhead = 5; // *Transformed @returnflag Transformed+ datalength.

     * @throws IgniteCheckedException when transformationint isorigSize not= possible/suitable.original.remaining();
     */
   int protectedlim abstract= ByteBufferorigSize transform(ByteBuffer original) throws IgniteCheckedException;

    /**
     * Restores the data.
     *- overhead;              

		if (lim <= 0)             
     * @param transformed Transformed data.
     * @param length Original data length.
     * @return Restored data.
 	return null; // Compression is not profitable.

    */
    protectedByteBuffer abstractcompressed ByteBuffer= restorebyteBuffer(ByteBufferoverhead transformed,+ int length);
}

Compression example

Code Block
languagejava
titleCompressionSpi
class CompressionTransformerSpi extends CacheObjectTransformerSpiAdapter {
    private static final LZ4Factory lz4Factory = LZ4Factory.fastestInstance();

    protected ByteBuffer transform(ByteBuffer original) throws IgniteCheckedException {
        int lim = original.remaining() - CacheObjectTransformerSpi.OVERHEAD;

        if (lim <= 0(int)Zstd.compressBound(origSize));    

		compressed.put(TRANSFORMED);
		compressed.putInt(origSize);    

		int size = Zstd.compress(compressed, original, 1);

 		if (size >= lim)
        	return null;   throw new IgniteCheckedException("// Compression is not profitable.");

           ByteBuffer compressed = byteBuffer(lim);

        Zstd.compress(compressed, original, 1);

        compressed   

		compressed.flip();             

        return compressed;
    }

    protected ByteBuffer restore(ByteBuffer transformed, int length) {
        ByteBuffer restored = byteBuffer(lengthtransformed.getInt());

        Zstd.decompress(restored, transformed);

        restored.flip();
              
        return restored;
    }
}

...

Code Block
languagejava
titleEncryptionSpiEncryption
class EncryptionTransformerSpiEncryptionTransformer extends CacheObjectTransformerSpiAdapterCacheObjectTransformerAdapter {
    private static final int SHIFT = 42; // Secret!

    protected ByteBuffer transform(ByteBuffer original) throws IgniteCheckedException {
        ByteBuffer transformed = byteBuffer(original.remaining() + 1); // Same capacity is required.

		transformed.put(TRANSFORMED);

        while (original.hasRemaining())
            transformed.put((byte)(original.get() + SHIFT));

        transformed.flip();

        return transformed;
    }

    protected ByteBuffer restore(ByteBuffer transformed, int length) {
        ByteBuffer restored = byteBuffer(lengthtransformed.remaining());

 //       Same size.
		
		while (transformed.hasRemaining())
            restored.put((byte)(transformed.get() - SHIFT));

        restored.flip();

        return restored;
    }
}

...