Page History

...

ID

IEP-97

Author

Sponsor

Created

22 Nov 2022

Status


colour	Green
title	active

...

No-Op transformer will produce [-3, 0, 3, 42, 0, 0, 0] or [-3, 0, 9, 11, 0, 0, 0, 84, 101, 115, 116, 32, 115, 116, 114, 105, 110, 103], where 0 is a protocol version.
Pseudo-Crypto transformer, which adds 1 to every original byte, will produce [-3, 0, 4, 43, 1, 1, 1] or [-3, 0, 10, 12, 1, 1, 1, 85, 102, 116, 117, 33, 116, 117, 115, 106, 111, 104]
Magic-Compressor will produce [-3, 0, 7] or [-3, 0, 17], where 7 and 17 are the result of a magic compression.

...

Code Block

language	java
title	CacheObjectAdapter transformation

protected byte[] valueBytesFromValue(CacheObjectValueContext ctx) throws IgniteCheckedException {
    byte[] bytes = ctx.kernalContext().cacheObjects().marshal(ctx, val);

    return CacheObjectTransformerCacheObjectTransformerUtils.transformIfNecessary(bytes, ctx);
}


protected Object valueFromValueBytes(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException {
    byte[] bytes = CacheObjectTransformerCacheObjectTransformerUtils.restoreIfNecessary(valBytes, ctx);

    return ctx.kernalContext().cacheObjects().unmarshal(ctx, bytes, ldr);
}


public void prepareMarshal(CacheObjectValueContext ctx) throws IgniteCheckedException {
	if (valBytes == null)
		valBytes = valueBytesFromValue(ctx);
}


public void finishUnmarshal(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException { 
	if (val == null) 
    	val = valueFromValueBytes(ctx, ldr);
}

...

It's not possible to just replace arr with valBytes because, unlike , for example, from CacheObjectImpl arr is not just a mashalled bytes, it's an object's value that is required, for example, to provide hashCode/schemaId/typeId/objectField, and we must keep it as is.

...

Code Block

language	java
title	BinaryObjectImpl (un)marshalling

private byte[] arrayFromValueBytes(CacheObjectValueContext ctx) {
    return CacheObjectTransformerCacheObjectTransformerUtils.restoreIfNecessary(valBytes, ctx);
}

private byte[] valueBytesFromArray(CacheObjectValueContext ctx) {
    return CacheObjectTransformerCacheObjectTransformerUtils.transformIfNecessary(arr, start, arr.length, ctx);
}


public void finishUnmarshal(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException {
	if (arr == null)
		arr = arrayFromValueBytes(ctx);
}


public void prepareMarshal(CacheObjectValueContext ctx) {
	if (valBytes == null)
		valBytes = valueBytesFromArray(ctx);
}

...

Transformer

Some customers may want to encrypt the data, some to compress it, while some just keep it as is.

So, we must provide a simple way to append any transformation.

API

The simplest way is to use Service Provider Interface (IgniteSpi):

Code Block

language	java
title	SPIInterface

public interface CacheObjectTransformerSpiCacheObjectTransformerManager extends IgniteSpiGridCacheSharedManager {
    /** Additional space required to store the transformed data. */
    public int OVERHEAD = 2;

    /**
     * Transforms the data.
     *
     * @param bytes  Byte array contains the data.
     * @param offset Data offset.
     * @param length Data lengthoriginal Original data.
     * @return Byte array contains the transformed Transformed data (started with non-filled area with {@link #OVERHEADGridBinaryMarshaller#TRANSFORMED} when size.restorable)
     * or @throws{@code IgniteCheckedExceptionnull} when transformation is not possible/suitable.
     */
    public @Nullable byte[]ByteBuffer transform(byte[] bytes, int offset, int length) throws IgniteCheckedException;      ByteBuffer original);

    /**
     * Restores the data.
     *
     * @param bytestransformed  Byte array ending with the transformed Transformed data.
     * @param offset Data offset.
     * @param length Data length.
     * @return Byte array contains the restored Restored data.
     */
    public byte[]ByteBuffer restore(byte[] bytes, int offset, int lengthByteBuffer transformed);
}

This API is known for the overhead used to store transformed data and is able to work with byte arrays with custom offsets, which is necessary to guarantee performance.

Every customer may implement this interface in a proper way if necessary and specify it in the via plugin configuration:

Code Block

language	java
title	Custom SPItransformer

IgniteConfiguration getConfiguration() {
	IgniteConfiguration cfg = ...

	cfg.setCacheObjectTransformerSpisetPluginProviders(new XXXTransformerSpiXXXPluginProvider());

	return // Which provides some XXXCacheObjectTransformerManager()

    return cfg;
}

Simplified API

...

Examples

Compression example

Code Block

language	java
title	CacheObjectTransformerSpiAdapterCompression

public abstract class CacheObjectTransformerSpiAdapterCompressionTransformer extends IgniteSpiAdapterCacheObjectTransformerAdapter implements CacheObjectTransformerSpi {
...
    /**
     * Transforms the data.
     *
     * @param original Original data.
     * @return Transformed data.
     * @throws IgniteCheckedException when transformation is not possible/suitable.
     */
    protected abstract{
	protected ByteBuffer transform(ByteBuffer original) throws IgniteCheckedException; {         

	/**
     * Restores the data.
     *
		int overhead = 5; // *Transformed @paramflag transformed+ Transformed datalength.
     * @return Restored data.
     */
   int protectedorigSize abstract ByteBuffer restore(ByteBuffer transformed= original.remaining(); 
}

Compression example

Code Block

language	java
title	CompressionSpi

class CompressionTransformerSpi extends CacheObjectTransformerSpiAdapter {
	protected ByteBuffer transform(ByteBuffer original) throws IgniteCheckedException {
        int lim = original.remaining()origSize - CacheObjectTransformerSpi.OVERHEADoverhead;

               

		if (lim <= 0)
                 
        throw new IgniteCheckedException("	return null; // Compression is not profitable.");

        ByteBuffer compressed = byteBuffer(lim);(overhead + (int)Zstd.compressBound(origSize));    

		int origSize = original.remaining(compressed.put(TRANSFORMED);

		compressed.positionputInt(locOverheadorigSize);    

		int size = Zstd.compress(compressed, original, 1);

 		if (size >= lim)
         compressed.flip(); 

		compressed.putInt(origSize);	return null; // Compression is not profitable.          

		compressed.rewindflip();          

        return compressed;
    }

    protected ByteBuffer restore(ByteBuffer transformed) {
        ByteBuffer restored = byteBuffer(transformed.getInt());

        Zstd.decompress(restored, transformed);

        restored.flip();
              
        return restored;
    }
}

...

Code Block

language	java
title	EncryptionSpiEncryption

class EncryptionTransformerSpiEncryptionTransformer extends CacheObjectTransformerSpiAdapterCacheObjectTransformerAdapter {
    private static final int SHIFT = 42; // Secret!

    protected ByteBuffer transform(ByteBuffer original) throws IgniteCheckedException {
        ByteBuffer transformed = byteBuffer(original.remaining() + 1); // Same capacity is required.

		transformed.put(TRANSFORMED);

        while (original.hasRemaining())
            transformed.put((byte)(original.get() + SHIFT));

        transformed.flip();

        return transformed;
    }

    protected ByteBuffer restore(ByteBuffer transformed, int length) {
        ByteBuffer restored = byteBuffer(transformed.remaining()); // Same size.
		
		while (transformed.hasRemaining())
            restored.put((byte)(transformed.get() - SHIFT));

        restored.flip();

        return restored;
    }
}

...

Page tree

Versions Compared

Old Version 60

New Version Current

Key

Transformer

API

Simplified API

Examples

Compression example

Compression example