Page History

...

ID

IEP-97

Author

Sponsor

Created

22 Nov 2022

Status


colour	Green
title	active

...

No-Op transformer will produce [-3, 0, 3, 42, 0, 0, 0] or [-3, 0, 9, 11, 0, 0, 0, 84, 101, 115, 116, 32, 115, 116, 114, 105, 110, 103], where 0 is a protocol version.
Pseudo-Crypto transformer, which adds 1 to every original byte, will produce [-3, 0, 4, 43, 1, 1, 1] or [-3, 0, 10, 12, 1, 1, 1, 85, 102, 116, 117, 33, 116, 117, 115, 106, 111, 104]
Magic-Compressor will produce [-3, 0, 7] or [-3, 0, 17], where 7 and 17 are the result of a magic compression.

...

Code Block

language	java
title	CacheObjectAdapter transformation

protected byte[] valueBytesFromValue(CacheObjectValueContext ctx) throws IgniteCheckedException {
    byte[] bytes = ctx.kernalContext().cacheObjects().marshal(ctx, val);

    return CacheObjectTransformerCacheObjectTransformerUtils.transformIfNecessary(bytes, ctx);
}


protected Object valueFromValueBytes(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException {
    byte[] bytes = CacheObjectTransformerCacheObjectTransformerUtils.restoreIfNecessary(valBytes, ctx);

    return ctx.kernalContext().cacheObjects().unmarshal(ctx, bytes, ldr);
}


public void prepareMarshal(CacheObjectValueContext ctx) throws IgniteCheckedException {
	if (valBytes == null)
		valBytes = valueBytesFromValue(ctx);
}


public void finishUnmarshal(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException { 
	if (val == null) 
    	val = valueFromValueBytes(ctx, ldr);
}

...

It's not possible to just replace arr with valBytes because, unlike , for example, from CacheObjectImpl arr is not just a mashalled bytes, it's an object's value that is required, for example, to provide hashCode/schemaId/typeId/objectField, and we must keep it as is.

...

Code Block

language	java
title	BinaryObjectImpl (un)marshalling

private byte[] arrayFromValueBytes(CacheObjectValueContext ctx) {
    return CacheObjectTransformerCacheObjectTransformerUtils.restoreIfNecessary(valBytes, ctx);
}

private byte[] valueBytesFromArray(CacheObjectValueContext ctx) {
    return CacheObjectTransformerCacheObjectTransformerUtils.transformIfNecessary(arr, start, arr.length, ctx);
}


public void finishUnmarshal(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException {
	if (arr == null)
		arr = arrayFromValueBytes(ctx);
}


public void prepareMarshal(CacheObjectValueContext ctx) {
	if (valBytes == null)
		valBytes = valueBytesFromArray(ctx);
}

...

Transformer

Some customers may want to encrypt the data, some to compress it, while some just keep it as is.

So, we must provide a simple way to append any transformation.

API

The simplest way is to use Service Provider Interface (IgniteSpi):

Code Block

language	java
title	SPIInterface

public interface CacheObjectTransformerSpiCacheObjectTransformerManager extends IgniteSpiGridCacheSharedManager {
    /** Additional space required to store the transformed data. */
    public int OVERHEAD = 2;

    /**
     * Transforms the data.
     *
     * @param bytes  Byte array contains the data.
     * @param offset Data offset.
     * @param length Data lengthoriginal Original data.
     * @return Byte array contains the transformed Transformed data (started with non-filled area with {@link #OVERHEADGridBinaryMarshaller#TRANSFORMED} when size.restorable)
     * or @throws{@code IgniteCheckedExceptionnull} when transformation is not possible/suitable.
     */
    public @Nullable byte[]ByteBuffer transform(byte[] bytes, int offset, int length) throws IgniteCheckedException;      ByteBuffer original);

    /**
     * Restores the data.
     *
     * @param bytes  Byte array contains the transformed data.
	 * @param offset Data offsettransformed Transformed data.
     * @param length Data length.
     * @return Byte array contains the restored @return Restored data.
     */
    public byte[]ByteBuffer restore(byte[] bytes, int offset, int lengthByteBuffer transformed);
}

...

Every customer may implement this interface in a proper way if necessary and specify it in the via plugin configuration:

Code Block

language	java
title	Custom SPItransformer

IgniteConfiguration getConfiguration() {
	IgniteConfiguration cfg = ...

	cfg.setCacheObjectTransformerSpisetPluginProviders(new XXXTransformerSpiXXXPluginProvider());

	return cfg;
}

Simplified API

But, most customers just want to transform the data, so, they may extend the adapter with the simplified API:

Code Block

language	java
title	CacheObjectTransformerSpiAdapter

public abstract class CacheObjectTransformerSpiAdapter extends IgniteSpiAdapter implements CacheObjectTransformerSpi {
...
    /**
     * Transforms the data.
     *
     * @param original Original data.
     * @return Transformed data.
     * @throws IgniteCheckedException when transformation is not possible/suitable.
 Which provides   */
    protected abstract ByteBuffer transform(ByteBuffer original) throws IgniteCheckedException;      

	/**
     * Restores the data.
     *
     * @param transformed Transformed data.
     * @return Restored data.
     */
    protected abstract ByteBuffer restore(ByteBuffer transformed); 
}some XXXCacheObjectTransformerManager()

    return cfg;
}

Examples

Compression example

Code Block

language	java
title	CompressionSpiCompression

class CompressionTransformerSpiCompressionTransformer extends CacheObjectTransformerSpiAdapterCacheObjectTransformerAdapter {
	protected ByteBuffer transform(ByteBuffer original) throws IgniteCheckedException {                     
		int locOverheadoverhead = 45; // OriginalTransformed length.
flag        int totalOverhead = CacheObjectTransformerSpi.OVERHEAD + locOverhead;+ length.

        int origSize = original.remaining();
        int lim = origSize - totalOverheadoverhead;              

		if (lim <= 0)
             
       throw new IgniteCheckedException(" 	return null; // Compression is not profitable.");

        ByteBuffer compressed = byteBuffer(lim);
(overhead + (int)Zstd.compressBound(origSize));    

		compressed.put(TRANSFORMED);
		compressed.positionputInt(locOverheadorigSize);    

		int size = Zstd.compress(compressed, original, 1);

 		if (size >= lim)
         compressed.flip(); 

		compressed.putInt(origSize);	return null; // Compression is not profitable.          

		compressed.rewindflip();          

        return compressed;
    }

    protected ByteBuffer restore(ByteBuffer transformed) {
        ByteBuffer restored = byteBuffer(transformed.getInt());

        Zstd.decompress(restored, transformed);

        restored.flip();
              
        return restored;
    }
}

...

Code Block

language	java
title	EncryptionSpiEncryption

class EncryptionTransformerSpiEncryptionTransformer extends CacheObjectTransformerSpiAdapterCacheObjectTransformerAdapter {
    private static final int SHIFT = 42; // Secret!

    protected ByteBuffer transform(ByteBuffer original) throws IgniteCheckedException {
        ByteBuffer transformed = byteBuffer(original.remaining() + 1); // Same capacity is required.

		transformed.put(TRANSFORMED);

        while (original.hasRemaining())
            transformed.put((byte)(original.get() + SHIFT));

        transformed.flip();

        return transformed;
    }

    protected ByteBuffer restore(ByteBuffer transformed, int length) {
        ByteBuffer restored = byteBuffer(transformed.remaining()); // Same size.
		
		while (transformed.hasRemaining())
            restored.put((byte)(transformed.get() - SHIFT));

        restored.flip();

        return restored;
    }
}

...

Page tree

Versions Compared

Old Version 62

New Version Current

Key

Transformer

API

Simplified API

Examples

Compression example