Motivation

Customers may want to 

• minimize (compress)
• protect (encrypt)

user's data at the network and memory layer.

Ignite supports Disk Compression and Transparent Data Encryption, but they are able to transform the data at the persistent layer only.

Description

To cover both layers (network and memory) and make the feature compatible with the existing data, it is proposed to transform/restore CacheObject's bytes on the fly.

A possible solution is to transform the byte arrays they provided during the marshaling/unmarshalling phase. This will cover both layers, messaging (network) and storage (in-memory + persist).

Transformation

All we need is to cover all CacheObjects.

CacheObjects

Most of them has the following structure:

 protected Object val; // Unmarshalled value.
 protected byte[] valBytes; // Marshalled value bytes.

and all we need - is to add transformation during the marshaling/unmarshalling phase:

protected byte[] valueBytesFromValue(CacheObjectValueContext ctx) throws IgniteCheckedException {
    byte[] bytes = ctx.kernalContext().cacheObjects().marshal(ctx, val);

    return CacheObjectsTransformer.transformIfNecessary(bytes, ctx);
}


protected Object valueFromValueBytes(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException {
    byte[] bytes = CacheObjectsTransformer.restoreIfNecessary(valBytes, ctx);

    return ctx.kernalContext().cacheObjects().unmarshal(ctx, bytes, ldr);
}

... 

public void prepareMarshal(CacheObjectValueContext ctx) throws IgniteCheckedException {
	if (valBytes == null)
		valBytes = valueBytesFromValue(ctx);
}

...

public void finishUnmarshal(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException { 
	if (val == null) 
    	val = valueFromValueBytes(ctx, ldr);
}

BinaryObjects

BinaryObject(Impl)s have the different structure:

private Object obj; // Deserialized value.
private byte[] arr; // Serialized bytes.

(De)serialization is a simmilar to (un)marshalling, it's a process to gain java class instance from bytes and or vice versa, but it happen at different time and code layer.

(Un)marshalling happens on putting/getting object to/from cache, but (de)serialization happens on building/deserializing of a binary object detached from any cache.

A lucky circumstance, BinaryObjectImpl require no marshalling, serialization already generates byte which can be used as marshalled bytes.

But, if we're going to transform the data during the marshaling/unmarshalling phase we need to add additional data layer to the BinaryObjectImpl:

private Object obj; // Deserialized value.
private byte[] arr; // Serialized bytes.
private byte[] valBytes; // Marshalled value bytes.

Where valBytes == arr when transformation is disabled.

It's not possible to just replace arr with valBytes because, unlike, for example, from CacheObjectImpl arr is not just a mashalled bytes, it's a object's value requred, for example, to provide hashCode/schemaId/typeId/objectField.

So, BinaryObjectImpl requres valBytes to arr conversion:

private byte[] arrayFromValueBytes(CacheObjectValueContext ctx) {
    return CacheObjectsTransformer.restoreIfNecessary(valBytes, ctx);
}

private byte[] valueBytesFromArray(CacheObjectValueContext ctx) {
    return CacheObjectsTransformer.transformIfNecessary(arr, start, arr.length, ctx);
}

...

public void finishUnmarshal(CacheObjectValueContext ctx, ClassLoader ldr) throws IgniteCheckedException {
	if (arr == null)
	arr = arrayFromValueBytes(ctx);
}

...

public void prepareMarshal(CacheObjectValueContext ctx) {
	if (valBytes == null)
		valBytes = valueBytesFromArray(ctx);
}

Risks and Assumptions

Transformation requires additional memory allocation and subsequent GC work.

Transformation requires additional CPU utilization.

Discussion Links

// Links to discussions on the devlist, if applicable.

Reference Links

// Links to various reference documents, if applicable.

Tickets