Page History

...

• ByteBufferPool(size) - constructor with maximum number of bytes to allocate by all buffers together.   

• ByteBuffer acquire(minSize) - returns ByteBuffer that has size = power of 2 and not smaller than minSize. If there is no buffer in pool, then a new buffer is allocated. If overall size occupied by all buffers has reached the limit, then this method will block until required buffer(s) returned to the pool.
• release(ByteBuffer) - returns buffer to the pool and signals waiter on acquire
• ByteBuffer acquireAny(preferredSize) - method for compression which don't require exact size buffer.

Using buffers only of size of power of 2 simplifies maintenance of the buffers and search. Internal structure to use:

...

• keep track of number of open files and last write time for each partition
• identify number of allowed open files for creating dump, for example, like half of ulimit -n
• when number of files reaches threshold, close the least recently written partition file without draining the partition queue
• reopen closed file when written
• log warning message about non-optimal configuration of partitions count and limit on open files

...

Encryption

In case of encryption 2 buffers will be required. First one as usual will be used for keeping serialized entry, and the second one will be for storing encrypted data from the first buffer. After encryption the first buffer will be returned back to the pool. The second buffer will go to the partition queue.

Note: Avoid deadlock when processing large objects not fitting into pool size. 

Compression

Compression is done after encryption (if enabled). This can remain in the same thread (part/trans thread).

Compression output is a sequence of buffers which can't be reordered. Compression per partition can't be done in 2 threads simultaneously.

The size of required buffer isn't known while requesting buffer from pool. It is preferable to use medium size buffers. And it is ok to get a bit smaller buffer for this, fill it, send it to queue and request another medium size buffer from pool.

Shutdown / Execution Completion

...

Once dump creating is over, all resources will be cleaned.

• Newly created threads finish.
• PolledByteBuffer will clean returning buffers to heap (HeapBB) or os (DirectBB).

Error handling

In case of error the whole execution will stop. Resuming creating dump isn't considered, the only option is to rerun creating dump.  

Questions

What happens if a dump entry doesn't fit into pool size? 

Is the order of entries in dump important? Is it acceptable to write ver 3 (from trans thread) before ver 2 (by par thread) ?

HeapByteBuffer or DirectByteBuffer ?

Is there a protection against simultaneous dump creation ? Do we need one?