...
Wraps several ByteBuffers, extends ByteBuffer. It is created in ByteBufferPool#acquire and destroyed in ByteBufferPool#release when all internal buffers returned to the pool.
We can often see that disk IO operation is the bottleneck in Ignite. So we should try to make writing to disk efficient.
...
The thread will periodically check disk writer queue size making sure that disk writer always has enough data to process. So, minimum number of elements will be 2 for single disk writer. Once the size goes down, the thread will scan sizes of partition queues and choose the winner with the maximum number of bytes. After that it will take all buffers from the queue and will put them together with file info (channel) to the disk writer queue.
In case of multiple disk writers additionally a flag should be passed, which will be considered when choosing largest partition queue and will be reset on buffer release.
They could be simple ArrayBlockingQueues.
Open question: Does it make sense to use non-blocking queue here for faster put/take queue operations?
Does 3 operations
Takes buffers from queue and returns to the pool. This operation will probably be fast enough so that extra thread could be redundant.
The solution assumes that all partition dump files are open simultaneously. This could break when number of partitions (default 1024) is bigger than OS limit for open files (ulimit -n).
...
In case of encryption 2 buffers will be required. First one as usual will be used for keeping serialized entry, and the second one will be for storing encrypted data from the first buffer. After encryption the first buffer will be returned back to the pool. The second buffer will go to the partition queue.
Note: Avoid deadlock when processing large objects not fitting into pool size.
Compression is done after encryption (if enabled).
...
The size of required buffer isn't known while requesting buffer from pool. It is preferable to use medium size buffers. And it is ok to get a bit smaller buffer for this, fill it, send it to queue and request another medium size buffer from pool.
Once dump creating is over, all resources will be cleaned.
In case of error the whole execution will stop. Resuming creating dump isn't considered, the only option is to rerun creating dump.
...