Page History

...

Takes buffers from queue and returns to the pool. This operation will probably be fast enough so that extra thread will be redundant.

Open file limit

The solution assumes that all partition dump files are open simultaneously. This could break when number of partitions (default 1024) is bigger than OS limit for open files (ulimit -n).

We can address this issue like this

• keep track of number of open files and last write time for each partition
• identify number of allowed open files for creating dump, for example, like half of ulimit -n
• when number of files reaches threshold, close the least recently written partition file without draining the partition queue
• reopen closed file when written
• log warning message about non-optimal configuration of partitions count and limit on open files

Compression

Encryption

In case of encryption 2 buffers will be required. First one as usual will be used for keeping serialized entry, and the second one will be for storing encrypted data from the first buffer. After encryption the first buffer will be returned back to the pool. The second buffer will go to the partition queue.

Note: Avoid deadlock when processing large objects not fitting into pool size. 

Shutdown / Execution Completion / Error handling

...