Page History

...

The QuotaManager utilizes a rolling window to track the bytes uploaded every second. There is a time bucket for each sample and it stores the total bytes uploaded in the time bucket.

61 buckets of size 1 second (default)

...

1. • This approach relies on reducing concurrency to achieve lower upload speed. In practice, we would know what the maximum upload rate our remote storage can support. It is not straightforward to translate this to the concurrency of upload threads and requires hit-and-trial approach.
   • Reducing concurrency can also introduce unfairness while uploading segments. When an RLMTask runs for a topic partition, it uploads all the eligible log segments in the same run preventing uploads for other topic partitions This can cause delays and lag buildup for other topic partitions, particularly when the thread pool size is small. If some topics have a high message-in rate, the corresponding RLMTasks would end up using all threads in the threadpool preventing uploads for smaller topics.
2. We could use the QuotaManager differently. Instead of tracking the global upload rate, we could track the upload rate per thread in the writer threadpool. Each thread records its upload speed with the quota manager and checks for quota violations before uploading the next log segment.

Cons:

1. • Each thread only gets a fraction of the global quota. This will lead to under-utilization of the resources.  For eg., if the global quota is 100 MBps and the threadpool size is 10, each thread can never upload at more than 10 MBps. But if some topic partitions have message-in rate higher than 10 MBps, the upload rate can never catch up with the message-in rate even though the overall upload rate is below 100 MBps.

The alternate architectures are rejected for the disadvantages discussed above.

...