Page History

...

• • It requires a hit-and-trial approach to set the threadpool size appropriately so as not to exceed a certain read rate from the remote storage.
  • The setting is dependent on the broker hardware and needs to be tuned accordingly.
• We could use the QuotaManager differently. It will be used by the worker threads in the reader threadpool to check if the read quota has been exceeded. If it isn’t exceeded, the read task is processed. Otherwise, the read task must wait. This can be implemented in two ways:
  • Read Task computes the throttle time using the quota framework and sleeps for that amount of time before executing the request. When the task wakes up, the read rate would have fallen within the specified quota. 
  • To avoid the above problem, Another approach is that instead of waiting, the RLMTask execution can be deferred and can be scheduled to run with some delay, i.e. throttle time. 

The drawback of this approach the above two approaches is that if the remote read quota has been exhausted, the RLM will keep accepting more read tasks to be executed later. The fetch request corresponding to the read task may have already timed out by the time the read task gets executed. This will lead to a waste of resources for the broker. 

...