Page History

General Information

The Qpid Java Broker 0.6 release introduces a simplistic producer-side flow control mechanism into the Java Messaging Broker, causing producers to be flow-controlled when they attempt to send messages to an overfull queue or overfull message store on a virtual host.

Server configuration

Configuring a Queue to use flow control

Flow control is enabled on a producer when it sends a message to a Queue which is "overfull". The producer flow control will be rescinded when all Queues on which a producer is blocking become "underfull". A Queue is defined as overfull when the size (in bytes) of the messages on the queue exceeds the "capacity" of the Queue. A Queue becomes "underfull" when its size becomes less than the "flowResumeCapacity".

...

Where no flowResumeCapacity is set, the flowResumeCapacity is set to be equal to the capacity. Where no capacity is set, capacity is defaulted to 0 meaning there is no capacity limit.

Broker Log Messages

There are four new Broker log messages that may occur if flow control through queue capacity limits is enabled.

Firstly, when a capacity limited queue becomes overfull, a log message similar to the following is produced

MESSAGE [vh(/test)/qu(MyQueue)] [vh(/test)/qu(MyQueue)] QUE-1003 : Overfull : Size : 1,200 bytes, Capacity : 1,000

Then for each channel which becomes blocked upon the overful queue a log message similar to the following is produced:

MESSAGE [con:2(guest@anonymous(713889609)/test)/ch:1] [con:2(guest@anonymous(713889609)/test)/ch:1] CHN-1005 : Flow Control Enforced (Queue MyQueue)

When enough messages have been consumed from the queue that it becomes underfull, then the following log is generated:

MESSAGE [vh(/test)/qu(MyQueue)] [vh(/test)/qu(MyQueue)] QUE-1004 : Underfull : Size : 600 bytes, Resume Capacity : 800

And for every channel which becomes unblocked you will see a message similar to:

MESSAGE [con:2(guest@anonymous(713889609)/test)/ch:1] [con:2(guest@anonymous(713889609)/test)/ch:1] CHN-1006 : Flow Control Removed

Obviously the details of connection, virtual host, queue, size, capacity, etc would depend on the configuration in use.

Disk quota-based flow control

Since version 0.18 of Qpid Broker, flow control can be triggered when a configured disk quota is exceeded. This is supported by the BDB and Derby message stores.

This functionality blocks all producers on reaching the disk overflow limit. When consumers consume the messages, causing disk space usage to falls below the underflow limit, the producers are unblocked and continue working as normal.

Two limits can be configured:

1. overflow limit - the maximum space on disk (in bytes) which can be used by store.
2. underflow limit - when the space on disk drops below this limit, producers are allowed to resume publishing.

An example of quota configuration for the BDB message store is provided below.

   <store>
       <class>org.apache.qpid.server.store.berkeleydb.BDBMessageStore</class>
       <environment-path>${work}/bdbstore/test</environment-path>
       <overfull-size>50000000</overfull-size>
       <underfull-size>45000000</underfull-size>
    </store>

The disk quota functionality is based on "best effort" principle. It means that broker cannot guarantee that disk space limit will not be exceeded. If several concurrent transactions are started before the limit is reached then all of them will be committed into disk even when disk quota is exceeded.

Broker Log Messages for quota flow control

There are 2 new Broker log messages that may occur if flow control through disk quota limits is enabled.
When virtual host is blocked due to exceeding of the disk quota limit the following message appears in the broker log

[vh(/test)/ms(BDBMessageStore)] MST-1008 : Store overfull, flow control will be enforced

When virtual host is unblocked after cleaning the disk space the following message appears in the broker log

[vh(/test)/ms(BDBMessageStore)] MST-1009 : Store overfull condition cleared

Client impact and configuration

...

-Dqpid.flow_control_wait_failure=60000
-Dqpid.flow_control_wait_notify_period=10000

Using failover with flow control

If flow control feature is used with a failover functionality and connection is lost when producer is blocked by the flow control, the failover is not started immediately due to implementation limitation. But it is resumed after the flow control wait interval is expired. However, failover functionality has 1 minute timeout interval to try to re-establish the connection and counting of this interval starts after losing of connection. If this timeout is exceeded the failover will be interrupted and will not be resumed. On other hand, the default flow control wait interval is set to 2 minutes and as result, the failover can timeout whilst trying to re-establish the connection for blocked producer(s).

In order to avoid this from happening the failover timeout can be set to the value bigger than flow control wait interval. For setting of failover timeout the JVM setting "qpid.failover_method_timeout" can be used as follows

-Dqpid.failover_method_timeout=180000

Setting of flow wait interval to the value lower than a failover timeout would allow failover to restore connectivity.

Older Clients

This feature was added for the 0.6 releaase of the Java Broker. If an older client connects to the broker then the flow control commands will be ignored and they will not be blocked. So to fully benefit from this new feature both Client and Broker need to be at least version 0.6.

Broker Log Messages

There are four new Broker log messages that may occur if flow control through queue capacity limits is enabled.

Firstly, when a capacity limited queue becomes overfull, a log message similar to the following is produced

MESSAGE [vh(/test)/qu(MyQueue)] [vh(/test)/qu(MyQueue)] QUE-1003 : Overfull : Size : 1,200 bytes, Capacity : 1,000

Then for each channel which becomes blocked upon the overful queue a log message similar to the following is produced:

MESSAGE [con:2(guest@anonymous(713889609)/test)/ch:1] [con:2(guest@anonymous(713889609)/test)/ch:1] CHN-1005 : Flow Control Enforced (Queue MyQueue)

When enough messages have been consumed from the queue that it becomes underfull, then the following log is generated:

MESSAGE [vh(/test)/qu(MyQueue)] [vh(/test)/qu(MyQueue)] QUE-1004 : Underfull : Size : 600 bytes, Resume Capacity : 800

And for every channel which becomes unblocked you will see a message similar to:

MESSAGE [con:2(guest@anonymous(713889609)/test)/ch:1] [con:2(guest@anonymous(713889609)/test)/ch:1] CHN-1006 : Flow Control Removed

Obviously the details of connection, virtual host, queue, size, capacity, etc would depend on the configuration in use.