Wiki Markup |
---|
h3. Aggregator
*This applies for Camel version |
Aggregator
...
2.3 or newer. If you use an older version then use this [Aggregator] link instead. |
...
The Aggregator from the EIP patterns allows you to combine a number of messages together into a single message.
A correlation Expression is used to determine the messages which should be aggregated together. If you want to aggregate all messages into a single message, just use a constant expression. An AggregationStrategy is used to combine all the message exchanges for a single correlation key into a single message exchange.
Aggregator options
The aggregator supports the following options:
...
Option
...
Default
...
Description
...
correlationExpression
...
...
Mandatory Expression which evaluates the correlation key to use for aggregation. The Exchange which has the same correlation key is aggregated together. If the correlation key could not be evaluated an Exception is thrown. You can disable this by using the ignoreBadCorrelationKeys
option.
...
aggregationStrategy
...
...
Mandatory AggregationStrategy
which is used to merge the incoming Exchange with the existing already merged exchanges. At first call the oldExchang
parameter is null
. On subsequent invocations the oldExchnage
contains the merged exchanges and newExchange
is of course the new incoming Exchange.
...
strategyRef
...
...
A reference to lookup the AggregationStrategy
in the Registry.
...
completionSize
...
...
Number of messages aggregated before the aggregation is complete. This option can be set as either a fixed value or using an Expression which allows you to evaluate a size dynamically - will use Integer
as result. If both are set Camel will fallback to use the fixed value if the Expression result was null
or 0
.
...
completionTimeout
...
...
Time in millis that an aggregated exchange should be inactive before its complete. Camel has a background task that runs once a minute to check for inactive aggregated exchanges. This option can be set as either a fixed value or using an Expression which allows you to evaluate a timeout dynamically - will use Long
as result. If both are set Camel will fallback to use the fixed value if the Expression result was null
or 0
. You cannot use this option together with completionInterval, only one of the can be used.
...
completionInterval
...
...
A repeating period in millis by which the aggregator will complete all current aggregated exchanges. Camel has a background tasks which is trigger every period. You cannot use this option together with completionTimeout, only one of the can be used.
...
completionPredicate
...
...
A Predicate to indicate when an aggregated exchange is complete.
...
completionFromBatchConsumer
...
false
...
This option is if the exchanges is coming from a Batch Consumer. Then when enabled the Aggregator2 will use the batch size determined by the Batch Consumer in the message header CamelBatchSize
. See more details at Batch Consumer. This can be used to aggregate all files consumed from a File endpoint in that given poll.
...
eagerCheckCompletion
...
false
...
Whether or not to eager check for completion when a new incoming Exchange has been received. This option influences the behavior of the completionPredicate
option as the Exchange being passed in changes accordingly. When false
the Exchange passed in the Predicate is the aggregated Exchange which means any information you may store on the aggregated Exchange from the AggregationStrategy
is avail for the Predicate. When true
the Exchange passed in the Predicate is the incoming Exchange, which means you can access data from the incoming Exchange.
...
groupExchanges
...
false
...
* The [Aggregator|http://www.enterpriseintegrationpatterns.com/Aggregator.html] from the [EIP patterns|Enterprise Integration Patterns] allows you to combine a number of messages together into a single message. !http://www.enterpriseintegrationpatterns.com/img/Aggregator.gif! A correlation [Expression] is used to determine the messages which should be aggregated together. If you want to aggregate all messages into a single message, just use a constant expression. An AggregationStrategy is used to combine all the message exchanges for a single correlation key into a single message exchange. h3. Aggregator options The aggregator supports the following options: {div:class=confluenceTableSmall} || Option || Default || Description || | correlationExpression | | Mandatory [Expression] which evaluates the correlation key to use for aggregation. The [Exchange] which has the same correlation key is aggregated together. If the correlation key could not be evaluated an Exception is thrown. You can disable this by using the {{ignoreBadCorrelationKeys}} option. | | aggregationStrategy | | Mandatory {{AggregationStrategy}} which is used to _merge_ the incoming [Exchange] with the existing already merged exchanges. At first call the {{oldExchang}} parameter is {{null}}. On subsequent invocations the {{oldExchnage}} contains the merged exchanges and {{newExchange}} is of course the new incoming Exchange. | | strategyRef | | A reference to lookup the {{AggregationStrategy}} in the [Registry]. | | completionSize | | Number of messages aggregated before the aggregation is complete. This option can be set as either a fixed value or using an [Expression] which allows you to evaluate a size dynamically - will use {{Integer}} as result. If both are set Camel will fallback to use the fixed value if the [Expression] result was {{null}} or {{0}}. | | completionTimeout | | Time in millis that an aggregated exchange should be inactive before its complete. Camel has a background task that runs once a minute to check for inactive aggregated exchanges. This option can be set as either a fixed value or using an [Expression] which allows you to evaluate a timeout dynamically - will use {{Long}} as result. If both are set Camel will fallback to use the fixed value if the [Expression] result was {{null}} or {{0}}. You cannot use this option together with completionInterval, only one of the can be used. | | completionInterval | | A repeating period in millis by which the aggregator will complete all current aggregated exchanges. Camel has a background tasks which is trigger every period. You cannot use this option together with completionTimeout, only one of the can be used. | | completionPredicate | | A [Predicate] to indicate when an aggregated exchange is complete. | | completionFromBatchConsumer | {{false}} | This option is if the exchanges is coming from a [Batch Consumer]. Then when enabled the [Aggregator2] will use the batch size determined by the [Batch Consumer] in the message header {{CamelBatchSize}}. See more details at [Batch Consumer]. This can be used to aggregate all files consumed from a [File|File2] endpoint in that given poll. | | eagerCheckCompletion | {{false}} | Whether or not to eager check for completion when a new incoming [Exchange] has been received. This option influences the behavior of the {{completionPredicate}} option as the [Exchange] being passed in changes accordingly. When {{false}} the [Exchange] passed in the [Predicate] is the _aggregated_ Exchange which means any information you may store on the aggregated Exchange from the {{AggregationStrategy}} is avail for the [Predicate]. When {{true}} the [Exchange] passed in the [Predicate] is the _incoming_ [Exchange], which means you can access data from the incoming Exchange. | | groupExchanges | {{false}} | If enabled then Camel will group all aggregated Exchanges into a single combined {{org.apache.camel.impl.GroupedExchange}} holder class that holds all the aggregated Exchanges. And as a result only one Exchange is being sent out from the aggregator. Can be used to combine many incomming Exchanges into a single output Exchange without coding a custom {{AggregationStrategy}} yourself. |
...
ignoreInvalidCorrelationKeys
...
false
...
Whether or not to ignore correlation keys which could not be evaluated to a value. By default Camel will thrown an Exception, but you can enable this option and ignore the situation instead.
...
closeCorrelationKeyOnCompletion
...
...
Whether or not too late Exchange should be accepted or not. You can enable this to indicate that if a correlation key has already been completed, then any new exchanges with the same correlation key be denied. Camel will then throw a closedCorrelationKeyException
exception. When using this option you pass in a integer
which is a number for a LRUCache which keeps that last X number of closed correlation keys. You can pass in 0 or a negative value to indicate a unbounded cache. By passing in a number you are ensured that cache wont grown too big if you use a log of different correlation keys.
...
aggregationRepository
...
...
| | ignoreInvalidCorrelationKeys | {{false}} | Whether or not to ignore correlation keys which could not be evaluated to a value. By default Camel will thrown an Exception, but you can enable this option and ignore the situation instead. | | closeCorrelationKeyOnCompletion | | Whether or not too _late_ Exchange should be accepted or not. You can enable this to indicate that if a correlation key has already been completed, then any new exchanges with the same correlation key be denied. Camel will then throw a {{closedCorrelationKeyException}} exception. When using this option you pass in a {{integer}} which is a number for a LRUCache which keeps that last X number of closed correlation keys. You can pass in 0 or a negative value to indicate a unbounded cache. By passing in a number you are ensured that cache wont grown too big if you use a log of different correlation keys. | | aggregationRepository | | Allows you to plugin you own implementation of {{org.apache.camel.spi.AggregationRepository}} which keeps track of the current inflight aggregated exchanges. Camel uses by default a memory based implementation. |
...
aggregationRepositoryRef
...
...
Reference to lookup a aggregationRepository
in the Registry.
...
parallelProcessing
...
false
...
When aggregated are completed they are being send out of the aggregator. This option indicates whether or not Camel should use a thread pool with multiple threads for concurrency. If not custom thread pool has been specified then Camel creates a default pool with 10 concurrent threads.
...
executorService
...
...
If using parallelProcessing
you can specify a custom thread pool to be used. In fact also if you are not using parallelProcessing
this custom thread pool is used to send out aggregated exchanges as well.
...
executorServiceRef
...
...
Reference to lookup a executorService
in the Registry
Exchange Properties
The following properties is set on each Exchange that are aggregated:
header | type | description |
---|---|---|
| int | The total number of Exchanges aggregated into this combined Exchange. |
| String | Indicator how the aggregation was completed as a value of either: |
About AggregationStrategy
The AggregationStrategy
is used for aggregate the old (lookup by its correlation id) and the new exchanges together into a single exchange. Possible implementations include performing some kind of combining or delta processing, such as adding line items together into an invoice or just using the newest exchange and removing old exchanges such as for state tracking or market data prices; where old values are of little use.
Notice the aggregation strategy is a mandatory option and must be provided to the aggregator.
About completion
When aggregation Exchanges at some point you need to indicate that the aggregated exchanges is complete, so they can be send out of the aggregator. Camel allows you to indicate completion in various ways as follows:
- completionTimeout - Is an inactivity timeout in which is triggered if no new exchanges has been aggregated for that particular correlation key within the period.
- completionInterval - Once every X period all the current aggregated exchanges is completed.
- completionSize - Is a number indicating that after X aggregated exchanges its complete.
- completionPredicate - Runs a Predicate when a new exchange is aggregated to determine if we are complete or not
- completionFromBatchConsumer - Special option for Batch Consumer which allows you to complete when all the messages from the batch has been aggregated. |
Notice that all the completion ways are per correlation key. And you can combine them in any way your like. Its basically the first which triggers that wins. So you can use a completion size together with a completion timeout. Only completionTimeout and completionInterval cannot be used at the same time.
Notice the completion is a mandatory option and must be provided to the aggregator. If not provided Camel will thrown an Exception on startup.
Persistent AggregationRepository
...
| | aggregationRepositoryRef | | Reference to lookup a {{aggregationRepository}} in the [Registry]. | | parallelProcessing | {{false}} | When aggregated are completed they are being send out of the aggregator. This option indicates whether or not Camel should use a thread pool with multiple threads for concurrency. If not custom thread pool has been specified then Camel creates a default pool with 10 concurrent threads. | | executorService | | If using {{parallelProcessing}} you can specify a custom thread pool to be used. In fact also if you are not using {{parallelProcessing}} this custom thread pool is used to send out aggregated exchanges as well. | | executorServiceRef | | Reference to lookup a {{executorService}} in the [Registry] | {div} h3. Exchange Properties The following properties is set on each Exchange that are aggregated: {div:class=confluenceTableSmall} || header || type || description || | {{CamelAggregatedSize}} | int | The total number of Exchanges aggregated into this combined Exchange. | | {{CamelAggregatedCompletedBy}} | String | Indicator how the aggregation was completed as a value of either: {{predicate}}, {{size}}, {{consumer}}, {{timeout}} or {{interval}}. | {div} h3. About AggregationStrategy The {{AggregationStrategy}} is used for aggregate the old (lookup by its correlation id) and the new exchanges together into a single exchange. Possible implementations include performing some kind of combining or delta processing, such as adding line items together into an invoice or just using the newest exchange and removing old exchanges such as for state tracking or market data prices; where old values are of little use. Notice the aggregation strategy is a mandatory option and must be provided to the aggregator. h3. About completion When aggregation [Exchange]s at some point you need to indicate that the aggregated exchanges is complete, so they can be send out of the aggregator. Camel allows you to indicate completion in various ways as follows: - completionTimeout - Is an inactivity timeout in which is triggered if no new exchanges has been aggregated for that particular correlation key within the period. - completionInterval - Once every X period all the current aggregated exchanges is completed. - completionSize - Is a number indicating that after X aggregated exchanges its complete. - completionPredicate - Runs a [Predicate] when a new exchange is aggregated to determine if we are complete or not - completionFromBatchConsumer - Special option for [Batch Consumer] which allows you to complete when all the messages from the batch has been aggregated. | Notice that all the completion ways are per correlation key. And you can combine them in any way your like. Its basically the first which triggers that wins. So you can use a completion size together with a completion timeout. Only completionTimeout and completionInterval cannot be used at the same time. Notice the completion is a mandatory option and must be provided to the aggregator. If not provided Camel will thrown an Exception on startup. h3. Persistent AggregationRepository The aggregator provides a pluggable repository which you can implement your own {{org.apache.camel.spi.AggregationRepository}}. |
...
If you need persistent repository then you can use the Camel [HawtDB] component. |
...
Examples
See some examples from the old Aggregator which is somewhat similar to this new aggregator.
Tip | ||
---|---|---|
| ||
Many of the options are configurable as attributes on the |
Using completionTimeout
In this example we want to aggregate all incoming messages and after 3 seconds of inactivity we want the aggregation to complete. This is done using the completionTimeout
option as shown:
Wiki Markup |
---|
h3. Examples See some examples from the old [Aggregator] which is somewhat similar to this new aggregator. {tip:title=Setting options in Spring XML} Many of the options are configurable as attributes on the {{<aggregate>}} tag when using Spring XML. {tip} h4. Using completionTimeout In this example we want to aggregate all incoming messages and after 3 seconds of inactivity we want the aggregation to complete. This is done using the {{completionTimeout}} option as shown: {snippet:id=e1|lang=java|url=camel/trunk/camel-core/src/test/java/org/apache/camel/processor/aggregator/AggregateSimpleTimeoutTest.java} And the same example using Spring XML: |
...
Wiki Markup |
---|
{snippet:id=e1|lang=xml|url=camel/trunk/components/camel-spring/src/test/resources/org/apache/camel/spring/processor/aggregator/SpringAggregateSimpleTimeoutTest.xml} h4. Using completionSize |
...
In this example we want to aggregate all incoming messages and when we have 3 messages aggregated (in the same correlation group) we want the aggregation to complete. This is done using the {{completionSize}} option as shown: |
...
Wiki Markup |
---|
{snippet:id=e1|lang=java|url=camel/trunk/camel-core/src/test/java/org/apache/camel/processor/aggregator/AggregateSimpleSizeTest.java} And the same example using Spring XML: |
...
Wiki Markup |
---|
{snippet:id=e1|lang=xml|url=camel/trunk/components/camel-spring/src/test/resources/org/apache/camel/spring/processor/aggregator/SpringAggregateSimpleSizeTest.xml} |
Using completionPredicate
In this example we want to aggregate all incoming messages and use a Predicate to determine when we are complete. The Predicate can be evaluated using either the aggregated exchange (default) or the incoming exchange. We will so both situations as examples. We start with the default situation as shown:
Wiki Markup |
---|
h4. Using completionPredicate In this example we want to aggregate all incoming messages and use a [Predicate] to determine when we are complete. The [Predicate] can be evaluated using either the aggregated exchange (default) or the incoming exchange. We will so both situations as examples. We start with the default situation as shown: {snippet:id=e1|lang=java|url=camel/trunk/camel-core/src/test/java/org/apache/camel/processor/aggregator/AggregateSimplePredicateTest.java} And the same example using Spring XML: |
...
Wiki Markup |
---|
{snippet:id=e1|lang=xml|url=camel/trunk/components/camel-spring/src/test/resources/org/apache/camel/spring/processor/aggregator/SpringAggregateSimplePredicateTest.xml} And the other situation where we use the {{eagerCheckCompletion}} option to tell Camel to use the incoming Exchange. Notice how we can just test in the completion predicate that the incoming message is the _END_ message: |
...
Wiki Markup |
---|
{snippet:id=e1|lang=java|url=camel/trunk/camel-core/src/test/java/org/apache/camel/processor/aggregator/AggregateSimplePredicateEagerTest.java} And the same example using Spring XML: |
...
Wiki Markup |
---|
{snippet:id=e1|lang=xml|url=camel/trunk/components/camel-spring/src/test/resources/org/apache/camel/spring/processor/aggregator/SpringAggregateSimplePredicateEagerTest.xml} h4. Using dynamic completionTimeout |
...
In this example we want to aggregate all incoming messages and after a period of inactivity we want the aggregation to complete. The period should be computed at runtime based on the {{timeout}} header in the incoming messages. This is done using the {{completionTimeout}} option as shown: |
...
Wiki Markup |
---|
{snippet:id=e1|lang=java|url=camel/trunk/camel-core/src/test/java/org/apache/camel/processor/aggregator/AggregateExpressionTimeoutTest.java} And the same example using Spring XML: |
...
Wiki Markup |
---|
{snippet:id=e1|lang=xml|url=camel/trunk/components/camel-spring/src/test/resources/org/apache/camel/spring/processor/aggregator/SpringAggregateExpressionTimeoutTest.xml} *Note:* You can also add a fixed timeout value and Camel will fallback to use this value if the dynamic value was {{null}} or {{0 |
...
Using dynamic completionSize
In this example we want to aggregate all incoming messages based on a dynamic size per correlation key. The size is computed at runtime based on the mySize
header in the incoming messages. This is done using the completionSize
option as shown:
Wiki Markup |
---|
}}. h4. Using dynamic completionSize In this example we want to aggregate all incoming messages based on a dynamic size per correlation key. The size is computed at runtime based on the {{mySize}} header in the incoming messages. This is done using the {{completionSize}} option as shown: {snippet:id=e1|lang=java|url=camel/trunk/camel-core/src/test/java/org/apache/camel/processor/aggregator/AggregateExpressionSizeTest.java} And the same example using Spring XML: |
...
Wiki Markup |
---|
{snippet:id=e1|lang=xml|url=camel/trunk/components/camel-spring/src/test/resources/org/apache/camel/spring/processor/aggregator/SpringAggregateExpressionSizeTest.xml} *Note:* You can also add a fixed size value and Camel will fallback to use this value if the dynamic value was {{null}} or {{0}}. |
...
{include |
...
:Using This |
...
See also
...
Pattern} h3. See also - The [Loan Broker Example] which uses an aggregator - [Blog post by Torsten Mielke|http://tmielke.blogspot.com/2009/01/using-camel-aggregator-correctly.html] about using the aggregator correctly. - The old [Aggregator] - [HawtDB] for persistence support - [Aggregate Example] for an example application |