Page History

...

This indicates an attempt to create an event body that is larger than the maximum allowed body size (default 32k). You can increase the size of the max event by setting flume.event.max.size.bytes in your flume-site.xml file to a larger value. We are addressing this with issue FLUME-712.

I'm getting OutOfMemoryExceptions in my collectors or agents.

Add -XX:+HeapDumpOnOutOfMemoryError to the JVM_MEM_OPTS env variable or flume-env.sh file. This should dump heap upon thse kinds of errors and allow you to determine what objects are consuming excessive memory by using the jhat java heap viewer program.

There have been instances of queues that are unbounded. Several of these have been fixed in v0.9.5.

There are situations where queue sizes are too large for certain messages. For example, if batching is used, each event can takes up more memory. The default queue size in thrift sources is 1000 items. With batching individual events can become megabytes in size which may cause memory exhaustion. For example making batches of 1000 1000-byte messages with a queue of 1000 events could result in flume requiring 1GB of memory!

In these cases, reduced the size of the thrift queue to bound potential the memory usage by setting flume.thrift.queuesize

<property> 
<name>flume.thrift.queuesize</name> 
<value>500</value> 
</property>