EventSerializer interface (HDFS & FILE_ROLL sinks)
The EventSerializer interface allows arbitrary serialization of an event. Most users should not have to implement this interface directly, although it is an option.
The recommended usage is to serialize your data as Avro. This is a great format that has a lot of advantages over platform- and language-specific serialization formats. For Avro-serialized events you have two options: avro_event
and a custom subclass of AbstractAvroEventSerializer.
Config file syntax
agent.sinks.svc_7_sink.serializer = avro_event agent.sinks.svc_7_sink.serializer.compressionCodec = snappy
Examples
Example config using avro_event with HDFS sink
agent.sources = svc_0_src agent.channels = svc_0_chan agent.sinks = svc_0_sink # Configuration for svc_0 agent.channels.svc_0_chan.type = memory agent.channels.svc_0_chan.capacity = 100000 agent.channels.svc_0_chan.transactionCapacity = 1000 agent.sources.svc_0_src.type = org.apache.flume.source.SyslogTcpSource agent.sources.svc_0_src.port = 10001 agent.sources.svc_0_src.channels = svc_0_chan agent.sinks.svc_0_sink.type = hdfs agent.sinks.svc_0_sink.hdfs.fileType = DataStream agent.sinks.svc_0_sink.hdfs.rollInterval = 300 agent.sinks.svc_0_sink.hdfs.rollSize = 0 agent.sinks.svc_0_sink.hdfs.rollCount = 0 agent.sinks.svc_0_sink.hdfs.batchSize = 1000 agent.sinks.svc_0_sink.hdfs.txnEventMax = 1000 agent.sinks.svc_0_sink.hdfs.path = hdfs://xxxxxxxxxx/user/mpercy/logs/20120521 agent.sinks.svc_0_sink.serializer = avro_event agent.sinks.svc_0_sink.serializer.compressionCodec = snappy agent.sinks.svc_0_sink.channel = svc_0_chan
Examples for using AbstractAvroEventSerializer to write a custom schema
An example is provided as a unit test:
https://svn.apache.org/viewvc/incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java?view=markup
Additional unit tests / examples:
https://svn.apache.org/viewvc/incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/