You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

EventSerializer interface (HDFS & FILE_ROLL sinks)

The EventSerializer interface allows arbitrary serialization of an event. Most users should not have to implement this interface directly, although it is an option.

The recommended usage is to serialize your data as Avro. This is a great format that has a lot of advantages over platform- and language-specific serialization formats. For Avro-serialized events you have two options: avro_event and a custom subclass of AbstractAvroEventSerializer.

Config file syntax

agent.sinks.svc_7_sink.serializer = avro_event
agent.sinks.svc_7_sink.serializer.compressionCodec = snappy

Examples

Example config using avro_event with HDFS sink

agent.sources = svc_0_src
agent.channels = svc_0_chan
agent.sinks = svc_0_sink

# Configuration for svc_0
agent.channels.svc_0_chan.type = memory
agent.channels.svc_0_chan.capacity = 100000
agent.channels.svc_0_chan.transactionCapacity = 1000

agent.sources.svc_0_src.type = org.apache.flume.source.SyslogTcpSource
agent.sources.svc_0_src.port = 10001
agent.sources.svc_0_src.channels = svc_0_chan

agent.sinks.svc_0_sink.type = hdfs
agent.sinks.svc_0_sink.hdfs.fileType = DataStream
agent.sinks.svc_0_sink.hdfs.rollInterval = 300 
agent.sinks.svc_0_sink.hdfs.rollSize = 0 
agent.sinks.svc_0_sink.hdfs.rollCount = 0 
agent.sinks.svc_0_sink.hdfs.batchSize = 1000
agent.sinks.svc_0_sink.hdfs.txnEventMax = 1000
agent.sinks.svc_0_sink.hdfs.path = hdfs://xxxxxxxxxx/user/mpercy/logs/20120521
agent.sinks.svc_0_sink.serializer = avro_event
agent.sinks.svc_0_sink.serializer.compressionCodec = snappy
agent.sinks.svc_0_sink.channel = svc_0_chan

Examples for using AbstractAvroEventSerializer to write a custom schema

An example is provided as a unit test: https://svn.apache.org/viewvc/incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java?view=markup

Additional unit tests / examples: https://svn.apache.org/viewvc/incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/

  • No labels