Versions Compared

Old Version 9

changes.mady.by.user Michael Percy Percy

Saved on

compared with

New Version 10

changes.mady.by.user Michael Percy Percy

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Fragment of flume.conf config file

# Number of
Code Block
langtext
  1. Number of sources, 
...
  1.  channels, 
...
  1.  and 
...
  1.  sinks 
...
  1.  varied 
...
  1.  depending 
...
  1.  on 
...
  1.  tests.
...
  2. In 
...
  1.  each 
...
  1.  case, 
...
  1.  they 
...
  1.  are 
...
  1.  independent 
...
  1.  flows 
...
  1.  and 
...
  1.  therefore 
...
  1.  do 
...
  1.  not 
...
  1.  share 
...
  1.  threads, 
...
  1.  data, 
...
  1.  or 
...
  1.  resources.
...
  2. This 
...
  1.  example 
...
  1.  only 
...
  1.  shows 
...
  1.  3 
...
  1.  flows. 
...
  1.  The 
...
  1.  # 
...
  1.  of 
...
  1.  flows 
...
  1.  were 
...
  1.  varied 
...
  1.  from 
...
  1.  6 
...
  1.  to 
...
  1.  16.
...

  1.  agent.sources 
...
  1.  = 
...
  1.  svc_0_src 
...
  1.  svc_1_src 
...
  1.  svc_2_src
...

  1.  agent.channels 
...
  1.  = 
...
  1.  svc_0_chan 
...
  1.  svc_1_chan 
...
  1.  svc_2_chan
...

  1.  agent.sinks 
...
  1.  = 
...
  1.  svc_0_sink 
...
  1.  svc_1_sink 
...
  1.  svc_2_sink
...
  1. example 
...
  1.  of 
...
  1.  one 
...
  1.  flow 
...
  1.  is 
...
  1.  below, 
...
  1.  i.e. 
...
  1.  "flow 
...
  1.  0"
...

  1.  agent.channels.svc_0_chan.type 
...
  1.  = 
...
  1.  memory
...

  1.  agent.channels.svc_0_chan.capacity 
...
  1.  = 
...
  1.  100000
...

  1.  agent.channels.svc_0_chan.transactionCapacity 
...
  1.  = 
...
  1.  1000
...
agent.sources.svc_0_src.type 
...
 = 
...
 org.apache.flume.source.SyslogTcpSource
...

 agent.sources.svc_0_src.port 
...
 = 
...
 10001
...

 agent.sources.svc_0_src.channels 
...
 = 
...
 svc_0_chan
...
agent.sinks.svc_0_sink.type 
...
 = 
...
 hdfs
...

 agent.sinks.svc_0_sink.hdfs.path 
...
 = 
...
 hdfs://xxxxxx.cloudera.com/service/20120430/flow0
...

 agent.sinks.svc_0_sink.hdfs.fileType 
...
 = 
...
 DataStream
...

 agent.sinks.svc_0_sink.hdfs.rollInterval 
...
 = 
...
 300
...

 agent.sinks.svc_0_sink.hdfs.rollSize 
...
 = 
...
 0
...

 agent.sinks.svc_0_sink.hdfs.rollCount 
...
 = 
...
 0
...

 agent.sinks.svc_0_sink.hdfs.batchSize 
...
 = 
...
 1000
...

 agent.sinks.svc_0_sink.hdfs.txnEventMax 
...
 = 
...
 1000
...

 agent.sinks.svc_0_sink.hdfs.kerberosPrincipal 
...
 = 
...
 flume/_HOST@CLOUDERA.COM
...

 agent.sinks.svc_0_sink.hdfs.kerberosKeytab 
...
 = 
...
 /etc/flume-ng/conf/flume-xxxxxx.keytab
...

 agent.sinks.svc_0_sink.serializer 
...
 = 
...
 avro_event
...

 agent.sinks.svc_0_sink.serializer.compressionCodec 
...
 = 
...
 snappy
...

 agent.sinks.svc_0_sink.channel 
...
 = 
...
 svc_0_chan
...
  1. ... 
...
  1.  define 
...
  1.  flow 
...
  1.  1 
...
  1.  ... 
...
  1.  Code Block
Hadoop configuration
 The HDFS sink was connected to a 9-node Hadoop cluster running CDH3u3 with MIT Kerberos v5 security enabled.
...