Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
tailDir("/var/log/app", ".*", startFromEnd=true, recurseDepth=1)  // positional args by default cannot be used as kwargs

I'm new and I'm having a problem using dfs, customDfs/formatDfs, or escapedCustomDfs/escapedFormatDfs sinks.

You should use the collectorSink. It is sufficient for most users and greatly simplifies configuration. The others sinks mentioned above are "low-level" and exposed for advanced users. HDFS files are not durable until they close or are synced, and these sinks do not automatically do this. The collectorSink is smarter and handles periodic closing of files.

Agent Side

Collector Side

Can I control the level of HDFS replication / block size / other client HDFS property?

Yes. HDFS block size and replication level are HDFS client parameters, so you should expect them to be set by client. The parameters you get are probably coming from hadoop-core.*.jar file (it usually contains hdfs-default.xml and friends). If you want to overwrite the default parameters, you need to set dfs.block.size and dfs.replication in your hdfs-site.xml or flume-site.xml file

What is a good amount of time for collector rolling?

...