...
Code Block |
---|
tailDir("/var/log/app", ".*", startFromEnd=true, recurseDepth=1) // positional args by default cannot be used as kwargs |
I'm new and I'm having a problem using dfs
, customDfs
/formatDfs
, or escapedCustomDfs
/escapedFormatDfs
sinks.
You should use the collectorSink
. It is sufficient for most users and greatly simplifies configuration. The others sinks mentioned above are "low-level" and exposed for advanced users. HDFS files are not durable until they close or are synced, and these sinks do not automatically do this. The collectorSink
is smarter and handles periodic closing of files.
Agent Side
Collector Side
Can I control the level of HDFS replication / block size / other client HDFS property?
Yes. HDFS block size and replication level are HDFS client parameters, so you should expect them to be set by client. The parameters you get are probably coming from hadoop-core.*.jar file (it usually contains hdfs-default.xml and friends). If you want to overwrite the default parameters, you need to set dfs.block.size and dfs.replication in your hdfs-site.xml or flume-site.xml file
What is a good amount of time for collector rolling?
...