You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »

General

I'm having a hard time with the argument syntax in the catalog section of the manual. Help!

The general form is 

sinkName(reqArg1, reqArg2[, optArg1="default" [optArg2=0]]{, kwarg1="default", kwarg2=0})

reqArg1 and reqArg2 are positional arguments and required in all instances. [ ] chars enclose optional positional arguments. All optional arguments have a default value and must be enumerated in order. Thus optArg1 and optArg2 are optional positional arguments, and have defaults that get filled in if the are not present. { } chars enclose optional keyword arguments. All keyword arguments are optional and have a default value and can be enumerated in any order. Thus kwarg1 and kwarg2 are keyword arguments with defaults.

Let's take tailDir as an example. Here's the definition in the manual.

tailDir("dirname"[, fileregex=".*"[, startFromEnd=false[, recurseDepth=0]]]{,delim="regex", delimMode="exclude|prev|next"})

Here are some valid examples:

tailDir("/var/log/app")            // all files 
tailDir("/var/log/app",".*\.log")  // all files with names that match the ".*\.log" regex (in shell this is *.log)
tailDir("/var/log/app",".*\.log", false, 1)  // all files with names that match the ".*\.log" regex, starting from beginning of file, with one level of recursion depth.
tailDir("/var/log/app", delim="\n\n", delimMode="exclude")  // all files with names that match the ".*\.log" regex, starting from beginning of file, with one level of recursion depth, that end with double new lines, excluding the double new lines
tailDir("/var/log/app",".*\.log", false, 1, delim="\n\n", delimMode="exclude")  // all files with names that match the ".*\.log" regex, starting from beginning of file, with one level of recursion depth, that end with double new lines, excluding the double new lines

Here are some invalid examples (should fail):

tailDir()                                            // must have at least one arg
tailDir("/var/log/app", ".*", startFromEnd=true, 1)  // positional args by default cannot be used as kwargs

Here are some currently valid but likely not what you want examples.

tailDir("/var/log/app", ".*", startFromEnd=true, recurseDepth=1)  // positional args by default cannot be used as kwargs

I'm new and I'm having a problem using dfs, customDfs, or escapedCustomDfs sinks.

You should use the collectorSink. It is sufficient for most users and greatly simplifies configuration. The others are "low-level" and exposed for advanced users.

Agent Side

Collector Side

What is a good amount of time for collector rolling?

Plugins

I have a plugin that uses version xxx of Thrift and Flume is using version yyy.

Thrift versions have been wire-compatible from 0.5.0 to 0.6.0. Thus an application with a thrift 0.5.0 server should accept data from a thrift 0.6.0 client and vice-versa. I believe it has been wire compatible since 0.2.0 (NEEDS verification). The API generated code by the thrift compiler and the runtime libraries for java jars however, break compatibility. This will require a regeneration of thrift generated code. We suggest modifying the plugin as opposed to modifying flume or the target application.

  • No labels