Page History

...

Code Block
hdfs://hostname[:port][/path][?options]

You can append query options to the URI in the following format, ?option=value&option=value&...
The path is treated in the following way:

as a consumer, if it's a file, it just reads the file, otherwise if it represents a directory it scans all the file under the path satisfying the configured pattern. All the files under that directory must be of the same type.
as a producer, if at least one split strategy is defined, the path is considered a directory and under that directory the producer creates a different file per split named seg0, seg1, seg2, etc.

Name	Default Value	Description
`AAAA`	`1000`	BBBB

Header	Description

...


`overwrite`	`true`	The file can be overwritten
`bufferSize`	`4096`	The buffer size used by HDFS
`replication`	`3`	The HDFS replication factor
`blockSize`	`67108864`	The size of the HDFS blocks
`fileType`	`NORMAL_FILE`	It can be SEQUENCE_FILE, MAP_FILE, ARRAY_FILE, or BLOOMMAP_FILE, see Hadoop
`fileSystemType`	`HDFS`	It can be LOCAL for local filesystem
`keyType`	`NULL`	The type for the key in case of sequence or map files. See below.
`valueType`	`TEXT`	The type for the key in case of sequence or map files. See below.
`splitStrategy`		A string describing the strategy on how to split the file based on different criteria. See below.
`openedSuffix`	`opened`	When a file is opened for reading/ writing the file is renamed with this suffix to avoid to read it during the writing phase.
`readSuffix`	`read`	Once the file has been read is renamed with this suffix to avoid to read it again.
`initialDelay`	`0`	For the consumer, how much to wait before to start scanning the directory.
`delay`	`0`	Then interval between the directory scans.
`pattern`	`*`	The pattern used for scanning the directory
`chunkSize`	`4096`	When reading a normal file, this is split into chunks producing a message per chunk

...

Versions Compared