You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

HDFS Component

The hdfs component enables you to read and write messages from/to an HDFS file system. HDFS is the distributed file system at the heart of Hadoop.

URI format

hdfs://hostname[:port][/path][?options]

You can append query options to the URI in the following format, ?option=value&option=value&...
The path is treated in the following way:

  1. as a consumer, if it's a file, it just reads the file, otherwise if it represents a directory it scans all the file under the path satisfying the configured pattern. All the files under that directory must be of the same type.
  2. as a producer, if at least one split strategy is defined, the path is considered a directory and under that directory the producer creates a different file per split named seg0, seg1, seg2, etc.

Options

Name

Default Value

Description

overwrite

true

The file can be overwritten

bufferSize

4096

The buffer size used by HDFS

replication

3

The HDFS replication factor

blockSize

67108864

The size of the HDFS blocks

fileType

NORMAL_FILE

It can be SEQUENCE_FILE,
MAP_FILE, ARRAY_FILE, or
BLOOMMAP_FILE, see Hadoop

fileSystemType

HDFS

It can be LOCAL for local filesystem

keyType

NULL

The type for the key in case of
sequence or map files. See below.

valueType

TEXT

The type for the key in case of
sequence or map files. See below.

splitStrategy

 

A string describing the strategy on
how to split the file based on different
criteria. See below.

openedSuffix

opened

When a file is opened for reading/
writing the file is renamed with this
suffix to avoid to read it during the
writing phase.

readSuffix

read

Once the file has been read is
renamed with this suffix to avoid to
read it again.

initialDelay

0

For the consumer, how much to wait
before to start scanning the directory.

delay

0

Then interval between the directory
scans.

pattern

*

The pattern used for scanning the
directory

chunkSize

4096

When reading a normal file, this is split
into chunks producing a message per
chunk

HDFS Usage Samples

Example 1:

Example 2:
  • No labels