Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: add heading for How to Write Your Own SerDe

...

See SerDe for detailed information about input and output processing. Also see Storage Formats in the HCatalog manual, including CTAS Issue with JSON SerDe. For information about how to create a table with a custom or native SerDe, see Row Format, Storage Format, and SerDe.

How to

...

Write Your Own SerDe

  • In most cases, users want to write a Deserializer instead of a SerDe, because users just want to read their own data format instead of writing to it.
  • For example, the RegexDeserializer will deserialize the data using the configuration parameter 'regex', and possibly a list of column names (see serde2.MetadataTypedColumnsetSerDe). Please see serde2/Deserializer.java for details.
  • If your SerDe supports DDL (basically, SerDe with parameterized columns and column types), you probably want to implement a Protocol based on DynamicSerDe, instead of writing a SerDe from scratch. The reason is that the framework passes DDL to SerDe through "Thrift DDL" format, and it's non-trivial to write a "Thrift DDL" parser.
  • For examples, see SerDe - how to add a new SerDe below.

...