Data Format

Camel supports a pluggable DataFormat to allow messages to be marshalled to and from binary or text formats to support a kind of Message Translator.

The following data formats are currently supported:

And related is the following:

• DataFormat Component for working with Data Formats as if it was a regular Component supporting Endpoints and URIs.
• Dozer Type Conversion using Dozer for type converting POJOs

Unmarshalling

If you receive a message from one of the Camel Components such as File, HTTP or JMS you often want to unmarshal the payload into some bean so that you can process it using some Bean Integration or perform Predicate evaluation and so forth. To do this use the unmarshal word in the DSL in Java or the Xml Configuration.

For example

DataFormat jaxb = new JaxbDataFormat("com.acme.model");

from("activemq:My.Queue").
  unmarshal(jaxb).
  to("mqseries:Another.Queue");

The above uses a named DataFormat of jaxb which is configured with a number of Java package names. You can if you prefer use a named reference to a data format which can then be defined in your Registry such as via your Spring XML file.

You can also use the DSL itself to define the data format as you use it. For example the following uses Java serialization to unmarshal a binary file then send it as an ObjectMessage to ActiveMQ

from("file://foo/bar").
  unmarshal().serialization().
  to("activemq:Some.Queue");

Marshalling

Marshalling is the opposite of unmarshalling, where a bean is marshalled into some binary or textual format for transmission over some transport via a Camel Component. Marshalling is used in the same way as unmarshalling above; in the DSL you can use a DataFormat instance, you can configure the DataFormat dynamically using the DSL or you can refer to a named instance of the format in the Registry.

The following example unmarshals via serialization then marshals using a named JAXB data format to perform a kind of Message Translator

from("file://foo/bar").
  unmarshal().serialization(). 
  marshal("jaxb").
  to("activemq:Some.Queue");

Using Spring XML

This example shows how to configure the data type just once and reuse it on multiple routes

You can also define reusable data formats as Spring beans

<bean id="myJaxb" class="org.apache.camel.model.dataformat.JaxbDataFormat">
  <property name="prettyPrint" value="true"/>
  <property name="contextPath" value="org.apache.camel.example"/>
</bean>  

XStream

XStream is a Data Format which uses the XStream library to marshal and unmarshal Java objects to and from XML.

To use XStream in your camel routes you need to add the a dependency on camel-xstream which implements this data format.

Maven users will need to add the following dependency to their pom.xml for this component:

Using the Java DSL

If you would like to configure the XStream instance used by the Camel for the message transformation, you can simply pass a reference to that instance on the DSL level.

XMLInputFactory and XMLOutputFactory

The XStream library uses the javax.xml.stream.XMLInputFactory and javax.xml.stream.XMLOutputFactory, you can control which implementation of this factory should be used.

The Factory is discovered using this algorithm:
1. Use the javax.xml.stream.XMLInputFactory , javax.xml.stream.XMLOutputFactory system property.
2. Use the lib/xml.stream.properties file in the JRE_HOME directory.
3. Use the Services API, if available, to determine the classname by looking in the META-INF/services/javax.xml.stream.XMLInputFactory, META-INF/services/javax.xml.stream.XMLOutputFactory files in jars available to the JRE.
4. Use the platform default XMLInputFactory,XMLOutputFactory instance.

How to set the XML encoding in Xstream DataFormat?

From Camel 2.2.0, you can set the encoding of XML in Xstream DataFormat by setting the Exchange's property with the key Exchange.CHARSET_NAME, or setting the encoding property on Xstream from DSL or Spring config.

{snippet:id=e1|lang=xml|url=camel/trunk/components/camel-xstream/src/test/resources/org/apache/camel/dataformat/xstream/SpringMarshalListTest.xml}

Setting the type permissions of Xstream DataFormat

In Camel, one can always use its own processing step in the route to filter and block certain XML documents to be routed to the XStream's unmarhall step. From Camel 2.16.1, 2.15.5, you can set XStream's type permissions to automatically allow or deny the instantiation of certain types.

The default type permissions setting used by Camel denies all types except for those from java.lang and java.util packages. This setting can be changed by setting System property org.apache.camel.xstream.permissions. Its value is a string of comma-separated permission terms, each representing a type being allowed or denied, depending on whether the term is prefixed with '+' (note '+' may be omitted) or with '-', respectively.

Each term may contain a wildcard character '*'. For example, value "-*,java.lang.*,java.util.*" indicates denying all types except for java.lang.* and java.util.* classes. Setting this value to an empty string "" reverts to the default XStream's type permissions handling which denies certain blacklisted classes and allow others.

The type permissions setting can be extended at an individual XStream DataFormat instance by setting its type permissions property.

HL7 DataFormat

The HL7 component ships with a HL7 data format that can be used to marshal or unmarshal HL7 model objects.

• marshal = from Message to byte stream (can be used when responding using the HL7 MLLP codec)
• unmarshal = from byte stream to Message (can be used when receiving streamed data from the HL7 MLLP

To use the data format, simply instantiate an instance and invoke the marshal or unmarshal operation in the route builder:

  DataFormat hl7 = new HL7DataFormat();
  ...
  from("direct:hl7in").marshal(hl7).to("jms:queue:hl7out");

In the sample above, the HL7 is marshalled from a HAPI Message object to a byte stream and put on a JMS queue.
The next example is the opposite:

  DataFormat hl7 = new HL7DataFormat();
  ...
  from("jms:queue:hl7out").unmarshal(hl7).to("patientLookupService");

Here we unmarshal the byte stream into a HAPI Message object that is passed to our patient lookup service.

There is a shorthand syntax in Camel for well-known data formats that are commonly used.
Then you don't need to create an instance of the HL7DataFormat object:

  from("direct:hl7in").marshal().hl7().to("jms:queue:hl7out");
  from("jms:queue:hl7out").unmarshal().hl7().to("patientLookupService");

EDI DataFormat

We encourage end users to look at the Smooks which supports EDI and Camel natively.

Flatpack DataFormat

The Flatpack component ships with the Flatpack data format that can be used to format between fixed width or delimited text messages to a List of rows as Map.

• marshal = from List<Map<String, Object>> to OutputStream (can be converted to String)
• unmarshal = from java.io.InputStream (such as a File or String) to a java.util.List as an org.apache.camel.component.flatpack.DataSetList instance.
  The result of the operation will contain all the data. If you need to process each row one by one you can split the exchange, using Splitter.

Notice: The Flatpack library does currently not support header and trailers for the marshal operation.

Options

The data format has the following options:

Usage

To use the data format, simply instantiate an instance and invoke the marshal or unmarshal operation in the route builder:

  FlatpackDataFormat fp = new FlatpackDataFormat();
  fp.setDefinition(new ClassPathResource("INVENTORY-Delimited.pzmap.xml"));
  ...
  from("file:order/in").unmarshal(df).to("seda:queue:neworder");

The sample above will read files from the order/in folder and unmarshal the input using the Flatpack configuration file INVENTORY-Delimited.pzmap.xml that configures the structure of the files. The result is a DataSetList object we store on the SEDA queue.

FlatpackDataFormat df = new FlatpackDataFormat();
df.setDefinition(new ClassPathResource("PEOPLE-FixedLength.pzmap.xml"));
df.setFixed(true);
df.setIgnoreFirstRecord(false);

from("seda:people").marshal(df).convertBodyTo(String.class).to("jms:queue:people");

In the code above we marshal the data from a Object representation as a List of rows as Maps. The rows as Map contains the column name as the key, and the the corresponding value. This structure can be created in Java code from e.g. a processor. We marshal the data according to the Flatpack format and convert the result as a String object and store it on a JMS queue.

Dependencies

To use Flatpack in your camel routes you need to add the a dependency on camel-flatpack which implements this data format.

If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions).

<dependency>
  <groupId>org.apache.camel</groupId>
  <artifactId>camel-flatpack</artifactId>
  <version>x.x.x</version>
</dependency>

Options

There are no options provided for this data format.

Marshal

In this example we marshal a regular text/XML payload to a compressed payload employing gzip compression format and send it an ActiveMQ queue called MY_QUEUE.

from("direct:start").marshal().gzip().to("activemq:queue:MY_QUEUE");

Unmarshal

In this example we unmarshal a gzipped payload from an ActiveMQ queue called MY_QUEUE to its original format, and forward it for processing to the UnGZippedMessageProcessor.

from("activemq:queue:MY_QUEUE").unmarshal().gzip().process(new UnGZippedMessageProcessor()); 

Dependencies

This data format is provided in camel-core so no additional dependencies is needed.

Syslog DataFormat

Available as of Camel 2.6

The syslog dataformat is used for working with RFC3164 and RFC5424 messages.

This component supports the following:

• UDP consumption of syslog messages
• Agnostic data format using either plain String objects or SyslogMessage model objects.
• Type Converter from/to SyslogMessage and String
• Integration with the camel-mina component.
• Integration with the camel-netty component.
• Camel 2.14: Encoder and decoder for the camel-netty component.
• Camel 2.14: Support for RFC5424 also.

Maven users will need to add the following dependency to their pom.xml for this component:

<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-syslog</artifactId>
    <version>x.x.x</version>
    <!-- use the same version as your Camel core version -->
</dependency>

RFC3164 Syslog protocol

Syslog uses the user datagram protocol (UDP) 1 as its underlying transport layer mechanism.
The UDP port that has been assigned to syslog is 514.

To expose a Syslog listener service we reuse the existing camel-mina component or camel-netty where we just use the Rfc3164SyslogDataFormat to marshal and unmarshal messages. Notice that from Camel 2.14 onwards the syslog dataformat is renamed to SyslogDataFormat.

RFC5424 Syslog protocol

Available as of Camel 2.14

To expose a Syslog listener service we reuse the existing camel-mina component or camel-netty where we just use the SyslogDataFormat to marshal and unmarshal messages

Exposing a Syslog listener

In our Spring XML file, we configure an endpoint to listen for udp messages on port 10514, note that in netty we disable the defaultCodec, this
will allow a fallback to a NettyTypeConverter and delivers the message as an InputStream:

<camelContext id="myCamel" xmlns="http://camel.apache.org/schema/spring">

    <dataFormats>
          <syslog id="mySyslog"/>
    </dataFormats>

    <route>
          <from uri="netty:udp://localhost:10514?sync=false&amp;allowDefaultCodec=false"/>
          <unmarshal ref="mySyslog"/>
          <to uri="mock:stop1"/>
    </route>

</camelContext>

The same route using camel-mina

<camelContext id="myCamel" xmlns="http://camel.apache.org/schema/spring">

    <dataFormats>
          <syslog id="mySyslog"/>
    </dataFormats>

    <route>
          <from uri="mina:udp://localhost:10514"/>
          <unmarshal ref="mySyslog"/>
          <to uri="mock:stop1"/>
    </route>

</camelContext>

Sending syslog messages to a remote destination

<camelContext id="myCamel" xmlns="http://camel.apache.org/schema/spring">

    <dataFormats>
        <syslog id="mySyslog"/>
    </dataFormats>

    <route>
        <from uri="direct:syslogMessages"/>
        <marshal ref="mySyslog"/>
        <to uri="mina:udp://remotehost:10514"/>
    </route>

</camelContext>