Data Format

Camel supports a pluggable DataFormat to allow messages to be marshalled to and from binary or text formats to support a kind of Message Translator.

The following data formats are currently supported:

And related is the following:

• DataFormat Component for working with Data Formats as if it was a regular Component supporting Endpoints and URIs.
• Dozer Type Conversion using Dozer for type converting POJOs

Unmarshalling

If you receive a message from one of the Camel Components such as File, HTTP or JMS you often want to unmarshal the payload into some bean so that you can process it using some Bean Integration or perform Predicate evaluation and so forth. To do this use the unmarshal word in the DSL in Java or the Xml Configuration.

For example

DataFormat jaxb = new JaxbDataFormat("com.acme.model");

from("activemq:My.Queue").
  unmarshal(jaxb).
  to("mqseries:Another.Queue");

The above uses a named DataFormat of jaxb which is configured with a number of Java package names. You can if you prefer use a named reference to a data format which can then be defined in your Registry such as via your Spring XML file.

You can also use the DSL itself to define the data format as you use it. For example the following uses Java serialization to unmarshal a binary file then send it as an ObjectMessage to ActiveMQ

from("file://foo/bar").
  unmarshal().serialization().
  to("activemq:Some.Queue");

Marshalling

Marshalling is the opposite of unmarshalling, where a bean is marshalled into some binary or textual format for transmission over some transport via a Camel Component. Marshalling is used in the same way as unmarshalling above; in the DSL you can use a DataFormat instance, you can configure the DataFormat dynamically using the DSL or you can refer to a named instance of the format in the Registry.

The following example unmarshals via serialization then marshals using a named JAXB data format to perform a kind of Message Translator

from("file://foo/bar").
  unmarshal().serialization(). 
  marshal("jaxb").
  to("activemq:Some.Queue");

Using Spring XML

This example shows how to configure the data type just once and reuse it on multiple routes

You can also define reusable data formats as Spring beans

<bean id="myJaxb" class="org.apache.camel.model.dataformat.JaxbDataFormat">
  <property name="prettyPrint" value="true"/>
  <property name="contextPath" value="org.apache.camel.example"/>
</bean>  

JAXB

JAXB is a Data Format which uses the JAXB2 XML marshalling standard which is included in Java 6 to unmarshal an XML payload into Java objects or to marshal Java objects into an XML payload.

Using the Java DSL

For example the following uses a named DataFormat of jaxb which is configured with a number of Java package names to initialize the JAXBContext.

You can if you prefer use a named reference to a data format which can then be defined in your Registry such as via your Spring XML file. e.g.

Using Spring XML

The following example shows how to use JAXB to unmarshal using Spring configuring the jaxb data type{snippet:id=example|lang=xml|url=camel/trunk/components/camel-jaxb/src/test/resources/org/apache/camel/example/springDataFormat.xml}This example shows how to configure the data type just once and reuse it on multiple routes.{snippet:id=example|lang=xml|url=camel/trunk/components/camel-jaxb/src/test/resources/org/apache/camel/example/marshalAndUnmarshalWithRef.xml}

Partial marshalling/unmarshalling

This feature is new to Camel 2.2.0.
JAXB 2 supports marshalling and unmarshalling XML tree fragments. By default JAXB looks for @XmlRootElement annotation on given class to operate on whole XML tree. This is useful but not always - sometimes generated code does not have @XmlRootElement annotation, sometimes you need unmarshall only part of tree.
In that case you can use partial unmarshalling. To enable this behaviours you need set property partClass. Camel will pass this class to JAXB's unmarshaler.{snippet:id=example|lang=xml|url=camel/trunk/components/camel-jaxb/src/test/resources/org/apache/camel/example/springDataFormatPartial.xml}For marshalling you have to add partNamespace attribute with QName of destination namespace. Example of Spring DSL you can find above.

Fragment

This feature is new to Camel 2.8.0.
JaxbDataFormat has new property fragment which can set the the Marshaller.JAXB_FRAGMENT encoding property on the JAXB Marshaller. If you don't want the JAXB Marshaller to generate the XML declaration, you can set this option to be true. The default value of this property is false.

Ignoring the NonXML Character

This feature is new to Camel 2.2.0.
JaxbDataFromat supports to ignore the NonXML Character, you just need to set the filterNonXmlChars property to be true, JaxbDataFormat will replace the NonXML character with " " when it is marshaling or unmarshaling the message. You can also do it by setting the Exchange property Exchange.FILTER_NON_XML_CHARS.

This feature has been tested with Woodstox 3.2.9 and Sun JDK 1.6 StAX implementation.

New for Camel 2.12.1
JaxbDataFormat now allows you to customize the XMLStreamWriter used to marshal the stream to XML. Using this configuration, you can add your own stream writer to completely remove, escape, or replace non-xml characters.

The following example shows using the Spring DSL and also enabling Camel's NonXML filtering:

Working with the ObjectFactory

If you use XJC to create the java class from the schema, you will get an ObjectFactory for you JAXB context. Since the ObjectFactory uses JAXBElement to hold the reference of the schema and element instance value, jaxbDataformat will ignore the JAXBElement by default and you will get the element instance value instead of the JAXBElement object form the unmarshaled message body.
If you want to get the JAXBElement object form the unmarshaled message body, you need to set the JaxbDataFormat object's ignoreJAXBElement property to be false.

Setting encoding

You can set the encoding option to use when marshalling. Its the Marshaller.JAXB_ENCODING encoding property on the JAXB Marshaller.
You can setup which encoding to use when you declare the JAXB data format. You can also provide the encoding in the Exchange property Exchange.CHARSET_NAME. This property will overrule the encoding set on the JAXB data format.

In this Spring DSL we have defined to use iso-8859-1 as the encoding:{snippet:id=example|lang=xml|url=camel/trunk/components/camel-jaxb/src/test/resources/org/apache/camel/example/springDataFormatWithEncoding.xml}

Controlling namespace prefix mapping

Available as of Camel 2.11

When marshalling using JAXB or SOAP then the JAXB implementation will automatic assign namespace prefixes, such as ns2, ns3, ns4 etc. To control this mapping, Camel allows you to refer to a map which contains the desired mapping.

Notice this requires having JAXB-RI 2.1 or better (from SUN) on the classpath, as the mapping functionality is dependent on the implementation of JAXB, whether its supported.

For example in Spring XML we can define a Map with the mapping. In the mapping file below, we map SOAP to use soap as prefix. While our custom namespace "http://www.mycompany.com/foo/2" is not using any prefix.

To use this in JAXB or SOAP you refer to this map, using the namespacePrefixRef attribute as shown below. Then Camel will lookup in the Registry a java.util.Map with the id "myMap", which was what we defined above.

Schema validation

Available as of Camel 2.11

The JAXB Data Format supports validation by marshalling and unmarshalling from/to XML. Your can use the prefix classpath:, file:* or *http: to specify how the resource should by resolved. You can separate multiple schema files by using the ',' character.

Using the Java DSL, you can configure it in the following way:

You can do the same using the XML DSL:

Camel will create and pool the underling SchemaFactory instances on the fly, because the SchemaFactory shipped with the JDK is not thread safe.
However, if you have a SchemaFactory implementation which is thread safe, you can configure the JAXB data format to use this one:

Schema Location

Available as of Camel 2.14

The JAXB Data Format supports to specify the SchemaLocation when marshaling the XML. 

Using the Java DSL, you can configure it in the following way:

You can do the same using the XML DSL:

Marshal data that is already XML

Available as of Camel 2.14.1

Dependencies

To use JAXB in your camel routes you need to add the a dependency on camel-jaxb which implements this data format.

If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions).

XStream

XStream is a Data Format which uses the XStream library to marshal and unmarshal Java objects to and from XML.

To use XStream in your camel routes you need to add the a dependency on camel-xstream which implements this data format.

Maven users will need to add the following dependency to their pom.xml for this component:

Using the Java DSL

If you would like to configure the XStream instance used by the Camel for the message transformation, you can simply pass a reference to that instance on the DSL level.

XMLInputFactory and XMLOutputFactory

The XStream library uses the javax.xml.stream.XMLInputFactory and javax.xml.stream.XMLOutputFactory, you can control which implementation of this factory should be used.

The Factory is discovered using this algorithm:
1. Use the javax.xml.stream.XMLInputFactory , javax.xml.stream.XMLOutputFactory system property.
2. Use the lib/xml.stream.properties file in the JRE_HOME directory.
3. Use the Services API, if available, to determine the classname by looking in the META-INF/services/javax.xml.stream.XMLInputFactory, META-INF/services/javax.xml.stream.XMLOutputFactory files in jars available to the JRE.
4. Use the platform default XMLInputFactory,XMLOutputFactory instance.

How to set the XML encoding in Xstream DataFormat?

From Camel 2.2.0, you can set the encoding of XML in Xstream DataFormat by setting the Exchange's property with the key Exchange.CHARSET_NAME, or setting the encoding property on Xstream from DSL or Spring config.

{snippet:id=e1|lang=xml|url=camel/trunk/components/camel-xstream/src/test/resources/org/apache/camel/dataformat/xstream/SpringMarshalListTest.xml}

Setting the type permissions of Xstream DataFormat

In Camel, one can always use its own processing step in the route to filter and block certain XML documents to be routed to the XStream's unmarhall step. From Camel 2.16.1, 2.15.5, you can set XStream's type permissions to automatically allow or deny the instantiation of certain types.

The default type permissions setting used by Camel denies all types except for those from java.lang and java.util packages. This setting can be changed by setting System property org.apache.camel.xstream.permissions. Its value is a string of comma-separated permission terms, each representing a type being allowed or denied, depending on whether the term is prefixed with '+' (note '+' may be omitted) or with '-', respectively.

Each term may contain a wildcard character '*'. For example, value "-*,java.lang.*,java.util.*" indicates denying all types except for java.lang.* and java.util.* classes. Setting this value to an empty string "" reverts to the default XStream's type permissions handling which denies certain blacklisted classes and allow others.

The type permissions setting can be extended at an individual XStream DataFormat instance by setting its type permissions property.

CSV

The CSV Data Format uses Apache Commons CSV to handle CSV payloads (Comma Separated Values) such as those exported/imported by Excel.

As of Camel 2.15.0, it now uses the Apache Commons CSV 1.1 which is based on a completely different set of options.

Available options until Camel 2.15

Available options as of Camel 2.15

<csv >
    <header>orderId</header>
    <header>amount</header>
</csv>

Marshalling a Map to CSV

The component allows you to marshal a Java Map (or any other message type that can be converted in a Map) into a CSV payload.

Map<String, Object> body = new LinkedHashMap<>();
body.put("foo", "abc");
body.put("bar", 123);

from("direct:start")
    .marshal().csv()
    .to("mock:result");

<route>
    <from uri="direct:start" />
    <marshal>
        <csv />
    </marshal>
    <to uri="mock:result" />
</route>

abc,123

Unmarshalling a CSV message into a Java List

Unmarshalling will transform a CSV messsage into a Java List with CSV file lines (containing another List with all the field values).

An example: we have a CSV file with names of persons, their IQ and their current activity.

Jack Dalton, 115, mad at Averell
Joe Dalton, 105, calming Joe
William Dalton, 105, keeping Joe from killing Averell
Averell Dalton, 80, playing with Rantanplan
Lucky Luke, 120, capturing the Daltons

We can now use the CSV component to unmarshal this file:

from("file:src/test/resources/?fileName=daltons.csv&noop=true")
    .unmarshal().csv()
    .to("mock:daltons");

The resulting message will contain a List<List<String>> like...

List<List<String>> data = (List<List<String>>) exchange.getIn().getBody();
for (List<String> line : data) {
    LOG.debug(String.format("%s has an IQ of %s and is currently %s", line.get(0), line.get(1), line.get(2)));
}

Marshalling a List<Map> to CSV

Available as of Camel 2.1

If you have multiple rows of data you want to be marshalled into CSV format you can now store the message payload as a List<Map<String, Object>> object where the list contains a Map for each row.

File Poller of CSV, then unmarshaling

Given a bean which can handle the incoming data...

// Some comments here
public void doHandleCsvData(List<List<String>> csvData)
{
    // do magic here
}

... your route then looks as follows

<route>
        <!-- poll every 10 seconds -->
        <from uri="file:///some/path/to/pickup/csvfiles?delete=true&amp;consumer.delay=10000" />
        <unmarshal><csv /></unmarshal>
        <to uri="bean:myCsvHandler?method=doHandleCsvData" />
</route>

Marshaling with a pipe as delimiter

Map<String, Object> body = new LinkedHashMap<>();
body.put("foo", "abc");
body.put("bar", 123);

// Camel version < 2.15
CsvDataFormat oldCSV = new CsvDataFormat();
oldCSV.setDelimiter("|");
from("direct:start")
    .marshal(oldCSV)
    .to("mock:result")
 
// Camel version >= 2.15
from("direct:start")
    .marshal(new CsvDataFormat().setDelimiter('|'))
    .to("mock:result")

<route>
  <from uri="direct:start" />
  <marshal>
    <csv delimiter="|" />
  </marshal>
  <to uri="mock:result" />
</route>

abc|123

Using autogenColumns, configRef and strategyRef attributes inside XML DSL

Available as of Camel 2.9.2 / 2.10 and deleted for Camel 2.15

You can customize the CSV Data Format to make use of your own CSVConfig and/or CSVStrategy. Also note that the default value of the autogenColumns option is true. The following example should illustrate this customization.

<route>
  <from uri="direct:start" />
  <marshal>
    <!-- make use of a strategy other than the default one which is 'org.apache.commons.csv.CSVStrategy.DEFAULT_STRATEGY' -->
    <csv autogenColumns="false" delimiter="|" configRef="csvConfig" strategyRef="excelStrategy" />
  </marshal>
  <convertBodyTo type="java.lang.String" />
  <to uri="mock:result" />
</route>

<bean id="csvConfig" class="org.apache.commons.csv.writer.CSVConfig">
  <property name="fields">
    <list>
      <bean class="org.apache.commons.csv.writer.CSVField">
        <property name="name" value="orderId" />
      </bean>
      <bean class="org.apache.commons.csv.writer.CSVField">
        <property name="name" value="amount" />
      </bean>
    </list>
  </property>
</bean>

<bean id="excelStrategy" class="org.springframework.beans.factory.config.FieldRetrievingFactoryBean">
  <property name="staticField" value="org.apache.commons.csv.CSVStrategy.EXCEL_STRATEGY" />
</bean>

Using skipFirstLine option while unmarshaling

Available as of Camel 2.10 and deleted for Camel 2.15

You can instruct the CSV Data Format to skip the first line which contains the CSV headers. Using the Spring/XML DSL:

<route>
  <from uri="direct:start" />
  <unmarshal>
    <csv skipFirstLine="true" />
  </unmarshal>
  <to uri="bean:myCsvHandler?method=doHandleCsv" />
</route>

Or the Java DSL:

CsvDataFormat csv = new CsvDataFormat();
csv.setSkipFirstLine(true);

from("direct:start")
  .unmarshal(csv)
.to("bean:myCsvHandler?method=doHandleCsv");

Unmarshaling with a pipe as delimiter

Using the Spring/XML DSL:

<route>
  <from uri="direct:start" />
  <unmarshal>
    <csv delimiter="|" />
  </unmarshal>
  <to uri="bean:myCsvHandler?method=doHandleCsv" />
</route>

Or the Java DSL:

CsvDataFormat csv = new CsvDataFormat();
CSVStrategy strategy = CSVStrategy.DEFAULT_STRATEGY;
strategy.setDelimiter('|');
csv.setStrategy(strategy);

from("direct:start")
  .unmarshal(csv)
  .to("bean:myCsvHandler?method=doHandleCsv");

CsvDataFormat csv = new CsvDataFormat();
csv.setDelimiter("|");

from("direct:start")
  .unmarshal(csv)
  .to("bean:myCsvHandler?method=doHandleCsv");

CsvDataFormat csv = new CsvDataFormat();
CSVConfig csvConfig = new CSVConfig();
csvConfig.setDelimiter(";");
csv.setConfig(csvConfig);

from("direct:start")
  .unmarshal(csv)
  .to("bean:myCsvHandler?method=doHandleCsv");

CSVConfig csvConfig = new CSVConfig();
csvConfig.setDelimiter(';');

Dependencies

To use CSV in your Camel routes you need to add a dependency on camel-csv, which implements this data format.

If you use Maven you can just add the following to your pom.xml, substituting the version number for the latest and greatest release (see the download page for the latest versions).

<dependency>
  <groupId>org.apache.camel</groupId>
  <artifactId>camel-csv</artifactId>
  <version>x.x.x</version>
</dependency>

Options

Marshal

In this example we marshal the file content to String object in UTF-8 encoding.

from("file://data.csv").marshal().string("UTF-8").to("jms://myqueue");

Unmarshal

In this example we unmarshal the payload from the JMS queue to a String object using UTF-8 encoding, before its processed by the newOrder processor.

from("jms://queue/order").unmarshal().string("UTF-8").processRef("newOrder");

Dependencies

This data format is provided in camel-core so no additional dependencies is needed.

EDI DataFormat

We encourage end users to look at the Smooks which supports EDI and Camel natively.

Flatpack DataFormat

The Flatpack component ships with the Flatpack data format that can be used to format between fixed width or delimited text messages to a List of rows as Map.

• marshal = from List<Map<String, Object>> to OutputStream (can be converted to String)
• unmarshal = from java.io.InputStream (such as a File or String) to a java.util.List as an org.apache.camel.component.flatpack.DataSetList instance.
  The result of the operation will contain all the data. If you need to process each row one by one you can split the exchange, using Splitter.

Notice: The Flatpack library does currently not support header and trailers for the marshal operation.

Options

The data format has the following options:

Usage

To use the data format, simply instantiate an instance and invoke the marshal or unmarshal operation in the route builder:

  FlatpackDataFormat fp = new FlatpackDataFormat();
  fp.setDefinition(new ClassPathResource("INVENTORY-Delimited.pzmap.xml"));
  ...
  from("file:order/in").unmarshal(df).to("seda:queue:neworder");

The sample above will read files from the order/in folder and unmarshal the input using the Flatpack configuration file INVENTORY-Delimited.pzmap.xml that configures the structure of the files. The result is a DataSetList object we store on the SEDA queue.

FlatpackDataFormat df = new FlatpackDataFormat();
df.setDefinition(new ClassPathResource("PEOPLE-FixedLength.pzmap.xml"));
df.setFixed(true);
df.setIgnoreFirstRecord(false);

from("seda:people").marshal(df).convertBodyTo(String.class).to("jms:queue:people");

In the code above we marshal the data from a Object representation as a List of rows as Maps. The rows as Map contains the column name as the key, and the the corresponding value. This structure can be created in Java code from e.g. a processor. We marshal the data according to the Flatpack format and convert the result as a String object and store it on a JMS queue.

Dependencies

To use Flatpack in your camel routes you need to add the a dependency on camel-flatpack which implements this data format.

If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions).

<dependency>
  <groupId>org.apache.camel</groupId>
  <artifactId>camel-flatpack</artifactId>
  <version>x.x.x</version>
</dependency>

Bindy

The goal of this component is to allow the parsing/binding of non-structured data (or to be more precise non-XML data)
to/from Java Beans that have binding mappings defined with annotations. Using Bindy, you can bind data from sources such as :

• CSV records,
• Fixed-length records,
• FIX messages,
• or almost any other non-structured data

to one or many Plain Old Java Object (POJO). Bindy converts the data according to the type of the java property. POJOs can be linked together with one-to-many relationships available in some cases. Moreover, for data type like Date, Double, Float, Integer, Short, Long and BigDecimal, you can provide the pattern to apply during the formatting of the property.

For the BigDecimal numbers, you can also define the precision and the decimal or grouping separators.

Decimal* = Double, Integer, Float, Short, Long

To work with camel-bindy, you must first define your model in a package (e.g. com.acme.model) and for each model class (e.g. Order, Client, Instrument, ...) add the required annotations (described hereafter) to the Class or field.

Annotations

The annotations created allow to map different concept of your model to the POJO like :

• Type of record (csv, key value pair (e.g. FIX message), fixed length ...),
• Link (to link object in another object),
• DataField and their properties (int, type, ...),
• KeyValuePairField (for key = value format like we have in FIX financial messages),
• Section (to identify header, body and footer section),
• OneToMany

This section will describe them :

1. CsvRecord

The CsvRecord annotation is used to identified the root class of the model. It represents a record = a line of a CSV file and can be linked to several children model classes.

case 1 : separator = ','

The separator used to segregate the fields in the CSV record is ',' :

10, J, Pauline, M, XD12345678, Fortis Dynamic 15/15, 2500, USD,08-01-2009

@CsvRecord( separator = "," )
public Class Order {
...
}

case 2 : separator = ';'

Compare to the previous case, the separator here is ';' instead of ',' :

10; J; Pauline; M; XD12345678; Fortis Dynamic 15/15; 2500; USD; 08-01-2009

@CsvRecord( separator = ";" )
public Class Order {
...
}

case 3 : separator = '|'

Compare to the previous case, the separator here is '|' instead of ';' :

10| J| Pauline| M| XD12345678| Fortis Dynamic 15/15| 2500| USD| 08-01-2009

@CsvRecord( separator = "\\|" )
public Class Order {
...
}

case 4 : separator = '\",\"'
Applies for Camel 2.8.2 or older

When the field to be parsed of the CSV record contains ',' or ';' which is also used as separator, we whould find another strategy
to tell camel bindy how to handle this case. To define the field containing the data with a comma, you will use simple or double quotes
as delimiter (e.g : '10', 'Street 10, NY', 'USA' or "10", "Street 10, NY", "USA").
Remark : In this case, the first and last character of the line which are a simple or double quotes will removed by bindy

"10","J","Pauline"," M","XD12345678","Fortis Dynamic 15,15" 2500","USD","08-01-2009"

@CsvRecord( separator = "\",\"" )
public Class Order {
...
}

From Camel 2.8.3/2.9 or never bindy will automatic detect if the record is enclosed with either single or double quotes and automatic remove those quotes when unmarshalling from CSV to Object. Therefore do not include the quotes in the separator, but simple do as below:

"10","J","Pauline"," M","XD12345678","Fortis Dynamic 15,15" 2500","USD","08-01-2009"

@CsvRecord( separator = "," )
public Class Order {
...
}

Notice that if you want to marshal from Object to CSV and use quotes, then you need to specify which quote character to use, using the quote attribute on the @CsvRecord as shown below:

@CsvRecord( separator = ",", quote = "\"" )
public Class Order {
...
}

case 5 : separator & skipfirstline

The feature is interesting when the client wants to have in the first line of the file, the name of the data fields :

order id, client id, first name, last name, isin code, instrument name, quantity, currency, date

To inform bindy that this first line must be skipped during the parsing process, then we use the attribute :

@CsvRecord(separator = ",", skipFirstLine = true)
public Class Order {
...
}

case 6 : generateHeaderColumns

To add at the first line of the CSV generated, the attribute generateHeaderColumns must be set to true in the annotation like this :

@CsvRecord( generateHeaderColumns = true )
public Class Order {
...
}

As a result, Bindy during the unmarshaling process will generate CSV like this :

order id, client id, first name, last name, isin code, instrument name, quantity, currency, date
10, J, Pauline, M, XD12345678, Fortis Dynamic 15/15, 2500, USD,08-01-2009

case 7 : carriage return

If the platform where camel-bindy will run is not Windows but Macintosh or Unix, than you can change the crlf property like this. Three values are available : WINDOWS, UNIX or MAC

@CsvRecord(separator = ",", crlf="MAC")
public Class Order {
...
}

Additionally, if for some reason you need to add a different line ending character, you can opt to specify it using the crlf parameter. In the following example, we can end the line with a comma followed by the newline character:

@CsvRecord(separator = ",", crlf=",\n")
public Class Order {
...
}

case 8 : isOrdered

Sometimes, the order to follow during the creation of the CSV record from the model is different from the order used during the parsing. Then, in this case, we can use the attribute isOrdered = true to indicate this in combination with attribute 'position' of the DataField annotation.

@CsvRecord(isOrdered = true)
public Class Order {

   @DataField(pos = 1, position = 11)
   private int orderNr;

   @DataField(pos = 2, position = 10)
   private String clientNr;

...
}

Remark : pos is used to parse the file, stream while positions is used to generate the CSV

2. Link

The link annotation will allow to link objects together.

e.g : If the model Class Client is linked to the Order class, then use annotation Link in the Order class like this :

@CsvRecord(separator = ",")
public class Order {

    @DataField(pos = 1)
    private int orderNr;

    @Link
    private Client client;
...

AND for the class Client :

@Link
public class Client {
...
}

3. DataField

The DataField annotation defines the property of the field. Each datafield is identified by its position in the record, a type (string, int, date, ...) and optionally of a pattern

case 1 : pos

This parameter/attribute represents the position of the field in the csv record

@CsvRecord(separator = ",")
public class Order {

    @DataField(pos = 1)
    private int orderNr;

    @DataField(pos = 5)
    private String isinCode;

...
}

As you can see in this example the position starts at '1' but continues at '5' in the class Order. The numbers from '2' to '4' are defined in the class Client (see here after).

public class Client {

    @DataField(pos = 2)
    private String clientNr;

    @DataField(pos = 3)
    private String firstName;

    @DataField(pos = 4)
    private String lastName;
...
}

case 2 : pattern

The pattern allows to enrich or validates the format of your data

@CsvRecord(separator = ",")
public class Order {

    @DataField(pos = 1)
    private int orderNr;

    @DataField(pos = 5)
    private String isinCode;

    @DataField(name = "Name", pos = 6)
    private String instrumentName;

    @DataField(pos = 7, precision = 2)
    private BigDecimal amount;

    @DataField(pos = 8)
    private String currency;

    @DataField(pos = 9, pattern = "dd-MM-yyyy") -- pattern used during parsing or when the date is created
    private Date orderDate;
...
}

case 3 : precision

The precision is helpful when you want to define the decimal part of your number

@CsvRecord(separator = ",")
public class Order {

    @DataField(pos = 1)
    private int orderNr;

    @Link
    private Client client;

    @DataField(pos = 5)
    private String isinCode;

    @DataField(name = "Name", pos = 6)
    private String instrumentName;

    @DataField(pos = 7, precision = 2) -- precision
    private BigDecimal amount;

    @DataField(pos = 8)
    private String currency;

    @DataField(pos = 9, pattern = "dd-MM-yyyy")
    private Date orderDate;
...
}

case 4 : Position is different in output

The position attribute will inform bindy how to place the field in the CSV record generated. By default, the position used corresponds to the position defined with the attribute 'pos'. If the position is different (that means that we have an asymetric processus comparing marshaling from unmarshaling) than we can use 'position' to indicate this.

Here is an example

@CsvRecord(separator = ",")
public class Order {
@CsvRecord(separator = ",", isOrdered = true)
public class Order {

    // Positions of the fields start from 1 and not from 0

    @DataField(pos = 1, position = 11)
    private int orderNr;

    @DataField(pos = 2, position = 10)
    private String clientNr;

    @DataField(pos = 3, position = 9)
    private String firstName;

    @DataField(pos = 4, position = 8)
    private String lastName;

    @DataField(pos = 5, position = 7)
    private String instrumentCode;

    @DataField(pos = 6, position = 6)
    private String instrumentNumber;
...
}

case 5 : required

If a field is mandatory, simply use the attribute 'required' setted to true

@CsvRecord(separator = ",")
public class Order {

    @DataField(pos = 1)
    private int orderNr;

    @DataField(pos = 2, required = true)
    private String clientNr;

    @DataField(pos = 3, required = true)
    private String firstName;

    @DataField(pos = 4, required = true)
    private String lastName;
...
}

If this field is not present in the record, than an error will be raised by the parser with the following information :

Some fields are missing (optional or mandatory), line :

case 6 : trim

If a field has leading and/or trailing spaces which should be removed before they are processed, simply use the attribute 'trim' setted to true

@CsvRecord(separator = ",")
public class Order {

    @DataField(pos = 1, trim = true)
    private int orderNr;

    @DataField(pos = 2, trim = true)
    private Integer clientNr;

    @DataField(pos = 3, required = true)
    private String firstName;

    @DataField(pos = 4)
    private String lastName;
...
}

case 7 : defaultValue

If a field is not defined then uses the value indicated by the defaultValue attribute

@CsvRecord(separator = ",")
public class Order {

    @DataField(pos = 1)
    private int orderNr;

    @DataField(pos = 2)
    private Integer clientNr;

    @DataField(pos = 3, required = true)
    private String firstName;

    @DataField(pos = 4, defaultValue = "Barin")
    private String lastName;
...
}

4. FixedLengthRecord

The FixedLengthRecord annotation is used to identified the root class of the model. It represents a record = a line of a file/message containing data fixed length formatted and can be linked to several children model classes. This format is a bit particular beause data of a field can be aligned to the right or to the left.
When the size of the data does not fill completely the length of the field, we can then add 'padd' characters.

case 1 : Simple fixed length record

This simple example shows how to design the model to parse/format a fixed message

10A9PaulineMISINXD12345678BUYShare2500.45USD01-08-2009

   @FixedLengthRecord(length=54, paddingChar=' ')
    public static class Order {

        @DataField(pos = 1, length=2)
        private int orderNr;

        @DataField(pos = 3, length=2)
        private String clientNr;

        @DataField(pos = 5, length=7)
        private String firstName;

        @DataField(pos = 12, length=1, align="L")
        private String lastName;

        @DataField(pos = 13, length=4)
        private String instrumentCode;

        @DataField(pos = 17, length=10)
        private String instrumentNumber;

        @DataField(pos = 27, length=3)
        private String orderType;

        @DataField(pos = 30, length=5)
        private String instrumentType;

        @DataField(pos = 35, precision = 2, length=7)
        private BigDecimal amount;

        @DataField(pos = 42, length=3)
        private String currency;

        @DataField(pos = 45, length=10, pattern = "dd-MM-yyyy")
        private Date orderDate;
        ...

case 2 : Fixed length record with alignment and padding

This more elaborated example show how to define the alignment for a field and how to assign a padding character which is ' ' here''

10A9 PaulineM ISINXD12345678BUYShare2500.45USD01-08-2009

   @FixedLengthRecord(length=60, paddingChar=' ')
    public static class Order {

        @DataField(pos = 1, length=2)
        private int orderNr;

        @DataField(pos = 3, length=2)
        private String clientNr;

        @DataField(pos = 5, length=9)
        private String firstName;

        @DataField(pos = 14, length=5, align="L")   // align text to the LEFT zone of the block
        private String lastName;

        @DataField(pos = 19, length=4)
        private String instrumentCode;

        @DataField(pos = 23, length=10)
        private String instrumentNumber;

        @DataField(pos = 33, length=3)
        private String orderType;

        @DataField(pos = 36, length=5)
        private String instrumentType;

        @DataField(pos = 41, precision = 2, length=7)
        private BigDecimal amount;

        @DataField(pos = 48, length=3)
        private String currency;

        @DataField(pos = 51, length=10, pattern = "dd-MM-yyyy")
        private Date orderDate;
        ...

case 3 : Field padding

Sometimes, the default padding defined for record cannnot be applied to the field as we have a number format where we would like to padd with '0' instead of ' '. In this case, you can use in the model the attribute paddingField to set this value.

10A9 PaulineM ISINXD12345678BUYShare000002500.45USD01-08-2009

    @FixedLengthRecord(length = 65, paddingChar = ' ')
    public static class Order {

        @DataField(pos = 1, length = 2)
        private int orderNr;

        @DataField(pos = 3, length = 2)
        private String clientNr;

        @DataField(pos = 5, length = 9)
        private String firstName;

        @DataField(pos = 14, length = 5, align = "L")
        private String lastName;

        @DataField(pos = 19, length = 4)
        private String instrumentCode;

        @DataField(pos = 23, length = 10)
        private String instrumentNumber;

        @DataField(pos = 33, length = 3)
        private String orderType;

        @DataField(pos = 36, length = 5)
        private String instrumentType;

        @DataField(pos = 41, precision = 2, length = 12, paddingChar = '0')
        private BigDecimal amount;

        @DataField(pos = 53, length = 3)
        private String currency;

        @DataField(pos = 56, length = 10, pattern = "dd-MM-yyyy")
        private Date orderDate;
        ...

case 4: Fixed length record with delimiter

Fixed-length records sometimes have delimited content within the record. The firstName and lastName fields are delimited with the '^' character in the following example:

10A9Pauline^M^ISINXD12345678BUYShare000002500.45USD01-08-2009

    @FixedLengthRecord()
    public static class Order {

        @DataField(pos = 1, length = 2)
        private int orderNr;

        @DataField(pos = 2, length = 2)
        private String clientNr;

        @DataField(pos = 3, delimiter = "^")
        private String firstName;

        @DataField(pos = 4, delimiter = "^")
        private String lastName;

        @DataField(pos = 5, length = 4)
        private String instrumentCode;

        @DataField(pos = 6, length = 10)
        private String instrumentNumber;

        @DataField(pos = 7, length = 3)
        private String orderType;

        @DataField(pos = 8, length = 5)
        private String instrumentType;

        @DataField(pos = 9, precision = 2, length = 12, paddingChar = '0')
        private BigDecimal amount;

        @DataField(pos = 10, length = 3)
        private String currency;

        @DataField(pos = 11, length = 10, pattern = "dd-MM-yyyy")
        private Date orderDate;

case 5 : Fixed length record with record-defined field length

Occasionally a fixed-length record may contain a field that define the expected length of another field within the same record. In the following example the length of the instrumentNumber field value is defined by the value of instrumentNumberLen field in the record.

10A9Pauline^M^ISIN10XD12345678BUYShare000002500.45USD01-08-2009

    @FixedLengthRecord()
    public static class Order {

        @DataField(pos = 1, length = 2)
        private int orderNr;

        @DataField(pos = 2, length = 2)
        private String clientNr;

        @DataField(pos = 3, delimiter = "^")
        private String firstName;

        @DataField(pos = 4, delimiter = "^")
        private String lastName;

        @DataField(pos = 5, length = 4)
        private String instrumentCode;

        @DataField(pos = 6, length = 2, align = "R", paddingChar = '0')
        private int instrumentNumberLen;
        
        @DataField(pos = 7, lengthPos=6)
        private String instrumentNumber;

        @DataField(pos = 8, length = 3)
        private String orderType;

        @DataField(pos = 9, length = 5)
        private String instrumentType;

        @DataField(pos = 10, precision = 2, length = 12, paddingChar = '0')
        private BigDecimal amount;

        @DataField(pos = 11, length = 3)
        private String currency;

        @DataField(pos = 12, length = 10, pattern = "dd-MM-yyyy")
        private Date orderDate;

case 6 : Fixed length record with header and footer

Bindy will discover fixed-length header and footer records that are configured as part of the model – provided that the annotated classes exist either in the same package as the primary @FixedLengthRecord class, or within one of the configured scan packages. The following text illustrates two fixed-length records that are bracketed by a header record and footer record.

101-08-2009
10A9 PaulineM ISINXD12345678BUYShare000002500.45USD01-08-2009
10A9 RichN ISINXD12345678BUYShare000002700.45USD01-08-2009
9000000002

@FixedLengthRecord(hasHeader = true, hasFooter = true)
public class Order {

    @DataField(pos = 1, length = 2)
    private int orderNr;

    @DataField(pos = 2, length = 2)
    private String clientNr;

    @DataField(pos = 3, length = 9)
    private String firstName;

    @DataField(pos = 4, length = 5, align = "L")
    private String lastName;

    @DataField(pos = 5, length = 4)
    private String instrumentCode;

    @DataField(pos = 6, length = 10)
    private String instrumentNumber;

    @DataField(pos = 7, length = 3)
    private String orderType;

    @DataField(pos = 8, length = 5)
    private String instrumentType;

    @DataField(pos = 9, precision = 2, length = 12, paddingChar = '0')
    private BigDecimal amount;

    @DataField(pos = 10, length = 3)
    private String currency;

    @DataField(pos = 11, length = 10, pattern = "dd-MM-yyyy")
    private Date orderDate;
...
}


@FixedLengthRecord(isHeader = true)
public  class OrderHeader {
    @DataField(pos = 1, length = 1)
    private int recordType = 1;
    
    @DataField(pos = 2, length = 10, pattern = "dd-MM-yyyy")
    private Date recordDate;
    
...
}


@FixedLengthRecord(isFooter = true)
public class OrderFooter {
    
    @DataField(pos = 1, length = 1)
    private int recordType = 9;
    
    @DataField(pos = 2, length = 9, align = "R", paddingChar = '0')
    private int numberOfRecordsInTheFile;

...
}

case 7 : Skipping content when parsing a fixed length record. (Camel 2.11.1)

It is common to integrate with systems that provide fixed-length records containing more information than needed for the target use case. It is useful in this situation to skip the declaration and parsing of those fields that we do not need. To accomodate this, Bindy will skip forward to the next mapped field within a record if the 'pos' value of the next declared field is beyond the cursor position of the last parsed field. Using absolute 'pos' locations for the fields of interest (instead of ordinal values) causes Bindy to skip content between two fields.

Similarly, it is possible that none of the content beyond some field is of interest. In this case, you can tell Bindy to skip parsing of everything beyond the last mapped field by setting the ignoreTrailingChars property on the @FixedLengthRecord declaration.

@FixedLengthRecord(ignoreTrailingChars = true)
public static class Order {

        @DataField(pos = 1, length = 2)
        private int orderNr;

        @DataField(pos = 3, length = 2)
        private String clientNr;

    ... any characters that appear beyond the last mapped field will be ignored

}

5. Message

The Message annotation is used to identified the class of your model who will contain key value pairs fields. This kind of format is used mainly in Financial Exchange Protocol Messages (FIX). Nevertheless, this annotation can be used for any other format where data are identified by keys. The key pair values are separated each other by a separator which can be a special character like a tab delimitor (unicode representation : \u0009) or a start of heading (unicode representation : \u0001)

case 1 : separator = 'u0001'

The separator used to segregate the key value pair fields in a FIX message is the ASCII '01' character or in unicode format '\u0001'. This character must be escaped a second time to avoid a java runtime error. Here is an example :

8=FIX.4.1 9=20 34=1 35=0 49=INVMGR 56=BRKR 1=BE.CHM.001 11=CHM0001-01 22=4 ...

and how to use the annotation

@Message(keyValuePairSeparator = "=", pairSeparator = "\u0001", type="FIX", version="4.1")
public class Order {
...
}

6. KeyValuePairField

The KeyValuePairField annotation defines the property of a key value pair field. Each KeyValuePairField is identified by a tag (= key) and its value associated, a type (string, int, date, ...), optionaly a pattern and if the field is required

case 1 : tag

This parameter represents the key of the field in the message

@Message(keyValuePairSeparator = "=", pairSeparator = "\u0001", type="FIX", version="4.1")
public class Order {

    @Link Header header;

    @Link Trailer trailer;

    @KeyValuePairField(tag = 1) // Client reference
    private String Account;

    @KeyValuePairField(tag = 11) // Order reference
    private String ClOrdId;

    @KeyValuePairField(tag = 22) // Fund ID type (Sedol, ISIN, ...)
    private String IDSource;

    @KeyValuePairField(tag = 48) // Fund code
    private String SecurityId;

    @KeyValuePairField(tag = 54) // Movement type ( 1 = Buy, 2 = sell)
    private String Side;

    @KeyValuePairField(tag = 58) // Free text
    private String Text;

...
}

case 2 : Different position in output

If the tags/keys that we will put in the FIX message must be sorted according to a predefine order, then use the attribute 'position' of the annotation @KeyValuePairField

@Message(keyValuePairSeparator = "=", pairSeparator = "\\u0001", type = "FIX", version = "4.1", isOrdered = true)
public class Order {

    @Link Header header;

    @Link Trailer trailer;

    @KeyValuePairField(tag = 1, position = 1) // Client reference
    private String account;

    @KeyValuePairField(tag = 11, position = 3) // Order reference
    private String clOrdId;

...
}

7. Section

In FIX message of fixed length records, it is common to have different sections in the representation of the information : header, body and section. The purpose of the annotation @Section is to inform bindy about which class of the model represents the header (= section 1), body (= section 2) and footer (= section 3)

Only one attribute/parameter exists for this annotation.

case 1 : Section

A. Definition of the header section

@Section(number = 1)
public class Header {

    @KeyValuePairField(tag = 8, position = 1) // Message Header
    private String beginString;

    @KeyValuePairField(tag = 9, position = 2) // Checksum
    private int bodyLength;
...
}

B. Definition of the body section

@Section(number = 2)
@Message(keyValuePairSeparator = "=", pairSeparator = "\\u0001", type = "FIX", version = "4.1", isOrdered = true)
public class Order {

    @Link Header header;

    @Link Trailer trailer;

    @KeyValuePairField(tag = 1, position = 1) // Client reference
    private String account;

    @KeyValuePairField(tag = 11, position = 3) // Order reference
    private String clOrdId;

C. Definition of the footer section

@Section(number = 3)
public class Trailer {

    @KeyValuePairField(tag = 10, position = 1)
    // CheckSum
    private int checkSum;

    public int getCheckSum() {
        return checkSum;
    }

8. OneToMany

The purpose of the annotation @OneToMany is to allow to work with a List<?> field defined a POJO class or from a record containing repetitive groups.

The relation OneToMany ONLY WORKS in the following cases :

• Reading a FIX message containing repetitive groups (= group of tags/keys)
• Generating a CSV with repetitive data

case 1 : Generating CSV with repetitive data

Here is the CSV output that we want :

Claus,Ibsen,Camel in Action 1,2010,35
Claus,Ibsen,Camel in Action 2,2012,35
Claus,Ibsen,Camel in Action 3,2013,35
Claus,Ibsen,Camel in Action 4,2014,35

Remark : the repetitive data concern the title of the book and its publication date while first, last name and age are common

and the classes used to modeling this. The Author class contains a List of Book.

@CsvRecord(separator=",")
public class Author {

	@DataField(pos = 1)
	private String firstName;

	@DataField(pos = 2)
	private String lastName;

	@OneToMany
	private List<Book> books;

	@DataField(pos = 5)
	private String Age;
...


public class Book {

	@DataField(pos = 3)
	private String title;

	@DataField(pos = 4)
	private String year;

Very simple isn't it !!!

case 2 : Reading FIX message containing group of tags/keys

Here is the message that we would like to process in our model :

"8=FIX 4.19=2034=135=049=INVMGR56=BRKR"
"1=BE.CHM.00111=CHM0001-0158=this is a camel - bindy test"
"22=448=BE000124567854=1"
"22=548=BE000987654354=2"
"22=648=BE000999999954=3"
"10=220"

tags 22, 48 and 54 are repeated

and the code

public class Order {

    @Link Header header;

    @Link Trailer trailer;

    @KeyValuePairField(tag = 1) // Client reference
    private String account;

    @KeyValuePairField(tag = 11) // Order reference
    private String clOrdId;

    @KeyValuePairField(tag = 58) // Free text
    private String text;

    @OneToMany(mappedTo = "org.apache.camel.dataformat.bindy.model.fix.complex.onetomany.Security")
    List<Security> securities;
...

public class Security {

    @KeyValuePairField(tag = 22) // Fund ID type (Sedol, ISIN, ...)
    private String idSource;

    @KeyValuePairField(tag = 48) // Fund code
    private String securityCode;

    @KeyValuePairField(tag = 54) // Movement type ( 1 = Buy, 2 = sell)
    private String side;

Using the Java DSL

The next step consists in instantiating the DataFormat bindy class associated with this record type and providing Java package name(s) as parameter.

For example the following uses the class BindyCsvDataFormat (who correspond to the class associated with the CSV record type) which is configured with "com.acme.model"
package name to initialize the model objects configured in this package.

// Camel 2.15 or older (configure by package name)
DataFormat bindy = new BindyCsvDataFormat("com.acme.model");

 
// Camel 2.16 onwards (configure by class name)
DataFormat bindy = new BindyCsvDataFormat(com.acme.model.MyModel.class);

Setting locale

Bindy supports configuring the locale on the dataformat, such as 

// Camel 2.15 or older (configure by package name)
BindyCsvDataFormat bindy = new BindyCsvDataFormat("com.acme.model");
// Camel 2.16 onwards (configure by class name)
BindyCsvDataFormat bindy = new BindyCsvDataFormat(com.acme.model.MyModel.class);

bindy.setLocale("us");

Or to use the platform default locale then use "default" as the locale name. Notice this requires Camel 2.14/2.13.3/2.12.5.

// Camel 2.15 or older (configure by package name)
BindyCsvDataFormat bindy = new BindyCsvDataFormat("com.acme.model");
// Camel 2.16 onwards (configure by class name)
BindyCsvDataFormat bindy = new BindyCsvDataFormat(com.acme.model.MyModel.class);

bindy.setLocale("default");

for older releases you can set it using Java code as shown

// Camel 2.15 or older (configure by package name)
BindyCsvDataFormat bindy = new BindyCsvDataFormat("com.acme.model");
// Camel 2.16 onwards (configure by class name)
BindyCsvDataFormat bindy = new BindyCsvDataFormat(com.acme.model.MyModel.class);


bindy.setLocale(Locale.getDefault().getISO3Country());

Unmarshaling

from("file://inbox")
  .unmarshal(bindy)
  .to("direct:handleOrders");

Alternatively, you can use a named reference to a data format which can then be defined in your Registry e.g. your Spring XML file:

from("file://inbox")
  .unmarshal("myBindyDataFormat")
  .to("direct:handleOrders");

The Camel route will pick-up files in the inbox directory, unmarshall CSV records into a collection of model objects and send the collection
to the route referenced by 'handleOrders'.

The collection returned is a List of Map objects. Each Map within the list contains the model objects that were marshalled out of each line of the CSV. The reason behind this is that each line can correspond to more than one object. This can be confusing when you simply expect one object to be returned per line.

Each object can be retrieve using its class name.

    List<Map<String, Object>> unmarshaledModels = (List<Map<String, Object>>) exchange.getIn().getBody();

    int modelCount = 0;
    for (Map<String, Object> model : unmarshaledModels) {
	  for (String className : model.keySet()) {
	     Object obj = model.get(className);
	     LOG.info("Count : " + modelCount + ", " + obj.toString());
	  }
	 modelCount++;
    }

    LOG.info("Total CSV records received by the csv bean : " + modelCount);

Assuming that you want to extract a single Order object from this map for processing in a route, you could use a combination of a Splitter and a Processor as per the following:

from("file://inbox")
    .unmarshal(bindy)
    .split(body())
        .process(new Processor() {
            public void process(Exchange exchange) throws Exception {
                Message in = exchange.getIn();
                Map<String, Object> modelMap = (Map<String, Object>) in.getBody();
                in.setBody(modelMap.get(Order.class.getCanonicalName()));
            }
        })
        .to("direct:handleSingleOrder")
    .end();

Take care of the fact that Bindy uses CHARSET_NAME property or the CHARSET_NAME header as define in the Exchange interface to do a characterset conversion of the inputstream received for unmarshalling. In some producers (e.g. file-endpoint) you can define a characterset. The characterset conversion can already been done by this producer. Sometimes you need to remove this property or header from the exchange before sending it to the unmarshal. If you don't remove it the conversion might be done twice which might lead to unwanted results.

from("file://inbox?charset=Cp922")
  .removeProperty(Exchange.CHARSET_NAME)
  .unmarshal("myBindyDataFormat")
  .to("direct:handleOrders");

Marshaling

To generate CSV records from a collection of model objects, you create the following route :

from("direct:handleOrders")
   .marshal(bindy)
   .to("file://outbox")

Using Spring XML

This is really easy to use Spring as your favorite DSL language to declare the routes to be used for camel-bindy. The following example shows two routes where the first will pick-up records from files, unmarshal the content and bind it to their model. The result is then send to a pojo (doing nothing special) and place them into a queue.

The second route will extract the pojos from the queue and marshal the content to generate a file containing the csv record. The example above is for using Camel 2.16 onwards.

<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="
       http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd
       http://camel.apache.org/schema/spring
       http://camel.apache.org/schema/spring/camel-spring.xsd">

        <!-- Queuing engine - ActiveMq - work locally in mode virtual memory -->
	<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
		<property name="brokerURL" value="vm://localhost:61616"/>
	</bean>


	<camelContext xmlns="http://camel.apache.org/schema/spring">

 
        <dataFormats>
          <bindy id="bindyDataformat" type="Csv" classType="org.apache.camel.bindy.model.Order"/>
        </dataFormats>

		<route>
			<from uri="file://src/data/csv/?noop=true" />
			<unmarshal ref="bindyDataformat" />
			<to uri="bean:csv" />
			<to uri="activemq:queue:in" />
		</route>

		<route>
			<from uri="activemq:queue:in" />
			<marshal ref="bindyDataformat" />
			<to uri="file://src/data/csv/out/" />
		</route>
	</camelContext>
</beans>

Dependencies

To use Bindy in your camel routes you need to add the a dependency on camel-bindy which implements this data format.

If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions).

<dependency>
  <groupId>org.apache.camel</groupId>
  <artifactId>camel-bindy</artifactId>
  <version>x.x.x</version>
</dependency>

XMLSecurity Data Format

The XMLSecurity Data Format facilitates encryption and decryption of XML payloads at the Document, Element, and Element Content levels (including simultaneous multi-node encryption/decryption using XPath). To sign messages using the XML Signature specification, please see the Camel XML Security component.

The encryption capability is based on formats supported using the Apache XML Security (Santuario) project. Symmetric encryption/decryption is currently supported using Triple-DES and AES (128, 192, and 256) encryption formats. Additional formats can be easily added later as needed. This capability allows Camel users to encrypt/decrypt payloads while being dispatched or received along a route.

Available as of Camel 2.9
The XMLSecurity Data Format supports asymmetric key encryption. In this encryption model a symmetric key is generated and used to perform XML content encryption or decryption. This "content encryption key" is then itself encrypted using an asymmetric encryption algorithm that leverages the recipient's public key as the "key encryption key". Use of an asymmetric key encryption algorithm ensures that only the holder of the recipient's private key can access the generated symmetric encryption key. Thus, only the private key holder can decode the message. The XMLSecurity Data Format handles all of the logic required to encrypt and decrypt the message content and encryption key(s) using asymmetric key encryption.

The XMLSecurity Data Format also has improved support for namespaces when processing the XPath queries that select content for encryption. A namespace definition mapping can be included as part of the data format configuration. This enables true namespace matching, even if the prefix values in the XPath query and the target xml document are not equivalent strings.

Basic Options

Asymmetric Encryption Options

These options can be applied in addition to relevant the Basic options to use asymmetric key encryption.

Key Cipher Algorithm

As of Camel 2.12.0, the default Key Cipher Algorithm is now XMLCipher.RSA_OAEP instead of XMLCipher.RSA_v1dot5. Usage of XMLCipher.RSA_v1dot5 is discouraged due to various attacks. Requests that use RSA v1.5 as the key cipher algorithm will be rejected unless it has been explicitly configured as the key cipher algorithm.

Marshal

In order to encrypt the payload, the marshal processor needs to be applied on the route followed by the secureXML() tag.

Unmarshal

In order to decrypt the payload, the unmarshal processor needs to be applied on the route followed by the secureXML() tag.

Examples

Given below are several examples of how marshalling could be performed at the Document, Element, and Content levels.

Full Payload encryption/decryption

from("direct:start")
    .marshal().secureXML()
    .unmarshal().secureXML()
    .to("direct:end");

Partial Payload Content Only encryption/decryption

String tagXPATH = "//cheesesites/italy/cheese";
boolean secureTagContent = true;
...
from("direct:start")
    .marshal().secureXML(tagXPATH, secureTagContent)
    .unmarshal().secureXML(tagXPATH, secureTagContent)
    .to("direct:end");

Partial Multi Node Payload Content Only encryption/decryption

String tagXPATH = "//cheesesites/*/cheese";
boolean secureTagContent = true;
...
from("direct:start")
    .marshal().secureXML(tagXPATH, secureTagContent)
    .unmarshal().secureXML(tagXPATH, secureTagContent)
    .to("direct:end");

Partial Payload Content Only encryption/decryption with choice of passPhrase(password)

String tagXPATH = "//cheesesites/italy/cheese";
boolean secureTagContent = true;
...
String passPhrase = "Just another 24 Byte key";
from("direct:start")
    .marshal().secureXML(tagXPATH, secureTagContent, passPhrase)
    .unmarshal().secureXML(tagXPATH, secureTagContent, passPhrase)
    .to("direct:end");

Partial Payload Content Only encryption/decryption with passPhrase(password) and Algorithm

import org.apache.xml.security.encryption.XMLCipher;
....
String tagXPATH = "//cheesesites/italy/cheese";
boolean secureTagContent = true;
String passPhrase = "Just another 24 Byte key";
String algorithm= XMLCipher.TRIPLEDES;
from("direct:start")
    .marshal().secureXML(tagXPATH, secureTagContent, passPhrase, algorithm)
    .unmarshal().secureXML(tagXPATH, secureTagContent, passPhrase, algorithm)
    .to("direct:end");

Partial Payload Content with Namespace support

Java DSL

final Map<String, String> namespaces = new HashMap<String, String>();
namespaces.put("cust", "http://cheese.xmlsecurity.camel.apache.org/");

final KeyStoreParameters tsParameters = new KeyStoreParameters();
tsParameters.setPassword("password");
tsParameters.setResource("sender.ts");

context.addRoutes(new RouteBuilder() {
    public void configure() {
        from("direct:start")
           .marshal().secureXML("//cust:cheesesites/italy", namespaces, true, "recipient",
                                testCypherAlgorithm, XMLCipher.RSA_v1dot5, tsParameters)
           .to("mock:encrypted");
    }
}

Spring XML

A namespace prefix that is defined as part of the camelContext definition can be re-used in context within the data format secureTag attribute of the secureXML element.

<camelContext id="springXmlSecurityDataFormatTestCamelContext" 
              xmlns="http://camel.apache.org/schema/spring"
              xmlns:cheese="http://cheese.xmlsecurity.camel.apache.org/">        
    <route>
        <from uri="direct://start"/>
            <marshal>
                <secureXML secureTag="//cheese:cheesesites/italy"
                           secureTagContents="true"/>
            </marshal> 
            ...

Asymmetric Key Encryption

Spring XML Sender

<!--  trust store configuration -->                          
<camel:keyStoreParameters id="trustStoreParams" resource="./sender.ts" password="password"/>

<camelContext id="springXmlSecurityDataFormatTestCamelContext" 
              xmlns="http://camel.apache.org/schema/spring"
              xmlns:cheese="http://cheese.xmlsecurity.camel.apache.org/">        
    <route>
        <from uri="direct://start"/>
            <marshal>
                <secureXML secureTag="//cheese:cheesesites/italy"
                           secureTagContents="true"
                           xmlCipherAlgorithm="http://www.w3.org/2001/04/xmlenc#aes128-cbc"       
                           keyCipherAlgorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5"
                           recipientKeyAlias="recipient"
                           keyOrTrustStoreParametersId="trustStoreParams"/>
            </marshal> 
            ...

Spring XML Recipient

 
<!--  key store configuration -->
<camel:keyStoreParameters id="keyStoreParams" resource="./recipient.ks" password="password" />

<camelContext id="springXmlSecurityDataFormatTestCamelContext" 
              xmlns="http://camel.apache.org/schema/spring"
              xmlns:cheese="http://cheese.xmlsecurity.camel.apache.org/">
    <route>    
        <from uri="direct://encrypted"/>
            <unmarshal>
                <secureXML secureTag="//cheese:cheesesites/italy"
                           secureTagContents="true"
                           xmlCipherAlgorithm="http://www.w3.org/2001/04/xmlenc#aes128-cbc"
                           keyCipherAlgorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5"
                           recipientKeyAlias="recipient"
                           keyOrTrustStoreParametersId="keyStoreParams"
                           keyPassword="privateKeyPassword" />
            </unmarshal>
            ...

Dependencies

This data format is provided within the camel-xmlsecurity component.

Protobuf - Protocol Buffers

"Protocol Buffers - Google's data interchange format"

Camel provides a Data Format to serialse between Java and the Protocol Buffer protocol. The project's site details why you may wish to choose this format over xml. Protocol Buffer is language-neutral and platform-neutral, so messages produced by your Camel routes may be consumed by other language implementations.

API Site
Protobuf Implementation
Protobuf Java Tutorial

Protobuf overview

This quick overview of how to use Protobuf. For more detail see the complete tutorial

Defining the proto format

The first step is to define the format for the body of your exchange. This is defined in a .proto file as so:

package org.apache.camel.component.protobuf;

option java_package = "org.apache.camel.component.protobuf";
option java_outer_classname = "AddressBookProtos";

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    required string number = 1;
    optional PhoneType type = 2 [default = HOME];
  }

  repeated PhoneNumber phone = 4;
}

message AddressBook {
  repeated Person person = 1;
}

Generating Java classes

The Protobuf SDK provides a compiler which will generate the Java classes for the format we defined in our .proto file. You can run the compiler for any additional supported languages you require.

protoc --java_out=. ./addressbook.proto

This will generate a single Java class named AddressBookProtos which contains inner classes for Person and AddressBook. Builders are also implemented for you. The generated classes implement com.google.protobuf.Message which is required by the serialisation mechanism. For this reason it important that only these classes are used in the body of your exchanges. Camel will throw an exception on route creation if you attempt to tell the Data Format to use a class that does not implement com.google.protobuf.Message. Use the generated builders to translate the data from any of your existing domain classes.

Java DSL

You can use create the ProtobufDataFormat instance and pass it to Camel DataFormat marshal and unmarsha API like this.

   ProtobufDataFormat format = new ProtobufDataFormat(Person.getDefaultInstance());

   from("direct:in").marshal(format);
   from("direct:back").unmarshal(format).to("mock:reverse");

Or use the DSL protobuf() passing the unmarshal default instance or default instance class name like this.

   // You don't need to specify the default instance for protobuf marshaling               
   from("direct:marshal").marshal().protobuf();
   from("direct:unmarshalA").unmarshal().
       protobuf("org.apache.camel.dataformat.protobuf.generated.AddressBookProtos$Person").
       to ("mock:reverse");
                
   from("direct:unmarshalB").unmarshal().protobuf(Person.getDefaultInstance()).to("mock:reverse");

Spring DSL

The following example shows how to use Castor to unmarshal using Spring configuring the protobuf data type

<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
  <route>
    <from uri="direct:start"/>
    <unmarshal>
      <protobuf instanceClass="org.apache.camel.dataformat.protobuf.generated.AddressBookProtos$Person" />
    </unmarshal>
    <to uri="mock:result"/>
  </route>
</camelContext>

Dependencies

To use Protobuf in your camel routes you need to add the a dependency on camel-protobuf which implements this data format.

If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions).

<dependency>
  <groupId>org.apache.camel</groupId>
  <artifactId>camel-protobuf</artifactId>
  <version>2.2.0</version>
</dependency>