Tutorial-Business-Partners

Under Construction

This tutorial is a work in progress.

Background and Introduction

So there's a company, which we'll call Acme. Acme sells widgets, in a fairly unusual way. Their customers are responsible for telling Acme what they purchased. The customer enters into their own systems (ERP or whatever) which widgets they bought from Acme. Then at some point, their systems emit a record of the sale which needs to go to Acme so Acme can bill them for it. Obviously, everyone wants this to be as automated as possible, so there needs to be integration between the customer's system and Acme.

Sadly, Acme's sales people are, technically speaking, doormats. They tell all their prospects, "you can send us the data in whatever format, using whatever protocols, whatever. You just can't change once it's up and running."

The result is pretty much what you'd expect. Taking a random sample of 3 customers:

Customer 1: XML over FTP
Customer 2: CSV over HTTP
Customer 3: Excel via e-mail

Now on the Acme side, all this has to be converted to a canonical XML format and submitted to the Acme accounting system via JMS. Then the Acme accounting system does its stuff and sends an XML reply via JMS, with a summary of what it processed (e.g. 3 line items accepted, line item #2 in error, total invoice $123.45). Finally, that data needs to be formatted into an e-mail, and sent to a contact at the customer in question ("Dear Joyce, we received an invoice on 1/2/08. We accepted 3 line items totaling $123.45, though there was an error with line items #2 [invalid quantity ordered]. Thank you for your business. Love, Acme.").

So it turns out Camel can handle all this:

Listen for HTTP, e-mail, and FTP files
Grab attachments from the e-mail messages
Convert XML, XLS, and CSV files to a canonical XML format
- using XSLT, automatic data format conversion, and/or POJOs that write to JAXB objects
read and write JMS messages
route based on company ID
format e-mails using Velocity templates
send outgoing e-mail messages

This tutorial will cover all that, plus setting up tests along the way.

High-Level Diagram

Here's more or less what the integration process looks like.

First, the input from the customers to Acme:

And then, the output from Acme to the customers:

Tutorial Tasks

To get through this scenario, we're going to break it down into smaller pieces, implement and test those, and then try to assemble the big scenario and test that.

Here's what we'll try to accomplish:

Create a Maven build for the project
Get sample files for the customer Excel, CSV, and XML input
Get a sample file for the canonical XML format that Acme's accounting system uses
Create an XSD for the canonical XML format
Create JAXB POJOs corresponding to the canonical XSD
Create an XSLT stylesheet to convert the Customer 1 (XML over FTP) messages to the canonical format
Create a unit test to ensure that a simple Camel route invoking the XSLT stylesheet works
Create a POJO that converts a List<List<String>> to the above JAXB POJOs
- Note that Camel can automatically convert CSV input to a List of Lists of Strings representing the rows and columns of the CSV, so we'll use this POJO to handle Customer 2 (CSV over HTTP)
Create a unit test to ensure that a simple Camel route invoking the CSV processing works
Create a POJO that converts a Customer 3 Excel file to the above JAXB POJOs (using POI to read Excel)
Create a unit test to ensure that a simple Camel route invoking the Excel processing works
Create a POJO that reads an input message, takes an attachment off the message, and replaces the body of the message with the attachment
- This is assuming for Customer 3 (Excel over e-mail) that the e-mail contains a single Excel file as an attachment, and the actual e-mail body is throwaway
Build a set of Camel routes to handle the entire input (Customer -> Acme) side of the scenario.
Build unit tests for the Camel input.
TODO: Tasks for the output (Acme -> Customer) side of the scenario

Let's Get Started!

Step 1: Initial Maven build

We'll use Maven for this project as there will eventually be quite a few dependencies and it's nice to have Maven handle them for us. You should have a current version of Maven (e.g. 2.0.9) installed.

You can start with a pretty empty project directory and a Maven POM file, or use a simple JAR archetype to create one.

Here's a sample POM. We've added a dependency on camel-core, and set the compile version to 1.5 (so we can use annotations):

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0">
    <modelVersion>4.0.0</modelVersion>
    <groupId>org.apache.camel.tutorial</groupId>
    <artifactId>business-partners</artifactId>
    <version>1.0-SNAPSHOT</version>
    <name>Camel Business Partners Tutorial</name>
    <dependencies>
        <dependency>
            <artifactId>camel-core</artifactId>
            <groupId>org.apache.camel</groupId>
            <version>1.4.0</version>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.5</source>
                    <target>1.5</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Step 2: Get Sample Files

You can make up your own if you like, but here are the "off the shelf" ones:

Customer 1 (XML): input-customer1.xml
Customer 2 (CSV): input-customer2.csv
Customer 3 (Excel): input-customer3.xls
Canonical Acme XML Request: canonical-acme-request.xml
Canonical Acme XML Response: TODO

If you look at these files, you'll see that the different input formats use different field names and/or ordering, because of course the sales guys were totally OK with that. Sigh.

Step 3: XSD and JAXB Beans for the Canonical XML Format

Here's the sample of the canonical XML file:

<?xml version="1.0" encoding="UTF-8"?>
<invoice xmlns="http://activemq.apache.org/camel/tutorial/partners/invoice">
  <partner-id>2</partner-id>
  <date-received>9/12/2008</date-received>
  <line-item>
    <product-id>134</product-id>
    <description>A widget</description>
    <quantity>3</quantity>
    <item-price>10.45</item-price>
    <order-date>6/5/2008</order-date>
  </line-item>
  <!-- // more line-item elements here -->
  <order-total>218.82</order-total>
</invoice>

If you're ambitions, you can write your own XSD (XML Schema) for files that look like this. If not, you can download mine. In any case, save it to src/main/xsd.

Generating JAXB Beans

Down the road we'll want to deal with the XML as Java POJOs. We'll take a moment now to set up those XML binding POJOs. So we'll update the Maven POM to generate JAXB beans from the XSD file.

We need a dependency:

<dependency>
    <artifactId>camel-jaxb</artifactId>
    <groupId>org.apache.camel</groupId>
    <version>1.4.0</version>
</dependency>

And a plugin configured:

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>jaxb2-maven-plugin</artifactId>
    <executions>
        <execution>
            <goals>
                <goal>xjc</goal>
            </goals>
        </execution>
    </executions>
</plugin>

That should do it (it automatically looks for XML Schemas in src/main/xsd to generate beans for). Run mvn install and it should emit the beans into target/generated-sources/jaxb. Your IDE should see them there, though you may need to update the project to reflect the new settings in the Maven POM.

Initial Work on Customer 1 Input (XML over FTP)

To get a start on Customer 1, we'll create an XSLT template to convert the Customer 1 sample file into the canonical XML format, write a small Camel route to test it, and build that into a unit test. If we get through this, we can be pretty sure that the XSLT template is valid and can be run safely in Camel.

Create an XSLT template

Start with the Customer 1 sample input. You want to create an XSLT template to generate XML like the canonical XML sample above – an invoice element with line-item elements (one per item in the original XML document). If you're especially clever, you can populate the current date and order total elements too.

My sample XSLT template isn't that smart, but it'll get you going if you don't want to write one of your own.

Create a unit test

Here's where we get to some meaty Camel work. We need to:

Set up a unit test
That loads a Camel configuration
That has a route invoking our XSLT
Where the test sends a message to the route
And ensures that some XML comes out the end of the route

The easiest way to do this is to set up a Spring context that defines the Camel stuff, and then use a base unit test class from Spring that knows how to load a Spring context to run tests against. So, the procedure is:

Set Up a Skeletal Camel/Spring Unit Test

Add dependencies on Camel-Spring, and the Spring test JAR (which will automatically bring in JUnit 3.8.x) to your POM:

<dependency>
    <artifactId>camel-spring</artifactId>
    <groupId>org.apache.camel</groupId>
    <version>1.4.0</version>
</dependency>
<dependency>
    <artifactId>spring-test</artifactId>
    <groupId>org.springframework</groupId>
    <version>2.5.5</version>
    <scope>test</scope>
</dependency>

Create a new unit test class in src/test/java/your-package-here, perhaps called XMLInputTest.java
Make the test extend Spring's AbstractJUnit38SpringContextTests class.
Create a Spring context configuration file in src/test/resources, perhaps called XMLInputTest-context.xml
In the unit test class, use the class-level @ContextConfiguration annotation to indicate that a Spring context should be loaded
- By default, this looks for a Context configuration file called TestClassName-context.xml in a subdirectory corresponding to the package of the test class. For instance, if your test class was org.apache.camel.tutorial.XMLInputTest, it would look for org/apache/camel/tutorial/XMLInputTest-context.xml
- To override this default, use the locations attribute on the @ContextConfiguration annotation to provide specific context file locations (starting each path with a / if you don't want it to be relative to the package directory)
Add a CamelContext instance variable to the test class, with the @Autowired annotation
Add a ProducerTemplate instance variable and a setUp method that instantiates it from the CamelContext. We'll use the ProducerTemplate later to send messages to the route.
```
protected ProducerTemplate<Exchange> template;

protected void setUp() throws Exception {
    super.setUp();
    template = camelContext.createProducerTemplate();
}
```
Put in an empty test method just for the moment

Add the Spring <beans> element (including the Camel Namespace) with an empty <camelContext> element to the Spring context, like this:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
                               http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
                           http://activemq.apache.org/camel/schema/spring
                               http://activemq.apache.org/camel/schema/spring/camel-spring-1.4.0.xsd">

    <camelContext id="camel" xmlns="http://activemq.apache.org/camel/schema/spring">
    </camelContext>
</beans>

Your test class might look something like this:

src/test/java/org/apache/camel/tutorial/XMLInputTest.java
src/test/resources/XMLInputTest-context.xml (same as just above)

Test it by running mvn install and make sure there are no build errors. So far it doesn't test much; just that your project and test and source files are all organized correctly.

Flesh Out the Unit Test

So now we're going to write a Camel route that applies the XSLT to the sample Customer 1 input file, and makes sure that some XML output comes out:

Save the input-customer1.xml file to src/test/resources
Save your XSLT file (created in the previous step) to src/main/resources
Write a Camel Route, either right in the Spring XML, or using the Java DSL (in another class under src/test/java somewhere). This route should:
1. Start from the endpoint direct:start (which lets the test conveniently pass messages into the route)
2. Call the endpoint xslt:YourXSLTFile.xsl to transform the message
3. Send the result to the endpoint mock:finish (which lets the test verify the route output)
Add a test method to the unit test class that:
1. Get a reference to the Mock endpoint mock:finish using code like this:
```
MockEndpoint finish = MockEndpoint.resolve(camelContext, "mock:finish");
```
2. Set the expectedMessageCount on that endpoint to 1
3. Get a reference to the Customer 1 input file, using code like this:
```
InputStream in = XMLInputTest.class.getResourceAsStream("/input-partner1.xml");
assertNotNull(in);
```
4. Send that InputStream as a message to the direct:start endpoint, using code like this:
```
    template.sendBody("direct:start", in);
```
5. Ensure that the message made it through the route to the final endpoint, by testing all configured Mock endpoints like this:
```
MockEndpoint.assertIsSatisfied(camelContext);
```
6. If you like, inspect the final message body using some code like finish.getExchanges().get(0).getIn().getBody().
  - If you do this, you'll need to know what format that body is – String, byte array, InputStream, etc.
Run your test with mvn install and make sure the build completes successfully.

Your finished test might look something like this:

src/test/java/org/apache/camel/tutorial/XMLInputTest.java
For XML Configuration:
- src/test/resources/XMLInputTest-context.xml
For Java DSL Configuration:
- src/test/resources/XMLInputTest-context.xml
- src/test/java/org/apache/camel/tutorial/routes/XMLInputTestRoutes.java

Test Base Class

Once your test class is working, you might want to extract things like the @Autowired CamelContext, the ProducerTemplate, and the setUp method to a custom base class that you extend with your other tests.

Child pages

Background and Introduction

High-Level Diagram

Tutorial Tasks

Let's Get Started!

Step 1: Initial Maven build

Step 2: Get Sample Files

Step 3: XSD and JAXB Beans for the Canonical XML Format

Generating JAXB Beans

Initial Work on Customer 1 Input (XML over FTP)

Create an XSLT template

Create a unit test

Set Up a Skeletal Camel/Spring Unit Test

Flesh Out the Unit Test

Initial Work on Customer 2 Input (CSV over HTTP)

Create a CSV-handling POJO

Create a unit test

Initial Work on Customer 3 Input (Excel over e-mail)

Create an Excel-handling POJO

Create a unit test

Put this all together into Camel routes for the Customer Input

Create a unit test for the Customer Input Routes