Page History

Camel 3.0 Ideas

Camel is now almost 6 years old and its second revision camel-2.x is more than 4.5 years old already. Camel is extremely mature, used in production by a large number of organizations from small to large and even governments. We feel like we really hit the initial target of simplifying integration. Camel's middleware abstraction api and the eip based routing brought a lot of positive feedback from users.

...

JDK support

(+1: claus, cmueller, mattrpav)
(-1: hadrian)

We should drop support for JDK6, and require JDK7 as minimim version. eg build and compile the release with JDK7.
We should aim to be compatible with JDK8.
(hz) Why? Isn't OpenJDK 6 still supported by RedHat and very much in use? Even Oracle's Java 6 is still in use. Maybe some bundles would require JDK7+, buy why not keep things like the api for instance still supported with Java 6? A strong argument may sway my opinion, but for now, I fail to see the reason.

(mattrpav) Consider JDK8 as the base. By the time Camel 3 is stable and ready for wide-spread use, the transitive deps will have caught up. A number of suggestions below target JDK8 features, and all would require an "add-on" approach to maintain JDK7 backward compat. 

JDK8 Java DSL

It would be good to have a camel-java8-dsl component that offers a JDK8 DSL which uses all the nice new stuff from JDK8 with higher order functions, closures, et all.
Though this may comes later in Camel 3.x when JDK8 is GA.
At least stuff like Predicate, Expression, AggregationStrategy etc. are "functional interfaces" (containing only one method) and Java 8 applications can implement them using lambdas. That's only a start, but it doesn't require a specific DSL.

Clearer Architecture of Camel Core

Goals:

• The camel components should know as little as possible about camel core
• The classes needed to setup camel should be separate from the things needed at run time
• Camel Core should be tiny as possible and only contain what really is core

So why should this be important? Currently components depend on camel-core as a whole and there are no further rules which classes the components should use and which classes should be private to core. Even classes from the impl package are needed. So this means that any refactoring we do in camel core could affect all components. As camel is growing steadily this can become quite problematic.

Split camel-core into multiple parts (hadrian)

Routing Core Re-engineering (raulk)

The routing core of Camel 2.x is heavily based on a recursive call pattern, where Processors are responsible for calling the next one along the chain. This results in lengthy and meaningless stacktraces (difficult to make sense out of and debug for newcomers) and higher memory usage due to retention of local variables for a longer time than strictly needed.

Moreover, Camel weaves a large number of "plumbing" processors along the way which should not really be processors because they form part of the very essence of the routing core, e.g. error handlers, stream caching interceptors, trace interceptors, async processor helpers, MDC, etc.

The proposal is to shift towards an iterative model, by redesigning the logic of Camel routing. The suggested model is defined by these pillars:

• A single class, or a limited set of them, contain the routing logic of Camel. Package name: org.apache.camel.core.routing. Central (abstract) class: RoutingCore. Concrete realisations could be: PipeliningRoutingCore, MulticastRoutingCore, depending on the fundamental routing pattern.
• The RoutingCore iteratively calls the routing steps, one after another. The routing steps return their result to the RoutingCore, who is in charge of calling the next element subsequently. OUT and IN are bridged if necessary (PipeliningRoutingCore).
• The Processor interface is crumbled up into its many specialisations, each of which represents a distinct concept of the Camel framework: RoutingDecider (EIPs should only take decisions about the routing, but not perform the routing itself, e.g. choice, filter, loop, throttle, etc.; see examples in subsection below.), Actions, ErrorHandler (already exists), Interceptor, etc.
• The RoutingCore is responsible of all the "magic" now disseminated across a number of processors. Assisted by Helper classes.

The goal of this idea isn't to zap off recursion altogether, just to consolidate the routing logic into a handful of cornerstone classes.

Camel is no longer a baby and the framework concepts are well mature, thus they should be transferred to the API and avoid making everything a raw Processor.

Converting some EIPs from "performers" to mere "deciders"

• choice() => evaluates the predicates and returns the List of Processors or Endpoints to invoke.
• filter() => same as choice(), but returning null if the filter doesn't match, to continue to the next routing step.
• loop() => evaluates whether the looping control predicate still stands. If yes, it returns the processors to invoke, where the last is itself (to trigger the looping logic again); else, it returns null to continue to the next routing step.
• throttle() => pauses accordingly and then returns the endpoint/processors to invoke.
• ...

Clearer Architecture of Camel Core

Goals:

• The camel components should know as little as possible about camel core
• The classes needed to setup camel should be separate from the things needed at run time
• Camel Core should be tiny as possible and only contain what really is core

So why should this be important? Currently components depend on camel-core as a whole and there are no further rules which classes the components should use and which classes should be private to core. Even classes from the impl package are needed. So this means that any refactoring we do in camel core could affect all components. As camel is growing steadily this can become quite problematic.

Split camel-core into multiple parts (hadrian)

(+1: cmueller, hadrian(+1: cmueller, hadrian, claus)

Claus: Important to be 99+% backwards compatible with Camel 2.x.

...

We should consider removing

• camel-bam
• camel-msv
• org.apache.camel.view from came-core

• dot maven generator

• ... (there could be other stuff to remove)

The BAM has not changed in 5 years, and very seldom used by end users. And neither has the functionality you need. There is much better solutions outside ASF Camel for a BAM solution.
The DOT generator is not up to date and maintained. Also it requires binary generator to generate a route diagram; we never managed to find a good java library for that.

...

We could add support for using HawtDB as the persistent store for streams which overflow to disk store.
This might be implemented with the message store when it is used for stream caching.

EIP

The Resequencer EIP currently doesn't support persistence, we could introduce this and let it leverage HawtDB such as we did with the Aggregator2 EIPwith the Aggregator2 EIP.
This might be implemented with the message store when it is used for temporarily saving exchanes until they are in order.

Schedule in DSL

We could consider adding DSL syntax sugar for scheduling routes. For example currently you have to use Quartz or a ScheduledPollingConsumer which has the delay option. We could add DSL which has something like:

schedule().every(5).minute().pollFrom("xxx").to("yyyy")

...

Unified statistics

-1: claus, mattrpav. We keep the JMX API and ppl can use jolokia for REST over JMX etc.

...

We already have REST support with CXFRS and Restlet but it can be better. We should make sure those components is dead easy to use and you can invoke REST services in one line of code etc. And we should make more examples and tidy up the CXFRS documentation.

More load tests (cmueller, +1 mattrpav)

More load tests for frequently used Camel components (jetty, jms ...) and camel-core. If we have an defined runtime environment which is in general accessable/available for all users (like a public image on Amazon EC2) and a data store for the performance numbers, than we could easily collect and share the numbers.

...

The implementation of DefaultUnitOfWork seems to have transformed itself into a to broad concern where unit of work is doing a bit more work than the transactional aspect that ties to its name.
Maybe this implementation should be named ExchangeContext and we can introduce a simpler UnitOfWork concept. This would also allow us to refactor the SubUnitOfWork into a general parent/child unit of work concept.
However this requires API changes and thus is best kept for Camel 3.0

This is also needed by the refactor of the Message History EIP.

Improvements to ThreadPoolProfile for thread management

We could move ThreadPoolProfile from org.apache.camel.spi to org.apache.camel and have it in the root package.

Tighten up onException

We should consider tighten up the onException DSL a bit, for example to make it more clear that if you have processing steps in there, they only occur after the message has been exhausted. So maybe we need to have a onExhausted to make this clear

3.0

This is also needed by the refactor of the Message History EIP.

Improvements to ThreadPoolProfile for thread management

We could move ThreadPoolProfile from org.apache.camel.spi to org.apache.camel and have it in the root package.

Tighten up onException

We should consider tighten up the onException DSL a bit, for example to make it more clear that if you have processing steps in there, they only occur after the message has been exhausted. So maybe we need to have a onExhausted to make this clear

onException(IOException.class).maximumRedeliveries(3)
  .onExhausted().handled(true).to("log:ignoreMe");

So in this example its only after the 3 failed redeliveries, then its exhausted, and then we handle the message and route it to a log where we ignore the message.

Also currently you can do this:

onException(IOException.class).maximumRedeliveries(3)
  .onExhaustedhandled(true);

Which will handle the message and after this let the error handler deal with the message. See CAMEL-5059.

...

We should in this case use a NoopProcessor so the message is handled, but the regular error handler does not react. Then its consistent.
However we have not changed this in Camel 2.x to keep backwards compatibility.

Likewise there has been ideas to move onRedeliverRef to <redeliveryPolicy> as currently its to be configured outside the policy.
It may make more sense to move onRedeliverRef to the policy to keep it together.

Add composite EIP to compose a number of EIPs into a single EIP

It would be nice if we have a DSL to compose a number of child EIPs into a single EIP itself. eg a bit like <pipeline>. This would allow people to make it easier to group together a number of EIPs into a single "unit". This allows for example the error handler to redeliver to the composite EIP instead of at the point of failure within the group. There are use-cases where people want to do that. And today the solution is to split this into a new route, which you then disable error handler by setting it to no error handler. And then call the route using the direct endpoint. So instead if you could do <composite> ... stuff goes here </composite> then that would be neater, as you dont have to split into multiple routes. I think there is an old JIRA ticket created about this a long time ago.

Remove Spring transaction dependency from JMS component (Removed because SJMS will be non-spring version)

(+1: cgeer)

Refactor the JMS component to remove the dependency on Spring PlatformTransactionManager and replace with JTA TransactionManager.

Default to use Transacted mode by default

One thing I regret about the design of Camel was defaulting to non-transacted mode by default and forcing users to explicitly enable transactional model with a Transaction Manager.

It might be nice in Camel 3 to assume everything is transacted by default; and mimmick a form of transaction manager for non-transactional resource.

e.g. its pretty common to work with messaging like JMS and databases as a common set of endpoints; the easiest thing that could possibly work for end users is for those to just be transacted by default without the user having to grok how to tinker with component configurations & Transaction Manager registration.

Note I'm not advocating XA here; I think the default for camel should be for Idempotent Consumer and simple approach to middleware

Add trace information to the Camel exceptions

If and exception occurs, it would be helpful if the exception contains trace information like the passed endpoints/processors and the current endpoint/processor. We may could also extend the error handler to log (or not) this information.
That make it much easier to dig into the problem when we encounter an exception.

Avoid throws Exception on end user API

Important: The org.apache.camel.Processor should stay as is. This API has been like this since the first commit, and we should let it be backwards compatible.

But there is a number of end user APIs such as Service, eg when you call start on a CamelContext. It would be better to have that not throws Exception but use runtime exceptions if failed to start etc.

An API which uses unchecked exceptions is the ProducerTemplate which end users uses. So its nice and easy to use.
Another example would be Main and MainSupport where the run method throws exception etc.
There is possible other end user APIs which we can make a bit more friendly.

So in this example its only after the 3 failed redeliveries, then its exhausted, and then we handle the message and route it to a log where we ignore the message.

Also currently you can do this:

onException(IOException.class).maximumRedeliveries(3)
  .handled(true);

Which will handle the message and after this let the error handler deal with the message. See CAMEL-5059.
We should in this case use a NoopProcessor so the message is handled, but the regular error handler does not react. Then its consistent.
However we have not changed this in Camel 2.x to keep backwards compatibility.

Likewise there has been ideas to move onRedeliverRef to <redeliveryPolicy> as currently its to be configured outside the policy.
It may make more sense to move onRedeliverRef to the policy to keep it together.

Add composite EIP to compose a number of EIPs into a single EIP

It would be nice if we have a DSL to compose a number of child EIPs into a single EIP itself. eg a bit like <pipeline>. This would allow people to make it easier to group together a number of EIPs into a single "unit". This allows for example the error handler to redeliver to the composite EIP instead of at the point of failure within the group. There are use-cases where people want to do that. And today the solution is to split this into a new route, which you then disable error handler by setting it to no error handler. And then call the route using the direct endpoint. So instead if you could do <composite> ... stuff goes here </composite> then that would be neater, as you dont have to split into multiple routes. I think there is an old JIRA ticket created about this a long time ago.

Remove Spring transaction dependency from JMS component (Removed because SJMS will be non-spring version)

(+1: cgeer)

Refactor the JMS component to remove the dependency on Spring PlatformTransactionManager and replace with JTA TransactionManager.

Default to use Transacted mode by default

One thing I regret about the design of Camel was defaulting to non-transacted mode by default and forcing users to explicitly enable transactional model with a Transaction Manager.

It might be nice in Camel 3 to assume everything is transacted by default; and mimmick a form of transaction manager for non-transactional resource.

e.g. its pretty common to work with messaging like JMS and databases as a common set of endpoints; the easiest thing that could possibly work for end users is for those to just be transacted by default without the user having to grok how to tinker with component configurations & Transaction Manager registration.

Note I'm not advocating XA here; I think the default for camel should be for Idempotent Consumer and simple approach to middleware