Page History

...

spark:{rdd|dataframe|hive}

RDD jobs 

...

@Bean
AbstractJavaRDDLike myRdd(JavaSparkContext sparkContext) {
  return sparkContext.textFile("testrdd.txt");
}

Deploying KuraRouter

Bundle containing your Kura router class should import the following packages in the OSGi manifest:

...

RDD jobs options

...

...

...

Keep in mind that you don't have to import every Camel component bundle you plan to use in your routes, as Camel components are resolved as the services on the runtime level.

Before you deploy your router bundle, be sure that you have deployed (and started) the following Camel core bundles (using Kura GoGo shell)...

...

install file:///home/user/.m2/repository/org/apache/camel/camel-core/2.15.0/camel-core-2.15.0.jar
start <camel-core-bundle-id>
install file:///home/user/.m2/repository/org/apache/camel/camel-core-osgi/2.15.0/camel-core-osgi-2.15.0.jar
start <camel-core-osgi-bundle-id>
install file:///home/user/.m2/repository/org/apache/camel/camel-kura/2.15.0/camel-kura-2.15.0.jar 
start <camel-kura-bundle-id>

Void RDD callbacks

If your RDD callback doesn't return any value back to a Camel pipeline, you can either return null value or use VoidRddCallback base class:

@Bean
RddCallback<Void> rddCallback() {
  return new VoidRddCallback() {
        @Override
    

...and all the components you plan to use in your routes:

...

install file:///home/user/.m2/repository/org/apache/camel/camel-stream/2.15.0/camel-stream-2.15.0.jar
start <camel-stream-bundle-id>

Then finally deploy your router bundle:

...

install file:///home/user/.m2/repository/com/example/myrouter/1.0/myrouter-1.0.jar
start <your-bundle-id>

KuraRouter utilities 

 Kura router base class provides many useful utilities. This section explores each of them.

SLF4J logger

Kura uses SLF4J facade for logging purposes. Protected member log returns SLF4J logger instance associated with the given Kura router.

public class MyKuraRouter extends KuraRouter {

    @Override
    public void configure() throws Exception {
		log.info("Configuring Camel routes!"doOnRdd(AbstractJavaRDDLike rdd, Object... payloads) {
            rdd.saveAsTextFile(output.getAbsolutePath());
        ...}
    };

}

CamelContext

Void RDD callbacks

If you know what type of the input data will be sent to the RDD callback, you can use ConvertingRddCallback and let Camel to automatically convert incoming messages before inserting those into the callback:Protected member camelContext is the CamelContext associated with the given Kura router.

@Bean
RddCallback<Void> rddCallback(CamelContext context) {
  return new ConvertingRddCallback<Long>(context, int.class, int.class) {
        public class MyKuraRouter extends KuraRouter {

    @Override
            public voidLong configure() throws Exception {
		camelContext.getStatus()doOnRdd(AbstractJavaRDDLike rdd, Object... payloads) {
                return rdd.count() * (int) payloads[0] * (int) payloads[1];
        ...
    }
        };
    };
}

ProducerTemplate

Protected member producerTemplate is the ProducerTemplate instance associated with the given Camel context.

...