...

OpenZipkin Brave is a distributed tracing implementation compatible with Twitter Zipkin backend services, written in Java. For quite a while OpenZipkin Brave offers a dedicated module to integrate with Apache CXF framework, namely brave-cxf3. However, lately the discussion had been initiated to make this integration a part of Apache CXF codebase so the CXF team is going to be responsible for maintaining it. As such, it is going to be available in upcoming since 3.2.0 release /3.1.12 releases under cxf-integration-tracing-brave module, with both client side and server side supported. This section gives a complete overview on how distributed tracing using OpenZipkin Brave (4.3.x+) could be integrated into JAX-RS / JAX-WS applications built on top of Apache CXF.

...

Under the hood spans are attached to their threads (in general, thread which created the span should close it), the same technique employed by other distributed tracing implementations. However, what is unique is that OpenZipkin Brave distinguishes three different types of tracers:

• server tracer (com.github.kristofa.brave.ServerTracer)
• client tracer (com.github.kristofa.brave.ClientTracer)
• local tracer (com.github.kristofa.brave.LocalTracer)

Apache CXF integration uses client tracer to instantiate spans on client side (providers and interceptors) to demarcate send / receive cycle, server tracer Apache CXF integration uses  HttpTracing (part of Brave HTTP instrumentation) to instantiate spans on client side (providers and interceptors) to demarcate send / receive cycle as well on the server side (providers and interceptors) to demarcate receive / send cycle, while using local tracer regular Tracer for any spans instantiated within a process.

...

The current integration of distributed tracing in Apache CXF supports OpenZipkin Brave (4.3.16x+ release branch) in JAX-RS 2.x+ and JAX-WS applications, including the applications deploying in OSGi containers. From high-level perspective, JAX-RS 2.x+ integration consists of three main parts:

...

Apache CXF uses HTTP headers to hand off tracing context from the client to the service and from the service to service. Those headers are used internally by OpenZipkin Brave and are not configurable at the moment. The header names are declared in the BraveHttpHeaders the B3Propagation class and at the moment include:

• X-B3-TraceId: 128 or 64-bit trace ID
• X-B3-SpanId: 64-bit span ID
• X-B3-ParentSpanId: 64-bit parent span ID

• X-B3-Sampled: "1" means report this span to the tracing system, "0" means do not

• X-B3-Flags: "1" implies sampled and is a request to override collection-tier sampling policy

By default, BraveClientProvider will try to pass the currently active span through HTTP headers on each service invocation. If there is no active spans, the new span will be created and passed through HTTP headers on per-invocation basis. Essentially, for JAX-RS applications just registering BraveClientProvider on the client and BraveProvider on the server is enough to have tracing context to be properly passed everywhere. The only configuration part which is necessary are span reports(s) and sampler(s).

...

There are a couple of ways the JAX-RS client could be configured, depending on the client implementation. Apache CXF provides its own WebClient which could be configured just like that (in future versions, there would be a simpler ways to do that using client specific features):

// Configure the spans transport sender
final Sender sender = ...; 

/**
 * For example:
 *
 * final Sender sender = LibthriftSender.create("localhost");; 
 */
        
final BraveTracing brave = Tracing
  new  Brave.Builder.newBuilder()
    .localServiceName("web-client")
    .reporter(AsyncReporter.builder(sender).build())
    .traceSampler(Sampler.ALWAYS_SAMPLE) /* or any other Sampler */
    .build();
        
Response response = WebClient
    .create("http://localhost:9000/catalog", Arrays.asList(new BraveClientProvider(brave)))
    .accept(MediaType.APPLICATION_JSON)
    .get();

The configuration based on using the standard JAX-RS Client is very similar:

// Configure the spans transport sender
final Sender sender = ...; 

/**
 * For example:
 *
 * final Sender sender = LibthriftSender.create("localhost");; 
 */
        
final BraveTracing brave = new Brave.Builder Tracing
    .newBuilder()
    .localServiceName("jaxrs-client")
    .reporter(AsyncReporter.builder(sender).build())
    .traceSampler(Sampler.ALWAYS_SAMPLE) /* or any other Sampler */
    .build();
                
final BraveClientProvider provider = new BraveClientProvider(brave);
final Client client = ClientBuilder.newClient().register(provider);

final Response response = client
    .target("http://localhost:9000/catalog")
    .request()
    .accept(MediaType.APPLICATION_JSON)
    .get();

...

@ApplicationPath("/")
public class CatalogApplication extends Application {
    @Override
    public Set<Object> getSingletons() {
        // Configure the spans transport sender
        final Sender sender = ...; 

        /**
         * For example:
         *
         * final Sender sender = LibthriftSender.create("localhost");; 
         */
        
        final BraveTracing brave = new Brave.Builder Tracing
            .newBuilder()
            .localServiceName("tracer")
            .reporter(AsyncReporter.builder(sender).build())
            .traceSampler(Sampler.ALWAYS_SAMPLE) /* or any other Sampler */ 
            .build();
            
        return new HashSet<>(
                Arrays.asList(
                    new BraveFeature(brave)
                )
            );
    }
}

...

// Configure the spans transport sender
final Sender sender = ...; 

/**
 * For example:
 *
 * final Sender sender = LibthriftSender.create("localhost");; 
 */
        
final BraveTracing brave = newTracing
    Brave.BuildernewBuilder("tracer")
    .reporter(localServiceName("tracer")
    .reporter(AsyncReporter.builder(sender).build())
    .traceSampler(Sampler.ALWAYS_SAMPLE) /* or any other Sampler */ 
    .build();

final JAXRSServerFactoryBean factory = RuntimeDelegate.getInstance().createEndpoint(/* application instance */, JAXRSServerFactoryBean.class);
factory.setProvider(new BraveFeature(brave));
...
return factory.create();

...

The actual invocation of the request by the client (with service name tracer-client) and consequent invocation of the service on the server side (service name traceser-server) is going to generate the following sample traces:

Image RemovedImage Added

Example #4: Client and Server with binary annotations (key/value)

...

In this example server-side implementation of the JAX-RS service is going to offload some work into thread pool and then return the response to the client, simulating parallel execution. For this example to work, the OpenZipkin Brave on server side should be configured a little bit differently (using InheritableServerClientAndLocalSpanState):The client-side code stays unchanged.

final Endpoint endpoint =  Endpoint.create("tracer-server",
    ByteBuffer.wrap(Inet4Address.getLocalHost().getAddress()).getInt());@Produces( { MediaType.APPLICATION_JSON } )
@GET
public Collection<Book> getBooks(@Context final TracerContext tracer) throws Exception {
    final Future<Book> book1 = 
final Brave brave = new Brave.Builder(new InheritableServerClientAndLocalSpanState(endpoint))executor.submit(
        tracer.wrap("Getting Book 1", new Traceable<Book>() {
    .reporter(AsyncReporter.builder(sender).build())
     .traceSampler(Sampler.ALWAYS_SAMPLE)
    .build();

The client-side code stays unchanged.

@Produces( { MediaType.APPLICATION_JSON } )
@GET
public Collection<Book> getBooks(@Context public Book call(final TracerContext tracercontext) throws Exception {
    final Future<Book> book1 = executor.submit(
        tracer.wrap("Getting Book 1", new Traceable<Book>() {
            public Book call(final TracerContext context) throws Exception {
                // Simulating a delay of 100ms required to call external system// Simulating a delay of 100ms required to call external system
                Thread.sleep(100);
                    
                return new Book("Apache CXF Web Service Development", 
                    "Naveen Balani, Rajeev Hathi");
            }
        })
    );
        
    final Future<Book> book2 = executor.submit(
        tracer.wrap("Getting Book 2", new Traceable<Book>() {
            public Book call(final TracerContext context) throws Exception {
                // Simulating a delay of 100ms required to call external system
                Thread.sleep(200);
                    
                return new Book("Developing Web Services with Apache CXF and Axis2", 
                    "Kent Ka Iok Tong");
            }
        })
    );
       
    return Arrays.asList(book1.get(), book2.get());
}

The actual invocation of the request by the client (with service name tracer-client) and consequent invocation of the service on the server side (process name tracer-server) is going to generate the following sample traces:

Image RemovedImage Added

Example #6: Client and Server with asynchronous JAX-RS service (server-side)

In this example server-side implementation of the JAX-RS service is going to be executed asynchronously. It poses a challenge from the tracing prospective as request and response are processed in different threads (in general). At the moment, Apache CXF does not support the transparent tracing spans management (except for default use case) but provides the simple ways to do that (by letting to transfer spans from thread to thread). As with the previous example, the OpenZipkin Brave on server side should be configured a little bit differently (using InheritableServerClientAndLocalSpanState):The client-side code stays unchanged.

final@Produces( Endpoint endpoint =  Endpoint.create("tracer-server",
    ByteBuffer.wrap(Inet4Address.getLocalHost().getAddress()).getInt());
        
final Brave brave = new Brave.Builder(new InheritableServerClientAndLocalSpanState(endpoint))
    .reporter(AsyncReporter.builder(sender).build())
    .traceSampler(Sampler.ALWAYS_SAMPLE)
    .build();

The client-side code stays unchanged.

@Produces( { MediaType.APPLICATION_JSON } )
@GET
public void getBooks(@Suspended final AsyncResponse response, @Context final TracerContext tracer) throws Exception {
    tracer.continueSpan(new Traceable<Future<Void>>() {
{ MediaType.APPLICATION_JSON } )
@GET
public void getBooks(@Suspended final AsyncResponse response, @Context final TracerContext tracer) throws Exception {
    tracer.continueSpan(new Traceable<Future<Void>>() {
        public Future<Void> call(final TracerContext context) throws Exception {
            return executor.submit(
                tracer.wrap("Getting Book", new Traceable<Void>() {
                    public Future<Void>Void call(final TracerContext context) throws Exception {
            return  executor.submit(
          // Simulating a processing delay  tracer.wrap("Getting Book", new Traceable<Void>() {
of 50ms
                       public Void call(final TracerContext context) throws Exception {
  Thread.sleep(50);
                       // Simulating a processing delay of 50ms
                        Threadresponse.sleepresume(50);
                            Arrays.asList(
                         response.resume(
       new Book("Apache CXF Web Service Development", "Naveen Balani, Rajeev Hathi")
            Arrays.asList(
                                new Book("Apache CXF Web Service Development", "Naveen Balani, Rajeev Hathi")
                            )
                        );
                            
                        return null;
                    }
                })
            );
        }
    });
}

The actual invocation of the request by the client (with service name tracer-client) and consequent invocation of the service on the server side (service name tracer-server) is going to generate the following sample traces:

Image RemovedImage Added

Example #7: Client and Server with asynchronous invocation (client-side)

...

In this respect, there is no difference from the caller prospective however a bit more work is going under the hood to transfer the active tracing span from JAX-RS client request filter to client response filter as in general those are executed in different threads (similarly to server-side asynchronous JAX-RS resource invocation). The actual invocation of the request by the client (with service name tracer-client) and consequent invocation of the service on the server side (service name tracer-server) is going to generate the following sample traces:

Image RemovedImage Added

Distributed Tracing with OpenZipkin Brave and JAX-WS support

// TODO

Migrating from brave-cxf3

Distributed tracing in the Apache CXF is build primarily around JAX-RS 2.x implementation. However, JAX-WS is also supported but it requires to write some boiler-plate code and use OpenZipkin Brave API directly (the JAX-WS integration is going to be enhanced in the future). Essentially, from the server-side prospective the in/out interceptors, BraveStartInterceptor and BraveStopInterceptor respectively, should be configured as part of interceptor chains, either manually or using BraveFeature. For example:

// Configure the spans transport sender
final Sender sender = ...; 

/**
 * For example:
 *
 * final Sender sender = LibthriftSender.create("localhost");; 
 */
        
final Tracing brave = Tracing
    .newBuilder()
    .localServiceName("tracer")
    .reporter(AsyncReporter.builder(sender).build())
    .traceSampler(Sampler.ALWAYS_SAMPLE) /* or any other Sampler */ 
    .build();
             
final JaxWsServerFactoryBean sf = new JaxWsServerFactoryBean();
...
sf.getFeatures().add(new BraveFeature(brave));
...
sf.create();

Similarly to the server-side, client-side needs own set of out/in interceptors, BraveClientStartInterceptor and BraveClientStopInterceptor (or BraveClientFeature). Please notice the difference from server-side:  BraveClientStartInterceptor becomes out-interceptor while BraveClientStopInterceptor becomes in-interceptor. For example:

// Configure the spans transport sender
final Sender sender = ...; 

/**
 * For example:
 *
 * final Sender sender = LibthriftSender.create("localhost");; 
 */
        
final Tracing brave = Tracing
    .newBuilder()
    .localServiceName("tracer")
    .reporter(AsyncReporter.builder(sender).build())
    .traceSampler(Sampler.ALWAYS_SAMPLE) /* or any other Sampler */ 
    .build();
             
final JaxWsProxyFactoryBean sf = new JaxWsProxyFactoryBean();
...
sf.getFeatures().add(new BraveClientFeature(brave));
...
sf.create();

Distributed Tracing with OpenZipkin Brave and OSGi

OpenZipkin Brave could be deployed into OSGi container and as such, distributed tracing integration is fully available for Apache CXF services running inside the container. For a complete example please take a look on jax_ws_tracing_brave_osgi sample project, but here is the typical OSGi  Blueprint snippet:

<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:cxf="http://cxf.apache.org/blueprint/core"
       xmlns:jaxws="http://cxf.apache.org/blueprint/jaxws"

       xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
                           http://cxf.apache.org/blueprint/core http://cxf.apache.org/schemas/blueprint/core.xsd
                           http://cxf.apache.org/blueprint/jaxws http://cxf.apache.org/schemas/blueprint/jaxws.xsd">

    <!-- CXF BraveFeature -->
    <bean id="braveFeature" class="org.apache.cxf.tracing.brave.BraveFeature">
        <argument index="0" ref="brave" />
    </bean>
    
    <cxf:bus>
        <cxf:features>
            <cxf:logging />
        </cxf:features>
    </cxf:bus>
    
    <bean id="catalogServiceImpl" class="demo.jaxws.tracing.server.impl.CatalogServiceImpl">
        <argument index="0" ref="brave" />
    </bean>
    
    <bean id="braveBuilder" class="brave.Tracing" factory-method="newBuilder" />
   
    <bean id="braveCatalogBuilder" factory-ref="braveBuilder" factory-method="localServiceName">
        <argument index="0" value="catalog-service" />
    </bean>
    
    <bean id="brave" factory-ref="braveCatalogBuilder" factory-method="build" />

    <jaxws:endpoint
        implementor="#catalogServiceImpl"
        address="/catalog"
        implementorClass="demo.jaxws.tracing.server.impl.CatalogServiceImpl">
        <jaxws:features>
            <ref component-id="braveFeature" />
        </jaxws:features>
    </jaxws:endpoint>
</blueprint>

Migrating from brave-cxf3

The migration path from OpenZipkin Brave / CXF to Apache CXF integration is pretty straightforward and essentially boils down to using JAX-RS ( BraveFeature for server side / BraveClientFeature for client side (imported from org.apache.cxf.tracing.brave.jaxrs package), for example:

JAXRSServerFactoryBean serverFactory = new JAXRSServerFactoryBean();
serverFactory.setServiceBeans(new RestFooService());
serverFactory.setAddress("http://localhost:9001/");
serverFactory.getFeatures().add(new BraveFeature(brave));
serverFactory.create();

Although you may continue to use OpenZipkin Brave API directly, for the server-side it is preferable to inject @Context TracerContext  into your JAX-RS services in order to interface with the tracer.

JAXRSClientFactoryBean clientFactory = new JAXRSClientFactoryBean();
clientFactory.setAddress("http://localhost:9001/");
clientFactory.setServiceClass(FooService.class);
clientFactory.getFeatures().add(new BraveClientFeature(brave));
FooService client = (FooService) clientFactory.create()

Similarly for JAX-WS BraveFeature for server side / BraveClientFeature for client side (imported from org.apache.cxf.tracing.brave package), for example:

JaxWsServerFactoryBean serverFactory = new JaxWsServerFactoryBean();
serverFactory.setAddress("http://localhost:9000/test");
serverFactory.setServiceClass(FooService.class);
serverFactory.setServiceBean(fooServiceImplementation);
serverFactory.getFeatures().add(new BraveFeature(brave));

serverFactory.create();

JAXRSClientFactoryBean clientFactory = new JAXRSClientFactoryBean();
clientFactory.setAddress("http://localhost:9001/");
clientFactory.setServiceClass(FooService.class);
clientFactory.getFeatures().add(new BraveClientFeature(brave));
FooService client = (FooService) clientFactory.create();

Spring XML-Configuration

If your project uses classic Spring XML-Configuration, you should consider using brave-spring-beans. The factory beans allow to create the config like this:

<bean id="braveFeature" class="org.apache.cxf.tracing.brave.BraveFeature"><!-- JAX-WS server feature -->
   <constructor-arg ref="httpTracing" />
</bean>

<bean id="httpTracing" class="brave.spring.beans.HttpTracingFactoryBean">
   <property name="tracing">
      <bean class="brave.spring.beans.TracingFactoryBean">
         <property name="localServiceName" value="myService"/>
         <property name="reporter">
            <bean class="brave.spring.beans.AsyncReporterFactoryBean">
               <property name="sender">
                  <bean class="zipkin.reporter.urlconnection.URLConnectionSender" factory-method="create">
                     <constructor-arg value="http://localhost:9411/api/v1/spans"/>
                  </bean>
               </property>
            </bean>
         </property>
         <property name="currentTraceContext">
            <bean class="brave.context.slf4j.MDCCurrentTraceContext" factory-method="create"/>
         </property>
      </bean>
   </property>
   <property name="clientParser">
      <bean class="org.apache.cxf.tracing.brave.HttpClientSpanParser" />
   </property>
   <property name="serverParser">
      <bean class="org.apache.cxf.tracing.brave.HttpServerSpanParser" />
   </property>
</bean>

Using non-JAX-RS clients

The  Apache CXF  uses native OpenZipkin Brave capabilities so the existing instrumentations for different HTTP clients work as expected. The usage of only JAX-RS client is not required. For example, the following snippet demonstrates the usage of traceble OkHttp client  to call JAX-RS resources, backed by Apache CXF .

final Tracing brave = Tracing
    .newBuilder()
    .localServiceName("web-client")
    .reporter(AsyncReporter.builder(sender).build())
    .traceSampler(Sampler.ALWAYS_SAMPLE) /* or any other Sampler */
    .build();

final OkHttpClient client = new OkHttpClient();
final Call.Factory factory = TracingCallFactory.create(brave, client);
            
final Request request = new Request.Builder()
    .url("http://localhost:9000/catalog")
    .header("Accept", "application/json")
    .build();

try (final Response response = factory.newCall(request).execute()) {
    // Do something with response.body()
}

Accessing Brave APIs

The Apache CXF  abstracts as much of the tracer-specific APIs behind TracerContext as possible. However, sometimes there is a need to get access to OpenZipkin Brave APIs in order to leverages the rich set of available instrumentations. To make it possible, TracerContext has a dedicated unwrap method which returns underlying HttpTracing, Tracer or Tracing instances. The snippet below shows off how to use this API and use OpenZipkin Brave instrumentation for Apache HttpClient.

@GET
@Path("/search")
@Produces(MediaType.APPLICATION_JSON)
public JsonObject search(@QueryParam("q") final String query, @Context final TracerContext tracing) throws Exception {
    final CloseableHttpClient httpclient = TracingHttpClientBuilder
        .create(tracing.unwrap(HttpTracing.class))
        .build();
    
    try {
        final URI uri = new URIBuilder("https://www.googleapis.com/books/v1/volumes")
            .setParameter("q", query)
            .build();
            
        final HttpGet request = new HttpGet(uri);
        request.setHeader("Accept", "application/json");
            
        final HttpResponse response = httpclient.execute(request);
        final String data = EntityUtils.toString(response.getEntity(), StandardCharsets.UTF_8);
        try (final StringReader reader = new StringReader(data)) {
            return Json.createReader(reader).readObject();
        }
    } finally {
        httpclient.close();
    }
}

The usage of tracer-specific APIs is not generally advisable (because of portability reasons) but in case there are no other options available, it is available.// TODO