Overview

OpenTracing is a vendor-neutral open standard for distributed tracing. Essentially, for Java-based projects the specification exists as a set of Java APIs which any distributed tracing solution is welcome to implement. There are quite a few distributed tracing frameworks available which are compatible with OpenTracing, notably Zipkin (via community contributions like bridge from Brave to OpenTracing ), Lightstep and Jaeger. Starting from 3.2.1 release, Apache CXF fully supports integration (through cxf-integration-tracing-opentracing module) with any distributed tracer that provides OpenTracing Java API implementation.

The section dedicated to Apache HTrace has pretty good introduction into distributed tracing basics however OpenTracing specification abstracts a lot of things, outlining just a general APIs to denote the Span lifecycle and injection points to propagate the context across many distributed components. As such, the intrinsic details about HTTP headers f.e. becomes an integral part of the distributed tracer of your choice, out of reach for Apache CXF.

Distributed Tracing in Apache CXF using OpenTracing

Apache CXF is a very popular framework for building services and web APIs. No doubts, it is going to play even more important role in context of microservices architecture letting developers to quickly build and deploy individual JAX-RS/JAX-WS services. Distributed tracing is an essential technique to observe the application platform as a whole, breaking the request to individual service traces as it goes through and crosses the boundaries of threads, processes and machines.

The current integration of distributed tracing in Apache CXF supports OpenTracing Java API 0.30.0+ and provides full-fledged support of JAX-RS 2.x / JAX-WS applications. From high-level prospective, the JAX-RS integration consists of three main parts:

• TracerContext (injectable through @Context annotation)
• OpenTracingProvider (server-side JAX-RS provider) and OpenTracingClientProvider (client-side JAX-RS provider)
• OpenTracingFeature (server-side JAX-RS feature) to simplify the configuration and integration

Similarly, from high-level perspective, JAX-WS integration includes:

• OpenTracingStartInterceptor / OpenTracingStopInterceptor / OpenTracingFeature Apache CXF feature (server-side JAX-WS support)
• OpenTracingClientStartInterceptor / OpenTracingClientStopInterceptor / OpenTracingClientFeature Apache CXF feature (client-side JAX-WS support)

Apache CXF uses HTTP headers to hand off tracing context from the client to the service and from the service to service. Those headers are specific to distributing tracing framework you have picked and are not configurable at the moment (unless the framework itself has a way to do that).

By default, OpenTracingClientProvider will try to pass the currently active span through HTTP headers on each service invocation. If there is no active spans, the new span will be created and passed through HTTP headers on per-invocation basis. Essentially, for JAX-RS applications just registering OpenTracingClientProvider on the client and OpenTracingProvider on the server is enough to have tracing context to be properly passed everywhere. The only configuration part which is necessary are span reports(s) and sampler(s) which are, not surprisingly, specific to distributing tracing framework you have chosen.

It is also worth to mention the way Apache CXF attaches the description to spans. With regards to the client integration, the description becomes a full URL being invoked prefixed by HTTP method, for example: GET http://localhost:8282/books. On the server side integration, the description becomes a relative JAX-RS resource path prefixed by HTTP method, f.e.: GET books, POST book/123

Configuring Client

In this section and below, all the code snippets are going to be based on Jaeger distributed tracing framework (release 0.20.6+), although everything we are going to discuss is equally applicable to any other existing alternatives. Essentially, the only dependency Apache CXF integration relies on is the Tracer instance.

There are a couple of ways the JAX-RS client could be configured, depending on the client implementation. Apache CXF provides its own WebClient which could be configured just like that (in future versions, there would be a simpler ways to do that using client specific features):

final Tracer tracer = new Configuration("web-client", 
        new Configuration.SamplerConfiguration(ConstSampler.TYPE, 1), /* or any other Sampler */
        new Configuration.ReporterConfiguration(new HttpSender("http://localhost:14268/api/traces")) /* or any other Sender */
    ).getTracer();
                
Response response = WebClient
    .create("http://localhost:9000/catalog", Arrays.asList(new OpenTracingClientProvider(tracer)))
    .accept(MediaType.APPLICATION_JSON)
    .get();

The configuration based on using the standard JAX-RS Client is very similar:

final Tracer tracer = new Configuration("jaxrs-client", 
        new Configuration.SamplerConfiguration(ConstSampler.TYPE, 1), /* or any other Sampler */
        new Configuration.ReporterConfiguration(new HttpSender("http://localhost:14268/api/traces")) /* or any other Sender */
    ).getTracer();
                
final OpenTracingClientProvider provider = new OpenTracingClientProvider(tracer);
final Client client = ClientBuilder.newClient().register(provider);
    
final Response response = client
  .target("http://localhost:9000/catalog")
  .request()
  .accept(MediaType.APPLICATION_JSON)
  .get();

Alternatively, you may use GlobalTracer to pass the tracer around, for example:

final Tracer tracer = new Configuration("jaxrs-client", 
        new Configuration.SamplerConfiguration(ConstSampler.TYPE, 1), /* or any other Sampler */
        new Configuration.ReporterConfiguration(new HttpSender("http://localhost:14268/api/traces")) /* or any other Sender */
    ).getTracer();

// This method should only be called once during the application initialization phase.
GlobalTracer.register(tracer);

// No explicit Tracer instance is required, it will be picked off the GlobalTracer using get() method
final OpenTracingClientProvider provider = new OpenTracingClientProvider();

Configuring Server

Server configuration is a bit simpler than the client one thanks to the feature class available, OpenTracingFeature. Depending on the way the Apache CXF is used to configure JAX-RS services, it could be part of JAX-RS application configuration, for example:

@ApplicationPath( "/" )
public class CatalogApplication extends Application {
    @Override
    public Set<Object> getSingletons() {
        final Tracer tracer = new Configuration("tracer-server", 
                new Configuration.SamplerConfiguration(ConstSampler.TYPE, 1), /* or any other Sampler */
                new Configuration.ReporterConfiguration(new HttpSender("http://localhost:14268/api/traces")) /* or any other Sender */
            ).getTracer();

            
        return new HashSet<>(
                Arrays.asList(
                    new OpenTracingFeature(tracer)
                )
            );
    } 
}

Or it could be configured using JAXRSServerFactoryBean as well, for example:

final Tracer tracer = new Configuration("tracer-server", 
        new Configuration.SamplerConfiguration(ConstSampler.TYPE, 1), /* or any other Sampler */
        new Configuration.ReporterConfiguration(new HttpSender("http://localhost:14268/api/traces")) /* or any other Sender */
    ).getTracer();

final JAXRSServerFactoryBean factory = RuntimeDelegate.getInstance().createEndpoint(/* application instance */, JAXRSServerFactoryBean.class);
factory.setProvider(new OpenTracingFeature(tracer));
...
return factory.create();

Alternatively, you may rely on GlobalTracer to pass the tracer around, so in this case the OpenTracingFeature will pick it up from there, for example:

@ApplicationPath( "/" )
public class CatalogApplication extends Application {
    @Override
    public Set<Object> getSingletons() {
        return new HashSet<>(
                Arrays.asList(
                    // No explicit Tracer instance is required, it will be picked off the GlobalTracer using get() method
                    new OpenTracingFeature()
                )
            );
    } 
}

Once the span reporter and sampler are properly configured, all generated spans are going to be collected and available for analysis and/or visualization.

Distributed Tracing In Action: Usage Scenarios

In the following subsections we are going to walk through many different scenarios to illustrate the distributed tracing in action, starting from the simplest ones and finishing with asynchronous JAX-RS services. All examples assume that configuration has been done (see please Configuring Client  and Configuring Server sections above).

Example #1: Client and Server with default distributed tracing configured

In the first example we are going to see the effect of using default configuration on the client and on the server, with only OpenTracingClientProvider  and OpenTracingProvider registered. The JAX-RS resource endpoint is pretty basic stubbed method:

@Produces( { MediaType.APPLICATION_JSON } )
@GET
public Collection<Book> getBooks() {
    return Arrays.asList(
        new Book("Apache CXF Web Service Development", "Naveen Balani, Rajeev Hathi")
    );
}

The client is as simple as that:

final Response response = client
    .target("http://localhost:8282/books")
    .request()
    .accept(MediaType.APPLICATION_JSON)
    .get();

The actual invocation of the request by the client (with service name tracer-client) and consequent invocation of the service on the server side (service name tracer-server) is going to generate the following sample traces (taken from Jaeger UI):

The same trace will be looking pretty similar using traditional Zipkin UI frontend:

Example #2: Client and Server with nested trace

In this example server-side implementation of the JAX-RS service is going to call an external system (simulated as a simple delay of 500ms) within its own span. The client-side code stays unchanged.

@Produces( { MediaType.APPLICATION_JSON } )
@GET
public Collection<Book> getBooks(@Context final TracerContext tracer) throws Exception {
    try(final ActiveSpan scope = tracer.startSpan("Calling External System")) {
        // Simulating a delay of 500ms required to call external system
        Thread.sleep(500);
            
        return Arrays.asList(
            new Book("Apache CXF Web Service Development", "Naveen Balani, Rajeev Hathi")
        );
    }
}

The actual invocation of the request by the client (with service name tracer-client) and consequent invocation of the service on the server side (service name tracer-server) is going to generate the following sample traces (taken from Jaeger UI):

The same trace will be looking pretty similar using traditional Zipkin UI frontend:

Example #3: Client and Server trace with timeline

In this example server-side implementation of the JAX-RS service is going to add timeline to the active span. The client-side code stays unchanged.

@Produces( { MediaType.APPLICATION_JSON } )
@GET
public Collection<Book> getBooks(@Context final TracerContext tracer) throws Exception {
    tracer.timeline("Preparing Books");
    // Simulating some work using a delay of 100ms
    Thread.sleep(100);
         
    return Arrays.asList(
        new Book("Apache CXF Web Service Development", "Naveen Balani, Rajeev Hathi")
    );
}

The actual invocation of the request by the client (with service name tracer-client) and consequent invocation of the service on the server side (service name traceser-server) is going to generate the following sample traces (taken from Jaeger UI):

The same trace will be looking pretty similar using traditional Zipkin UI frontend: