Overview

OpenTelemetry (formed through a merger of the OpenTracing and OpenCensus projects) is a collection of APIs, SDKs, and tools to instrument, generate, collect, and export telemetry data like metrics, logs, and distributed traces. OpenTelemetry is vendor-neutral open-source observability framework and as an industry-standard, it is natively supported by a number of vendors. From the instrumentation perspective, there is support for quite a few programming languages, including dedicated Java APIs. Starting from 3.5.7 /3.6.2 / 4.0.3 releases, Apache CXF fully supports integration (through cxf-integration-tracing-opentelemetry module) with OpenTelemetry distributed tracing capabilities.

The section dedicated to Apache HTrace has pretty good introduction into distributed tracing basics however OpenTelemetry specification abstracts a lot of things, outlining just a general APIs to denote the Span lifecycle and injection points to propagate the context across many distributed components. As such, the intrinsic details about HTTP headers f.e. becomes an integral part of the distributed tracer of your choice, out of reach for Apache CXF.

Distributed Tracing in Apache CXF using OpenTelemetry

The current integration of the OpenTelemetry's distributed tracing in Apache CXF supports OpenTelemetry Java SDK 1.28.0+ (using semantic conventions) and provides full-fledged support of JAX-RS 2.x / JAX-WS applications. From high-level prospective, the JAX-RS integration consists of three main parts:

TracerContext (injectable through @Context annotation)
OpenTelemetryProvider (server-side JAX-RS provider) and OpenTelemetryClientProvider (client-side JAX-RS provider)
OpenTelemetryFeature (server-side JAX-RS feature) to simplify the configuration and integration

Similarly, from high-level perspective, JAX-WS integration includes:

OpenTelemetryStartInterceptor / OpenTelemetryStopInterceptor / OpenTelemetryFeature Apache CXF feature (server-side JAX-WS support)
OpenTelemetryClientStartInterceptor / OpenTelemetryClientStopInterceptor / OpenTelemetryClientFeature Apache CXF feature (client-side JAX-WS support)

Apache CXF uses HTTP headers to hand off tracing context from the client to the service and from the service to service. Those headers are specific to distributing tracing framework you have picked and are not configurable at the moment (unless the framework itself has a way to do that).

By default, OpenTelemetryClientProvider will use configured propagators to pass the currently active span through HTTP headers on each service invocation. If there is no active spans, the new span will be created and passed through HTTP headers on per-invocation basis. Essentially, for JAX-RS applications just registering OpenTelemetryClientProvider on the client and OpenTelemetryProvider on the server is enough to have tracing context to be properly passed everywhere. The only configuration part which is necessary are span exporters(s) and sampler(s) which are, not surprisingly, specific to distributing tracing vendor you have chosen.

It is also worth to mention the way Apache CXF attaches the description to spans. With regards to the client integration, the description becomes a full URL being invoked prefixed by HTTP method, for example: GET http://localhost:8282/books. On the server side integration, the description becomes a relative JAX-RS resource path prefixed by HTTP method, f.e.: GET books, POST book/123

Configuring Client

There are a couple of ways the JAX-RS client could be configured, depending on the client implementation. Apache CXF provides its own WebClient which could be configured just like that (in future versions, there would be a simpler ways to do that using client specific features):

final SdkTracerProvider sdkTracerProvider = SdkTracerProvider
     .builder()
     ...
     .build();

final OpenTelemetry openTelemetry = OpenTelemetrySdk
    .builder()
    .setTracerProvider(sdkTracerProvider)
    ...
    .buildAndRegisterGlobal();

final Tracer tracer =  openTelemetry.getTracer("web-client");
                  
Response response = WebClient
    .create("http://localhost:9000/catalog", Arrays.asList(new OpenTelemetryClientProvider(openTelemetry,  tracer))
    .accept(MediaType.APPLICATION_JSON)
    .get();

The configuration based on using the standard JAX-RS Client is very similar:

final SdkTracerProvider sdkTracerProvider = SdkTracerProvider
     .builder()
     ...
     .build();

final OpenTelemetry openTelemetry = OpenTelemetrySdk
    .builder()
    .setTracerProvider(sdkTracerProvider)
    ...
    .buildAndRegisterGlobal();

final Tracer tracer =  openTelemetry.getTracer("jaxrs-client");
final OpenTelemetryClientProvider provider = new OpenTelemetryClientProvider(openTelemetry, tracer);
final Client client = ClientBuilder.newClient().register(provider);
 
final Response response = client
    .target("http://localhost:9000/catalog")
    .request()
    .accept(MediaType.APPLICATION_JSON)
    .get();

Configuring Server

Server configuration is a bit simpler than the client one thanks to the feature class available, OpenTelemetryFeature. Depending on the way the Apache CXF is used to configure JAX-RS services, it could be part of JAX-RS application configuration, for example:

@ApplicationPath("/")
public class CatalogApplication extends Application {
    @Override
    public Set<Object> getSingletons() {
        final SdkTracerProvider sdkTracerProvider = SdkTracerProvider
           .builder()
           ...
           .build();

       final OpenTelemetry openTelemetry = OpenTelemetrySdk
           .builder()
           .setTracerProvider(sdkTracerProvider)
           ...
           .buildAndRegisterGlobal();

        final Tracer tracer =  openTelemetry.getTracer("tracer");
        return new HashSet<>(
                Arrays.asList(
                    new OpenTelemetryFeature(openTelemetry,  tracer)
                )
            );
    }
}

Or it could be configured using JAXRSServerFactoryBean as well, for example:

final SdkTracerProvider sdkTracerProvider = SdkTracerProvider
    .builder()
    ...
    .build();

final OpenTelemetry openTelemetry = OpenTelemetrySdk
    .builder()
    .setTracerProvider(sdkTracerProvider)
    ...
    .buildAndRegisterGlobal(); 
 
final Tracer tracer =  openTelemetry.getTracer("tracer"); 
final JAXRSServerFactoryBean factory = RuntimeDelegate.getInstance().createEndpoint(/* application instance */, JAXRSServerFactoryBean.class);
factory.setProvider(new OpenTelemetryFeature(openTelemetry,  tracer));
...
return factory.create();

Once the span processor(s) and sampler are properly configured, all generated spans are going to be collected and available for analysis and/or visualization.

Distributed Tracing In Action: Usage Scenarios

TBD

Example #1: Client and Server with default distributed tracing configured

TBD

Example #2: Client and Server with nested trace

TBD

Example #3: Client and Server trace with annotations

TBD

Example #4: Client and Server with binary annotations (key/value)

TBD

Example #5: Client and Server with parallel trace (involving thread pools)

TBD

Example #6: Client and Server with asynchronous JAX-RS service (server-side)

TBD

Example #7: Client and Server with asynchronous invocation (client-side)

TBD

Distributed Tracing with OpenTelemetry and JAX-WS support

TBD

Distributed Tracing with OpenTelemetry and OSGi

TBD

Spring XML-Configuration

TBD

Using non-JAX-RS clients

TBD

Child pages

Using OpenTelemetry