Status
Current state: Draft
Discussion thread: -
JIRA: -
Released: -
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Proof of concept demo available here: https://github.com/yukim/cassandra-opentelemetry-demo
Scope
- Provide unified way of exporting tracing, metrics, and logging to external monitoring system with ease of configuration
Goals
The goal of this proposal is to integrate OpenTelemetry into Apache Cassandra so that exporting telemetries and setting up the monitoring system is much easier. This feature is opt-in and does not remove the currently available telemetries and the way to export.
Approach
While the goal of this CEP is to integrate all the OpenTelemetry features, the proposal is separated into several steps. OpenTelemetry spec and libraries are still evolving. Opentelemetry Java library provides stable support for Tracing and Metrics, however Logging support is experimental, as of the time of writing (May 2023). See https://opentelemetry.io/status/.
The steps need to be agreed are:
- OpenTelemetry Configuration
- How to enable OpenTelemetry export
- OpenTelemetry Tracing integration
- Standardize the protocol for context propagation
- The way to implement instrumentation
- Attributes to export along with the tracing
- OpenTelemetry Metrics integration
- The way to implement instrumentation
- Attributes to export along with the tracing
- OpenTelemetry Logging integration
- The way to implement instrumentation
Timeline
todo
Mailing list / Slack channels
Mailing list:
Slack channel:
Discussion threads:
Related JIRA tickets
JIRA(s):
- -
Motivation
Troubleshooting Apache Cassandra when failures or performance problems arise is time consuming and sometimes difficult to get to the root cause without a proper observability system is set up. For three pillars of observability - Tracing, Metrics and Logging, Apache Cassandra already either implements its own way or uses external libraries to give operators and administrators the deep insights of the complex distributed database:
- For tracing, Apache Cassandra has its own Tracing API.
- For metrics collection and reporting, Apache Cassandra uses Codahale’s Metrics library to collect and expose through JMX.
- For the logging, Apache Cassandra uses Slf4J/Logback logging library.
However, these features are much more useful when they are available in the observability system that can correlate these telemetries together. Otherwise, operators and admins need to pull out telemetries one by one, and put together the information by hand, to create an assumption that can be a cause of problem.
For Apache Cassandra, operators need to implement their own way to pull out these telemetries and establish their own monitoring stack, for example by using open source software like Prometheus / Grafana, or commercial services like Datadog, etc. Implementing observability of Apache Cassandra depends on what observation software is used where the method to set up is different from software to software, and oftentimes is ignored by operators for the complexity of setting up the stack.
OpenTelemetry is the project hosted at CNCF to provide "A single, vendor-agnostic instrumentation library per language with support for both automatic and manual instrumentation"(https://opentelemetry.io/docs/concepts/what-is-opentelemetry/). It specifies APIs to collect tracing, metrics, and logging, and protocols to export to the external observation software.
By introducing OpenTelemetry, exporting telemetries and integrating observability software should be much easier with fewer configurations to be done. It also opens doors for the observability software projects / venders / service providers to implement Cassandra specific solutions based on the standard that OpenTelemetry provides.
Audience
- Cassandra contributors / DevOps / DBAs
- APM, Observation OSS projects / software venders / service providers
Proposed Changes
This CEP aims to integrate OpenTelemetry Tracing, Metrics and Logging with the existing implementations (Cassandra’s own Tracing API, Codahale/Dropwizard metrics, and Slf4J/Logback), while maintaining the backward compatibility.
Configuration
Apache Cassandra uses OpenTelemetry SDK to export tracing, metrics and logs. The operator can configure how these telemetry are exported to the external system by changing the configuration in cassandra.yaml
file and jvm-server.options
file.
Apache Cassandra uses OpenTelemetry SDK Autoconfigure to configure OpenTelemetry exporters, and only the OTLP exporter jars are included in the distribution. Operators can provide the necessary jars and configuration to use other exporters (i.e. Jeager for tracing) as well.
For example, if you want to export tracing to Jeager, you need to add opentelemetry-exporter-jeager.jar
file in the classpath, and configure through jvm-server.options
:
# jvm-server.options
# Configure tracing export to Jeager
-Dotel.traces.exporter=jeager
-Dotel.exporter.jaeger.endpoint=http://jeager:14250
OpenTelemetry can be configured using environmental variables as well. This is useful for containerized environment like Kubernetes.
Exporting Trace through OpenTelemetry
- To propagate OpenTelemetry
Context
from the application through Native Protocol, the standardized way to use custom payload will be defined. - To propagate OpenTelemetry
Context
between the nodes, new inter-node messaging header will be introduced. - To propagate OpenTelemetry
Context
between threads,ExecutorLocals
will hold the currentContext
. - OpenTelemetry Tracing root
Span
will be created when a coordinator node receives the message from the client (the same timing when Cassandra Tracing starts). - In order to avoid re-instrumenting code with OpenTelemetry API,
Tracing
object will produce OpenTelemetry tracing event when itstrace
method is called.- Note that OpenTelemetry Tracing will not produce anything when it is not enabled.
Context propagation
From the client to Cassandra node through Native Protocol
The CEP proposes standardized way to propagate OpenTelemetry Context
from the applications through Native Protocol’s custom payload, rather than introducing new header definitions.
By using custom payload, the clients can easily start propagating Context
without upgrading the driver they are using.
Right now, only text propagator based on W3C Trace Context is officially defined and provided. Binary propagator will be added in the future (https://github.com/open-telemetry/opentelemetry-specification/issues/437).
By following the W3C standard, the CEP proposes to use the following custom payload key value pairs to propagate OpenTelemetry Context
:
Name | Description | Type of value | Example value |
---|---|---|---|
traceparent | The same header as W3C Trace Context carrying parent tracing information. https://www.w3.org/TR/trace-context/#traceparent-header | String | 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01 |
tracestate | The same header as W3C Trace Context carrying vendor specific information. https://www.w3.org/TR/trace-context/#tracestate-header | String | vendorname1=opaqueValue1,vendorname2=opaqueValue2 |
In the client side, we can use the standard class to inject the Context
into custom payload. For java, it would be like:
Map<String, ByteBuffer> payload = new HashMap<>();
W3CTraceContextPropagator.getInstance().inject(Context.current(), payload, (carrier, key, value) -> {
if (carrier != null) {
carrier.put(key, ByteBuffer.wrap(value.getBytes(StandardCharsets.UTF_8)));
}
});
Statement<?> injected = statement.setCustomPayload(payload);
session.execute(injected);
Between nodes through inter-node messaging protocol
To propagate OpenTelemetry Context
between the nodes, the sender adds the new ParamType
to the message only when:
- The message is a request message (not response message).
- Current
Span
is recording.- This is to prevent creating unnecessary root Spans for every Message Verbs.
The new ParamType Context takes the following values from the current Context.
- Trace parent info
- trace id
- Span Id
- Flags
- Trace state
Upon receiving the message containing Context ParamType, the receiver constructs the remote span context, which becomes the parent of the span in this node.
Between threads through ExecutorLocals
Like Cassandra TracingState
, OpenTelemetry Context
will be held in ExecutorLocals
to propagate Context
between threads. Alternatively, Context
can be directly passed to Runnable
s to propagate context between threads.
List of Attribute
s associated in Tracing
The following Attribute
s are associated to Span
s and events. Attribute
names should follow the Attribute Naming rule.
Name | Description | Example value |
---|---|---|
cassandra.query.message_type | Message type of the query (QUERY / EXECUTE / BATCH / PREPARE) | QUERY |
cassandra.query.client.ip | Client IP address | |
cassandra.query.coordinator.ip cassandra.query.coordinator.port | Coordinator node’s broadcast rpc address and native transport port | 10.0.0.1 9042 |
cassandra.query.page_size | 5000 | |
cassandra.query.consistency_level | ||
cassandra.query.serial_consistency_level | ||
cassandra.net.verb | ||
net.peer.ip net.peer.port | ||
thread.name | Name of the thread when the event is recorded | MutationStage-1 |
thread.id | ID of the thread when the event is recorded | 105 |
New or Changed Public Interfaces
New org.apache.cassandra.telemetry.Telemetry
class
- Holds the global
io.opentelemtry.api.OpenTelemetry
instance. - Initialize
OpenTelemetry
instance using OpenTelemetry SDK Autoconfigure whenopentelemetry
configuration incassandra.yaml
hasenabled: true
.- By using
OpenTelemetry SDK Autoconfigure
, users can configure OpenTelemetry exporters through Java system properties and environmental variables.
- By using
- By default, OpenTelemetry is disabled.
- When OpenTelemetry is disabled, No-Op
OpenTelemetry
related instances are created through OpenTelemetry SDK.
- When OpenTelemetry is disabled, No-Op
- Sets up OpenTelemetry tracing, metrics and logs providers.
New cassandra.yaml entry
# Export Apache Cassandra telemetries(tracing, metrics and logs)
# through OpenTelemetry.
#
# Use jvm-server.options file to configure OpenTelemetry SDK with
# system properties.
# <https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/autoconfigure>
# Apache Cassandra only comes with the libraries to export telemetry
# through OLTP.
# You need appropriate jar files if you plan to use other exporters.
opentelemetry:
enabled: false # default: false
Necessary config change
-Xss
needs to be increased (from 256k to 512k) for OLTP export (Okhttp3 based gRPC call) to work
New library dependencies
The following dependencies will be added to the distribution.
OpenTelemetry instrumentation and configuration
- io.opentelemetry:opentelemetry-api
- io.opentelemetry:opentelemetry-sdk
- io.opentelemetry:opentelemetry-sdk-extension-autoconfigure
Runtime dependencies for exporting via OTLP
- io.opentelemetry:opentelemetry-exporter-otlp
Extra dependencies to consider
- io.opentelemetry.instrumentation:opentelemetry-runtime-metrics
- The library to enable JVM metrics instrumentation
- io.opentelemetry:opentelemetry-exporter-otlp-logs
- Necessary if Apache Cassandra decides to adapt unstable OpenTelemetry Log export. The library is separated at the moment, with
-alpha
in its version.
- Necessary if Apache Cassandra decides to adapt unstable OpenTelemetry Log export. The library is separated at the moment, with
- io.opentelemetry.instrumentation::opentelemetry-logback-appender-1.0
- The library to enable exporting logs through Logback appender
Exporting Metrics through OpenTelemetry
TODO
Exporting Logging through OpenTelemetry
TODO
Semantic convention
OpenTelemetry Resource
s are immutable attributes (key-value pairs) that describes the entity producing telemetries, in this case, Apache Cassandra server itself. Resource
s are created once when configuring OpenTelemetry, and those are associated to every telemetries produced by Apache Cassandra.
See Attribute Naming for the specification of naming of Resource
s.
Namespacing
Every Attribute
names (including Resource
names) begin with cassandra
. For example, to describe Cassandra version, the Resource name will be cassandra.version
.
Note that this namespace is used by OpenTelemetry’s JMX metrics gatherer already, however, since the source entity of telemetries is Apache Cassandra, conflicting with JMX metric gatherer shouldn’t be a problem.
List of Resources provided by Apache Cassandra
The following Resource
s are proposed to be configured when OpenTelemetry is enabled. OpenTelemetry standard Resource
s as well as Cassandra specific Resource
s are configured.
Resource name | Value | Description | Example |
---|---|---|---|
service.name | cassandra | • This is required Resource to be set by OpenTelemetry. | |
• It is possible to override this value through https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/autoconfigure#opentelemetry-resource or https://opentelemetry.io/docs/reference/specification/sdk-environment-variables/#general-sdk-configuration. | |||
service.namespace | Apache Cassandra cluster name | Test Cluster | |
service.instance.id | Apache Cassandra Host ID | 058321f5-29b3-4f12-9766-2ad793adb3a0 | |
service.version | Apache Cassandra version in use | 4.2.0 | |
net.host.ip net.host.port | Apache Cassandra node’s endpoint address (listen_address or broadcast_address) and port | 192.168.1.1 and 7000 |
TODO
- Should we use Cassandra specific Resources such as
cassandra.cluster.name
over standard convention likeservice.namespace
?- Using standard may be better to use existing telemetry services.
- Do we need anything else?
Compatibility, Deprecation, and Migration Plan
OpenTelemetry integration adds capability to export existing Tracing/Metrics/Logs through OpenTelemetry Protocol. This feature will not prevent existing telemetry collection. Users should be able to use existing Cassandra tracing, export Metrics through JMX, and output logs to files.
Test Plan
OpenTelemetry Java library provides in-memory telemetry sinks for testing. Use it to verify the exported telemetries.
Rejected Alternatives
- Apache Skywalking provides similar protocol to export observability, however, there seems no APM that supports protocol other than Apache Skywalking itself.