...
ID | IEP-[NUMBER] |
Author | Alexander |
Sponsor | |
Created |
|
Status | DRAFT |
Table of Contents |
---|
Tracing provides information useful for debugging that both helps with regular, daily-basic system monitoring and with incidents analysis. Within the scope of Apache Ignite almost every process or sub-system could be traced including:
...
Seems that it makes sense to support an approach with extensibility and configurability of the trace/span handlers through common Ignite Service Provider Interfaces.
And cause Open Census is de-factor facto a standard (or at least one of the most popular solutions right now) it has sence has sense to use it as build in tracing-handler implementation.
As an initial step following interfaces are proposed:
Manager for Span instances.
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
/** * Manager for {@link Span} instances. */ public interface SpanManager { /** * Creates Span with given name. * * @param spanType Type of span to create. */ default Span create(@NotNull SpanType spanType) { return create(spanType, (Span)null); } /** * Creates Span given name and explicit parent. * * @param spanType Type of span to create. * @param parentSpan Parent span. * @return Created span. */ Span create(@NotNull SpanType spanType, @Nullable Span parentSpan); /** * Creates Span given name and explicit parent. * * @param spanType Type of span to create. * @param serializedParentSpan Parent span as serialized bytes. * @return Created span. */ Span create(@NotNull SpanType spanType, @Nullable byte[] serializedParentSpan); /** * Creates Span given name and explicit parent. * * @param spanType Type of span to create. * @param parentSpan Parent span. * @param lb Label. * @return Created span. */ @NotNull Span create ( @NotNull SpanType spanType, @Nullable Span parentSpan, @Nullable String lb); /** * Serializes span to byte array to send context over network. * * @param span Span. */ byte[] serialize(@NotNull Span span); } |
Logical piece of a trace that represents a single operation. Each unit work is called a Span in a trace. Spans include metadata about the work, including the time spent in the step (latency), status, time events, attributes, links.
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
/** * Logical piece of a trace that represents a single operation. * Each unit work is called a Span in a trace. * Spans include metadata about the work, including the time spent in the step (latency), * status, time events, attributes, links. * You can use tracing to debug errors and latency issues in your applications. */ public interface Span { /** * Adds tag to span with {@code String} value. * * @param tagName Tag name. * @param tagVal Tag value. */ Span addTag(String tagName, String tagVal); /** * Adds tag to span with {@code long} value. * * @param tagName Tag name. * @param tagVal Tag value. */ Span addTag(String tagName, long tagVal); /** * Logs work to span. * * @param logDesc Log description. */ Span addLog(String logDesc); /** * Adds log to span with additional attributes. * * @param logDesc Log description. * @param attrs Attributes. */ Span addLog(String logDesc, Map<String, String> attrs); /** * Explicitly set status for span. * * @param spanStatus Status. */ Span setStatus(SpanStatus spanStatus); /** * Ends span. This action sets default status if not set and mark the span as ready to be exported. */ Span end(); /** * @return {@code true} if span has already ended. */ boolean isEnded(); /** * @return Type of given span. |
Allows to configure tracing, read the configuration and restore it to the defaults.
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
/** * Allows to configure tracing, read the configuration and restore it to the defaults. */ public interface TracingConfigurationManager { /** * Set new tracing configuration for the specific tracing coordinates (scope, label, etc.). * If tracing configuration with specified coordinates already exists it'll be overrided, * otherwise new one will be created. * * @param coordinates {@link TracingConfigurationCoordinates} Specific set of locators like {@link Scope} and label, * that defines subset of traces and/or spans that'll use given configuration. * @param parameters {@link TracingConfigurationParameters} e.g. sampling rate, set of included scopes etc. * @throws IgniteException If failed to set tracing configuration. */ void set(@NotNull TracingConfigurationCoordinates coordinates, @NotNull TracingConfigurationParameters parameters) throws IgniteException; /** * Get the most specific tracing parameters for the specified tracing coordinates (scope, label, etc.). * The most specific means: * <ul> * <li> * If there's tracing configuration that matches all tracing configuration attributes (scope and label) — * it'll be returned. * </li> * <li> * If there's no tracing configuration with specified label, or label wasn't specified — * scope specific tracing configuration will be returned. * </li> * <li> * If there's no tracing configuration with specified scope — * default scope specific configuration will be returned. * </li> * </ul> * * @param coordinates {@link TracingConfigurationCoordinates} Specific set of locators like {@link Scope} and label * that defines a subset of traces and/or spans that'll use given configuration. * @return {@link TracingConfigurationParameters} instance. * @throws IgniteException If failed to get tracing configuration. */ default @NotNull TracingConfigurationParameters get( @NotNull TracingConfigurationCoordinates coordinates) throws IgniteException { switch (coordinates.scope()) { case TX: { return DEFAULT_TX_CONFIGURATION; } case EXCHANGE: { return DEFAULT_EXCHANGE_CONFIGURATION; } default: { return NOOP_CONFIGURATION; } } } /** * List all pairs of tracing configuration coordinates and tracing configuration parameters * or list all pairs of tracing configuration and parameters for the specific scope. * * @param scope Nullable scope of tracing configuration to be retrieved. * If null - all configuration will be returned. * @return The whole set of tracing configuration. * @throws IgniteException If failed to get tracing configuration. */ @NotNull Map<TracingConfigurationCoordinates, TracingConfigurationParameters> getAll( @Nullable Scope scope) throws IgniteException; /** * Reset tracing configuration for the specific tracing coordinates (scope, label, etc.) to default values. * Please pay attention, that there's no default values for label specific coordinates, * so such kinds of configurations will be removed. * * @param coordinates {@link TracingConfigurationCoordinates} specific set of locators like {@link Scope} and label * that defines a subset of traces and/or spans that will be reset. * @throws IgniteException If failed to reset tracing configuration. */ void reset(@NotNull TracingConfigurationCoordinates coordinates) throws IgniteException; /** * Reset tracing configuration for the specific scope, or all tracing configurations if scope not specified. * * @param scope {@link Scope} that defines a set of applicable tracing configurations. * @throws IgniteException If failed to reset tracing configuration. */ void resetAll(@Nullable Scope scope) throws IgniteException; } |
Tracing sub-system interface.
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
/** * */ public interface Tracing extends SpanManager { /** * @return Helper to handle traceable messages. */ public TraceableMessagesHandler messages(); /** * Returns the {@link TracingConfigurationManager} instance that allows to * <ul> * <li>Configure tracing parameters such as sampling rate for the specific tracing coordinates * such as scope and label.</li> * <li>Retrieve the most specific tracing parameters for the specified tracing coordinates (scope and label)</li> * <li>Restore the tracing parameters for the specified tracing coordinates to the default.</li> * <li>List all pairs of tracing configuration coordinates and tracing configuration parameters.</li> * </ul> * @return {@link TracingConfigurationManager} instance. */ public @NotNull TracingConfigurationManager configuration(); } |
Despite the fact that it's not an interface it's also extremly also extremely important class that incapsulates encapsulates logic of a thread-local span's storage.
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
public class MTC { /** * @return Span which corresponded to current thread or null if it doesn't not set. */ @NotNull public static Span span() { return span.get(); } /** * Attach given span to current thread if it isn't null or attach empty span if it is null. Detach given span, close * it and return previous span when {@link TraceSurroundings#close()} would be called. * * @param startSpan Span which should be added to current thread. * @return {@link TraceSurroundings} for manage span life cycle. */ public static TraceSurroundings support(Span startSpan) { ... } /** * Support initial span. * * @param startSpan Span which should be added to current thread. */ public static void supportInitial(Span startSpan) { ... } /** * Attach given span to current thread if it isn't null or attach empty span if it is null. * * @param startSpan Span which should be added to current thread. * @return {@link TraceSurroundings} for manage span life cycle. */ public static TraceSurroundings supportContinual(Span startSpan) { ... } } |
...
Below there are few examples of tracing configuration and span creation.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
ignite.tracingConfiguration().set(
new TracingConfigurationCoordinates.Builder(Scope.DISCOVERY).build(),
new TracingConfigurationParameters.Builder().withSamplingRate(SAMPLING_RATE_ALWAYS).build());
ignite.tracingConfiguration().set(
new TracingConfigurationCoordinates.Builder(Scope.TX).withLabel("Some specific label").build(),
new TracingConfigurationParameters.Builder().withSamplingRate(0.1).with
(Collections.singleton(Scope.COMMUNICATION)).build());
|
where
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
Span txRootSpan = cctx.kernalContext().tracing().create(/*Span type */ TX, /*parent span */, null, lb));
Span rootSpan = tracing.create(TraceableMessagesTable.traceName(msg.getClass()))
.addTag(SpanTags.tag(SpanTags.EVENT_NODE, SpanTags.ID), getLocalNodeId().toString())
.addTag(SpanTags.tag(SpanTags.EVENT_NODE, SpanTags.CONSISTENT_ID), locNode.consistentId().toString())
.addTag(SpanTags.MESSAGE_CLASS, ((CustomMessageWrapper)evt).delegate().getClass().getSimpleName())
.addLog("Created");
|
where
It's possible to store span with Mapped Tracing Context or as a common field. First approach is preferred one cause it produces lots of span maintenance stuff out of the box. Here's an example:
Code Block | ||||
---|---|---|---|---|
| ||||
private void proceedMapping() throws IgniteCheckedException {
try (TraceSurroundings ignored =
MTC.support(cctx.kernalContext().tracing().create(SpanType.TX_MAP_PROCEED, MTC.span()))) {
...
}
} |
It seems useful to have an ability to configure tracing configuration via control sh. Following interface is proposed:
Code Block | ||||
---|---|---|---|---|
| ||||
Print tracing configuration:
control.(sh|bat) --tracing-configuration
Print tracing configuration:
control.(sh|bat) --tracing-configuration get_all [--scope DISCOVERY|EXCHANGE|COMMUNICATION|TX]
Print specific tracing configuration based on specified --scope and --label:
control.(sh|bat) --tracing-configuration get (--scope DISCOVERY|EXCHANGE|COMMUNICATION|TX) [--label]
Reset all specific tracing configuration the to default. If --scope is specified, then remove all label specific configuration for the given scope and reset given scope specific configuration to the default, if --scope is skipped then reset all tracing configurations to the default. Print tracing configuration.
control.(sh|bat) --tracing-configuration reset_all [--scope DISCOVERY|EXCHANGE|COMMUNICATION|TX]
Reset specific tracing configuration to the default. If both --scope and --label are specified then remove given configuration, if only --scope is specified then reset given configuration to the default. Print reseted configuration.
control.(sh|bat) --tracing-configuration reset (--scope DISCOVERY|EXCHANGE|COMMUNICATION|TX) [--label]
Set new tracing configuration. If both --scope and --label are specified then add or override label specific configuration, if only --scope is specified, then override scope specific configuration. Print applied configuration.
control.(sh|bat) --tracing-configuration set (--scope DISCOVERY|EXCHANGE|COMMUNICATION|TX [--label] [--sampling-rate Decimal value between 0 and 1, where 0 means never and 1 means always. More or less reflects the probability of sampling specific trace.] [--supported-scopes Set of scopes with comma as separator DISCOVERY|EXCHANGE|COMMUNICATION|TX]) |
1 byte: special flags;
1 bytes: spi type;
2 bytes: major protocol version;
2 bytes: minor protocol version;
4 bytes: spi specific serialized span length;
n bytes: spi specific serialized span body;
4 bytes: span type
4 bytes included scopes size;
2 * included scopes size: included scopes items one by one;
Bellow there's an example of trace generated by pessimistic serializable transaction.
// Links to discussions on the devlist, if applicable.
...