Overview

2 minute read

Distributed tracing enables users to track a request through mesh that is distributed across multiple services. This allows a deeper understanding about request latency, serialization and parallelism via visualization.

Istio leverages Envoy’s distributed tracing feature to provide tracing integration out of the box.

Most tracing backends now accept OpenTelemetry protocol to receive traces, though Istio also supports legacy protocols for projects like Zipkin and Apache SkyWalking.

Configuring tracing

Istio provides a Telemetry API which can be used to configure distributed tracing, including selecting a provider, setting sampling rate and header modification.

Extension providers

Extension providers are defined in MeshConfig, and allow defining the configuration for a trace backend. Supported providers are OpenTelemetry, Zipkin, SkyWalking, Datadog and Stackdriver.

Building applications to support trace context propagation

Although Istio proxies can automatically send spans, extra information is needed to join those spans into a single trace. Applications must propagate this information in HTTP headers, so that when proxies send spans, the backend can join them together into a single trace.

To do this, each application must collect headers from each incoming request and forward the headers to all outgoing requests triggered by that incoming request. The choice of headers to forward depends on the configured trace backend. The set of headers to forward are described in each trace backend-specific task page. The following is a summary:

All applications should forward the following headers:

x-request-id: an Envoy-specific header that is used to consistently sample logs and traces.
traceparent and tracestate: W3C standard headers

For Zipkin, the B3 multi-header format should be forwarded:

x-b3-traceid
x-b3-spanid
x-b3-parentspanid
x-b3-sampled
x-b3-flags

For commercial observability tools, refer to their documentation.

If you look at the sample Python productpage service, for example, you see that the application extracts the required headers for all tracers from an HTTP request using OpenTelemetry libraries:

def getForwardHeaders(request):
    headers = {}

    # x-b3-*** headers can be populated using the OpenTelemetry span
    ctx = propagator.extract(carrier={k.lower(): v for k, v in request.headers})
    propagator.inject(headers, ctx)

    # ...

        incoming_headers = ['x-request-id',
        'x-ot-span-context',
        'x-datadog-trace-id',
        'x-datadog-parent-id',
        'x-datadog-sampling-priority',
        'traceparent',
        'tracestate',
        'x-cloud-trace-context',
        'grpc-trace-bin',
        'user-agent',
        'cookie',
        'authorization',
        'jwt',
    ]

    # ...

    for ihdr in incoming_headers:
        val = request.headers.get(ihdr)
        if val is not None:
            headers[ihdr] = val

    return headers

The reviews application (Java) does something similar using requestHeaders:

@GET
@Path("/reviews/{productId}")
public Response bookReviewsById(@PathParam("productId") int productId, @Context HttpHeaders requestHeaders) {

  // ...

  if (ratings_enabled) {
    JsonObject ratingsResponse = getRatings(Integer.toString(productId), requestHeaders);

When you make downstream calls in your applications, make sure to include these headers.