blog

SEAN K.H. LIAO

otel

opentelemetry, the are making full use of the "no stability guarantee" of semver v0.x.y.

tldr: Go microservices with W3C TraceContext propagation between HTTP services, using OTLP exporters to the collector v0.20.0 and onto jaeger.

test code: [testrepo-otel-v0-16-0]https://github.com/seankhliao/testrepo-otel-v0-16-0)

dependencies

OTel Go has constantly been restructuring the library during beta, and with frequent releases, everything pretty much has to be on the same version to work. Right now, that is v0.16.0

Make sure you prime your editor autocomplete with the right versions of dependencies, or you and it will get very confused.

module go.seankhliao.com/otel-test-v16

go 1.16

require (
        go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.16.0
        go.opentelemetry.io/otel v0.16.0
        go.opentelemetry.io/otel/exporters/otlp v0.16.0
        go.opentelemetry.io/otel/sdk v0.16.0
)

setup

We're writing the code, so we get to use the shiny (always changing) stuff.

exporter

otlp is a push based protocol, so create an exporter and specify an endpoint. Docs: otlp otlpgrpc

exporter, err := otlp.NewExporter(ctx, otlpgrpc.NewDriver(
        otlpgrpc.WithInsecure(),
        otlpgrpc.WithEndpoint("otel-collector.otel.svc.cluster.local:55680"),
))
defer exporter.Shutdown(ctx)
resources

While not strictly necessary, it's a good idea to set some attributes, at the very least ServiceName, or your trace shows up as from OTLPResourceNoServiceName (even if you set a named tracer 🤷). Some standard keys and values are defined in semconv.

Docs: resource semconv

res, err := resource.New(ctx,
        resource.WithAttributes(
                semconv.ServiceNameKey.String("service-a"),
        ),
)
trace provider

What do you do with your configured exporter and resources? You tie them together into a traceprovider.

aside: BatchSpanProcessor ate up a core for each service when I tried to use it, so this example uses the simple one.

Docs: sdktrace

traceProvider := sdktrace.NewTracerProvider(
        sdktrace.WithConfig(sdktrace.Config{
                DefaultSampler: sdktrace.AlwaysSample(),
        }), sdktrace.WithResource(
                res,
        ), sdktrace.WithSpanProcessor(
                sdktrace.NewSimpleSpanProcessor(exporter),
        ),
)
defer traceProvider.Shutdown(ctx)
propagation

Now to choose the (default) format for communicating spans between services, W3C TraceContext is builtin, the rest are in contrib.

Baggage is the ability to pass extra key/values along with the traceid, use it by setting your propagator to:

propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{})

Docs: TraceContext Jaeger Zipkin B3

global

So you could be explicit and get the tracer from the traceprovider each time, but it's much easier to set it globally. Don't ask why it's called textmappropagator, I don't know either

Docs: otel

otel.SetTracerProvider(traceProvider)
otel.SetTextMapPropagator(propagation.TraceContext{})
all together

Here's everything lumped together in a setup function, you probably want to make it configurable.

package main

import (
        "context"
        "fmt"

        "go.opentelemetry.io/otel"
        "go.opentelemetry.io/otel/exporters/otlp"
        "go.opentelemetry.io/otel/exporters/otlp/otlpgrpc"
        "go.opentelemetry.io/otel/propagation"
        "go.opentelemetry.io/otel/sdk/resource"
        sdktrace "go.opentelemetry.io/otel/sdk/trace"
        "go.opentelemetry.io/otel/semconv"
)

func installOtlpPipeline(ctx context.Context) (func(), error) {
        exporter, err := otlp.NewExporter(ctx, otlpgrpc.NewDriver(
                otlpgrpc.WithInsecure(),
                otlpgrpc.WithEndpoint("otel-collector.otel.svc.cluster.local:55680"),
        ))
        if err != nil {
                return nil, fmt.Errorf("otlp setup: create exporter: %w", err)
        }
        res, err := resource.New(ctx,
                resource.WithAttributes(
                        // the service name used to display traces in backends
                        semconv.ServiceNameKey.String("service-a"),
                ),
        )
        if err != nil {
                return nil, fmt.Errorf("otlp setup: create resource: %w", err)
        }

        traceProvider := sdktrace.NewTracerProvider(sdktrace.WithConfig(
                sdktrace.Config{
                        DefaultSampler: sdktrace.AlwaysSample(),
                },
        ), sdktrace.WithResource(
                res,
        ), sdktrace.WithSpanProcessor(
                sdktrace.NewSimpleSpanProcessor(exporter),
        ))
        otel.SetTracerProvider(traceProvider)
        otel.SetTextMapPropagator(propagation.TraceContext{})

        return func() {
                ctx := context.TODO()
                err := traceProvider.Shutdown(ctx)
                if err != nil {
                        otel.Handle(err)
                }
                err = exporter.Shutdown(ctx)
                if err != nil {
                        otel.Handle(err)
                }
        }, nil
}

service

With all the setup out of the way, (finally) time to write your service. Remember propagation? You still need to get/pass the context from/to requests. And who remembers to do Inject/Extract manually? So there's the instrumentation wrappers we can use in contrib.

Docs: contrib/instrumentation otelgrpc interceptors otelhttp wrappers

func main() {
        ctx := context.Background()
        shutdown, _ := installOtlpPipeline(ctx)
        defer shutdown()

        tracer := otel.Tracer("example") // This shows up as an attribute called otlp.instrumentation.library.name

        client := &http.Client{ // custom http client that propagates using the global propagator
                Transport: otelhttp.NewTransport(http.DefaultTransport),
        }

        handler := otelhttp.NewHandler(http.DefaultServeMux, "svc") // shows up as http.server_name

        http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
                span := tracer.SpanFromContext(r.Context()) // no need to start a new context as one is started by the otelhttp handler
                span.AddEvent("eevee event") // shows up as a Log with: message = eevee event

                ctx, span := tracer.Start(r.Context())

                // pass context in request for our propagator
                req, _ := http.NewRequestWithContext(ctx, "GET", "http://example.com", nil)
                res, err := client.Do(req)

                // alternatively use convenience function
                //res, err := otelhttp.Get(ctx, "http://example.com")

                _ = res
        })

        http.ListenAndServe(":8080", handler)
}