opentelemetry,
the are making full use of the "no stability guarantee" of semver v0.x.y.
tldr: Go microservices with W3C TraceContext propagation between HTTP services, using OTLP exporters to the collector v0.20.0 and onto jaeger.
test code: [testrepo-otel-v0-16-0]https://github.com/seankhliao/testrepo-otel-v0-16-0)
OTel Go has constantly been restructuring the library during beta,
and with frequent releases, everything pretty much has to be on the same version to work.
Right now, that is v0.16.0
Make sure you prime your editor autocomplete with the right versions of dependencies, or you and it will get very confused.
module go.seankhliao.com/otel-test-v16
go 1.16
require (
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.16.0
go.opentelemetry.io/otel v0.16.0
go.opentelemetry.io/otel/exporters/otlp v0.16.0
go.opentelemetry.io/otel/sdk v0.16.0
)
We're writing the code, so we get to use the shiny (always changing) stuff.
otlp
is a push based protocol, so create an exporter and specify an endpoint. Docs:
otlp
otlpgrpc
1exporter, err := otlp.NewExporter(ctx, otlpgrpc.NewDriver(
2 otlpgrpc.WithInsecure(),
3 otlpgrpc.WithEndpoint("otel-collector.otel.svc.cluster.local:55680"),
4))
5defer exporter.Shutdown(ctx)
While not strictly necessary, it's a good idea to set some attributes, at the very least ServiceName,
or your trace shows up as from OTLPResourceNoServiceName (even if you set a named tracer 🤷).
Some standard keys and values are defined in semconv.
1res, err := resource.New(ctx,
2 resource.WithAttributes(
3 semconv.ServiceNameKey.String("service-a"),
4 ),
5)
What do you do with your configured exporter and resources? You tie them together into a traceprovider.
aside: BatchSpanProcessor ate up a core for each service when I tried to use it,
so this example uses the simple one.
Docs: sdktrace
1traceProvider := sdktrace.NewTracerProvider(
2 sdktrace.WithConfig(sdktrace.Config{
3 DefaultSampler: sdktrace.AlwaysSample(),
4 }), sdktrace.WithResource(
5 res,
6 ), sdktrace.WithSpanProcessor(
7 sdktrace.NewSimpleSpanProcessor(exporter),
8 ),
9)
10defer traceProvider.Shutdown(ctx)
Now to choose the (default) format for communicating spans between services, W3C TraceContext is builtin, the rest are in contrib.
Baggage is the ability to pass extra key/values along with the traceid, use it by setting your propagator to:
1propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{})
Docs: TraceContext Jaeger Zipkin B3
So you could be explicit and get the tracer from the traceprovider each time, but it's much easier to set it globally. Don't ask why it's called textmappropagator, I don't know either
Docs: otel
1otel.SetTracerProvider(traceProvider)
2otel.SetTextMapPropagator(propagation.TraceContext{})
Here's everything lumped together in a setup function, you probably want to make it configurable.
1package main
2
3import (
4 "context"
5 "fmt"
6
7 "go.opentelemetry.io/otel"
8 "go.opentelemetry.io/otel/exporters/otlp"
9 "go.opentelemetry.io/otel/exporters/otlp/otlpgrpc"
10 "go.opentelemetry.io/otel/propagation"
11 "go.opentelemetry.io/otel/sdk/resource"
12 sdktrace "go.opentelemetry.io/otel/sdk/trace"
13 "go.opentelemetry.io/otel/semconv"
14)
15
16func installOtlpPipeline(ctx context.Context) (func(), error) {
17 exporter, err := otlp.NewExporter(ctx, otlpgrpc.NewDriver(
18 otlpgrpc.WithInsecure(),
19 otlpgrpc.WithEndpoint("otel-collector.otel.svc.cluster.local:55680"),
20 ))
21 if err != nil {
22 return nil, fmt.Errorf("otlp setup: create exporter: %w", err)
23 }
24 res, err := resource.New(ctx,
25 resource.WithAttributes(
26 // the service name used to display traces in backends
27 semconv.ServiceNameKey.String("service-a"),
28 ),
29 )
30 if err != nil {
31 return nil, fmt.Errorf("otlp setup: create resource: %w", err)
32 }
33
34 traceProvider := sdktrace.NewTracerProvider(sdktrace.WithConfig(
35 sdktrace.Config{
36 DefaultSampler: sdktrace.AlwaysSample(),
37 },
38 ), sdktrace.WithResource(
39 res,
40 ), sdktrace.WithSpanProcessor(
41 sdktrace.NewSimpleSpanProcessor(exporter),
42 ))
43 otel.SetTracerProvider(traceProvider)
44 otel.SetTextMapPropagator(propagation.TraceContext{})
45
46 return func() {
47 ctx := context.TODO()
48 err := traceProvider.Shutdown(ctx)
49 if err != nil {
50 otel.Handle(err)
51 }
52 err = exporter.Shutdown(ctx)
53 if err != nil {
54 otel.Handle(err)
55 }
56 }, nil
57}
With all the setup out of the way, (finally) time to write your service. Remember propagation? You still need to get/pass the context from/to requests. And who remembers to do Inject/Extract manually? So there's the instrumentation wrappers we can use in contrib.
Docs: contrib/instrumentation otelgrpc interceptors otelhttp wrappers
1func main() {
2 ctx := context.Background()
3 shutdown, _ := installOtlpPipeline(ctx)
4 defer shutdown()
5
6 tracer := otel.Tracer("example") // This shows up as an attribute called otlp.instrumentation.library.name
7
8 client := &http.Client{ // custom http client that propagates using the global propagator
9 Transport: otelhttp.NewTransport(http.DefaultTransport),
10 }
11
12 handler := otelhttp.NewHandler(http.DefaultServeMux, "svc") // shows up as http.server_name
13
14 http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
15 span := tracer.SpanFromContext(r.Context()) // no need to start a new context as one is started by the otelhttp handler
16 span.AddEvent("eevee event") // shows up as a Log with: message = eevee event
17
18 ctx, span := tracer.Start(r.Context())
19
20 // pass context in request for our propagator
21 req, _ := http.NewRequestWithContext(ctx, "GET", "http://example.com", nil)
22 res, err := client.Do(req)
23
24 // alternatively use convenience function
25 //res, err := otelhttp.Get(ctx, "http://example.com")
26
27 _ = res
28 })
29
30 http.ListenAndServe(":8080", handler)
31}