SEANK.H.LIAO

grafana trace to log to trace

connecting traces and logs in grafana

grafana

The recently released Grafana 7.4.0 includes support for jumping between logs and traces (only in the explore tab).

apps

your apps need to: include traceids in logs and include labels in spans (can probably be done with otel-collector...)

setup

Get logs into grafana: promtail to scrape the logs, loki to aggregate / store. Add Loki as a datasource to grafana, add a derived field extracting the traceid and linking to jaeger.

Get traces into grafana: jaeger to collect the logs. Add jaeger as a datasource to grafana, select loki for trace to logs, add extra tags if you have them (to make filtering more specific).

play

Go to the explore tab.

traces

Currently you can only retrieve a trace if you know the traceID or you can select from a few in the query selector. No searching (yet). Trace to log is available as a little button next to the span name.

logs

Crash course LogQL (see docs for details):

{app="svc"} |= "info" | logfmt | x > 5 | label_format ... | line_format ...

Link to trace is available once you click open a log line. A button will show next to to the trace field (configured in the datasource).