The recently released Grafana 7.4.0 includes support for jumping between logs and traces (only in the explore tab).
your apps need to: include traceids in logs and include labels in spans (can probably be done with otel-collector...)
Get logs into grafana: promtail to scrape the logs, loki to aggregate / store. Add Loki as a datasource to grafana, add a derived field extracting the traceid and linking to jaeger.
Get traces into grafana: jaeger to collect the logs. Add jaeger as a datasource to grafana, select loki for trace to logs, add extra tags if you have them (to make filtering more specific).
Go to the explore tab.
Currently you can only retrieve a trace if you know the traceID or you can select from a few in the query selector. No searching (yet). Trace to log is available as a little button next to the span name.
Crash course LogQL (see docs for details):
{app="svc"} |= "info" | logfmt | x > 5 | label_format ... | line_format ...
{app="svc"}
: select logs using labels|=
: initial filter over raw log line (contains, regex)logfmt
: parse into structured data, also supports json
, regex
x > 5
: query structured datalabel_format
: modify labelsline_format
: output as raw log lineLink to trace is available once you click open a log line. A button will show next to to the trace field (configured in the datasource).