blog

12020-03-30

SEAN K.H. LIAO

Goals

tldr

config: seankhliao/kluster @ v0.19.0

use with:

monitoring

prometheus

prometheus works like magic if you copy the giant kubernetes scrape config from somewhere

annotations to specify what to scrape

apiVersion: apps/v1
kind: Deployment
metadata:
spec:
  template:
    metadata:
      labels:
        app: example
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9000"
        prometheus.io/path: "/metrics"
    spec: ...

promtail

promtail gets all the logs from all the pods, also copy config block from somewhere

loki

loki collects all the logs from promtail, who knows why this needs to be a separate service

grafana

grafana is where all the data ends up as charts

grafana config

[security]
disable_initial_admin_creation = true

[users]
allow_sign_up = false
auto_assign_org = true
auto_assign_org_role = Admin

[auth.proxy]
enabled = true
header_name = X-User-Email
header_property = email
auto_sign_up = true

[analytics]
check_for_updates = false

[log]
mode = console
[log.console]
format = json

[paths]
data = /var/lib/grafana/data
logs = /var/log/grafana
plugins = /var/lib/grafana/plugins
provisioning = /etc/grafana/provisioning

[tracing.jaeger]
address = jaeger-agent:6831

extra routing and middleware because pomerium can't do it

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: grafana
spec:
  entryPoints:
    - https
  routes:
    - kind: Rule
      match: Host(`grafana.api.seankhliao.com`)
      middlewares:
        - name: auth-grafana
        - name: auth-grafana-email
      services:
        - kind: Service
          name: grafana
          namespace: monitor
          port: 80
  tls: {}
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: auth-grafana
spec:
  forwardAuth:
    address: http://pomerium.networking.svc.cluster.local/?uri=https://grafana.api.seankhliao.com
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: auth-grafana-email
spec:
  headers:
    customRequestHeaders:
      X-User-Email: admin@api.seankhliao.com

jaeger

jaeger 1 of 2 competing tracing standards

traefik

tracing:
  jaeger:
    samplingServerURL: "http://jaeger-agent.monitor.svc.cluster.local:5778/sampling"
    localAgentHostPort: "jaeger-agent.monitor.svc.cluster.local:6831"
    gen128Bit: true

grafana

[tracing.jaeger]
address = jaeger-agent:6831

pomerium

tracing_provider: jaeger
tracing_debug: true
tracing_jaeger_agent_endpoint: jaeger-agent.monitor.svc.cluster.local:6831