SEANK.H.LIAO

kubernetes cluster 19 part 3

19th attempt to get a k8s setup working

Goals

tldr

config: seankhliao/kluster @ v0.19.0

use with:

monitoring

prometheus

prometheus works like magic if you copy the giant kubernetes scrape config from somewhere

annotations to specify what to scrape

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4spec:
 5  template:
 6    metadata:
 7      labels:
 8        app: example
 9      annotations:
10        prometheus.io/scrape: "true"
11        prometheus.io/port: "9000"
12        prometheus.io/path: "/metrics"
13    spec: ...

promtail

promtail gets all the logs from all the pods, also copy config block from somewhere

loki

loki collects all the logs from promtail, who knows why this needs to be a separate service

grafana

grafana is where all the data ends up as charts

grafana config

 1[security]
 2disable_initial_admin_creation = true
 3
 4[users]
 5allow_sign_up = false
 6auto_assign_org = true
 7auto_assign_org_role = Admin
 8
 9[auth.proxy]
10enabled = true
11header_name = X-User-Email
12header_property = email
13auto_sign_up = true
14
15[analytics]
16check_for_updates = false
17
18[log]
19mode = console
20[log.console]
21format = json
22
23[paths]
24data = /var/lib/grafana/data
25logs = /var/log/grafana
26plugins = /var/lib/grafana/plugins
27provisioning = /etc/grafana/provisioning
28
29[tracing.jaeger]
30address = jaeger-agent:6831

extra routing and middleware because pomerium can't do it

 1apiVersion: traefik.containo.us/v1alpha1
 2kind: IngressRoute
 3metadata:
 4  name: grafana
 5spec:
 6  entryPoints:
 7    - https
 8  routes:
 9    - kind: Rule
10      match: Host(`grafana.api.seankhliao.com`)
11      middlewares:
12        - name: auth-grafana
13        - name: auth-grafana-email
14      services:
15        - kind: Service
16          name: grafana
17          namespace: monitor
18          port: 80
19  tls: {}
20---
21apiVersion: traefik.containo.us/v1alpha1
22kind: Middleware
23metadata:
24  name: auth-grafana
25spec:
26  forwardAuth:
27    address: http://pomerium.networking.svc.cluster.local/?uri=https://grafana.api.seankhliao.com
28---
29apiVersion: traefik.containo.us/v1alpha1
30kind: Middleware
31metadata:
32  name: auth-grafana-email
33spec:
34  headers:
35    customRequestHeaders:
36      X-User-Email: admin@api.seankhliao.com

jaeger

jaeger 1 of 2 competing tracing standards

traefik

1tracing:
2  jaeger:
3    samplingServerURL: "http://jaeger-agent.monitor.svc.cluster.local:5778/sampling"
4    localAgentHostPort: "jaeger-agent.monitor.svc.cluster.local:6831"
5    gen128Bit: true

grafana

1[tracing.jaeger]
2address = jaeger-agent:6831

pomerium

1tracing_provider: jaeger
2tracing_debug: true
3tracing_jaeger_agent_endpoint: jaeger-agent.monitor.svc.cluster.local:6831