kubernetes cluster 19 part 3

19th attempt to get a k8s setup working

Goals

• monitoring

tldr

config: seankhliao/kluster @ v0.19.0

use with:

• make create-cluster
• make decrypt // or create appropriate secret files
• kubectl apply -k . // maybe repeat a few times if things fail

monitoring

• prometheus metrics
• promtail + loki logs
• grafana visualization
• jaeger tracing

prometheus

prometheus works like magic if you copy the giant kubernetes scrape config from somewhere

• node-expoter: stats about the k8s node (host machine)
• kube-state-metrics: stats about k8s runtime

annotations to specify what to scrape

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4spec:
 5  template:
 6    metadata:
 7      labels:
 8        app: example
 9      annotations:
10        prometheus.io/scrape: "true"
11        prometheus.io/port: "9000"
12        prometheus.io/path: "/metrics"
13    spec: ...

promtail

promtail gets all the logs from all the pods, also copy config block from somewhere

loki

loki collects all the logs from promtail, who knows why this needs to be a separate service

grafana

grafana is where all the data ends up as charts

• grafana has its own user system
• TODO: create charts

grafana config

• trust header to create user
• provisioning

 1[security]
 2disable_initial_admin_creation = true
 3
 4[users]
 5allow_sign_up = false
 6auto_assign_org = true
 7auto_assign_org_role = Admin
 8
 9[auth.proxy]
10enabled = true
11header_name = X-User-Email
12header_property = email
13auto_sign_up = true
14
15[analytics]
16check_for_updates = false
17
18[log]
19mode = console
20[log.console]
21format = json
22
23[paths]
24data = /var/lib/grafana/data
25logs = /var/log/grafana
26plugins = /var/lib/grafana/plugins
27provisioning = /etc/grafana/provisioning
28
29[tracing.jaeger]
30address = jaeger-agent:6831

extra routing and middleware because pomerium can't do it

 1apiVersion: traefik.containo.us/v1alpha1
 2kind: IngressRoute
 3metadata:
 4  name: grafana
 5spec:
 6  entryPoints:
 7    - https
 8  routes:
 9    - kind: Rule
10      match: Host(`grafana.api.seankhliao.com`)
11      middlewares:
12        - name: auth-grafana
13        - name: auth-grafana-email
14      services:
15        - kind: Service
16          name: grafana
17          namespace: monitor
18          port: 80
19  tls: {}
20---
21apiVersion: traefik.containo.us/v1alpha1
22kind: Middleware
23metadata:
24  name: auth-grafana
25spec:
26  forwardAuth:
27    address: http://pomerium.networking.svc.cluster.local/?uri=https://grafana.api.seankhliao.com
28---
29apiVersion: traefik.containo.us/v1alpha1
30kind: Middleware
31metadata:
32  name: auth-grafana-email
33spec:
34  headers:
35    customRequestHeaders:
36      X-User-Email: admin@api.seankhliao.com

jaeger

jaeger 1 of 2 competing tracing standards

• other is Zipkin
• OpenTelemetry merges OpenTracing and OpenCensus
• run in all in one mode: who has time for a database
• app -> jaeger-agent -> jaeger-collector -> jaeger-query

traefik

1tracing:
2  jaeger:
3    samplingServerURL: "http://jaeger-agent.monitor.svc.cluster.local:5778/sampling"
4    localAgentHostPort: "jaeger-agent.monitor.svc.cluster.local:6831"
5    gen128Bit: true

grafana

1[tracing.jaeger]
2address = jaeger-agent:6831

pomerium

• only works internally (doesn't coordinate with other services)
• has the most useless name (all) in all in one mode

1tracing_provider: jaeger
2tracing_debug: true
3tracing_jaeger_agent_endpoint: jaeger-agent.monitor.svc.cluster.local:6831