So you want to run a high availability service, but your code can only really handle having a single instance do the critical work. What do you do? You run multiple replicas, choosing one to be the leader to do the critical sections.
Now you could implement raft, paxos, or something else yourself, but if you're running in kubernetes, there's a ready made Lease resource, and leaderelection libraries to use.
We need a service account to attach permissions to
1apiVersion: v1
2kind: ServiceAccount
3metadata:
4 name: test-leader
The leaderelection library will create and update the Leases on demand
1apiVersion: rbac.authorization.k8s.io/v1
2kind: Role
3metadata:
4 name: test-leader
5rules:
6 - apiGroups:
7 - coordination.k8s.io
8 resources:
9 - leases
10 verbs:
11 - "*"
12---
13apiVersion: rbac.authorization.k8s.io/v1
14kind: RoleBinding
15metadata:
16 name: test-leader
17roleRef:
18 apiGroup: rbac.authorization.k8s.io
19 kind: Role
20 name: test-leader
21subjects:
22 - kind: ServiceAccount
23 name: test-leader
We need to run our pods.
3 replicas so we actually see the effects of leader election.
We inject POD_NAME as a unique identity
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 name: test-leader
5spec:
6 replicas: 3
7 selector:
8 matchLabels:
9 app.kubernetes.io/name: test-leader
10 template:
11 metadata:
12 labels:
13 app.kubernetes.io/name: test-leader
14 spec:
15 serviceAccountName: test-leader
16 containers:
17 - name: leader
18 image: go.seankhliao.com/test-leaderelection/cmd/leader
Now for actual code:
1package main
2
3import (
4 "context"
5 "os"
6 "time"
7
8 metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
9 "k8s.io/client-go/kubernetes"
10 "k8s.io/client-go/rest"
11 "k8s.io/client-go/tools/leaderelection"
12 "k8s.io/client-go/tools/leaderelection/resourcelock"
13 "k8s.io/klog/v2"
14)
15
16func main() {
17 // setup client to talk to api server using service account credentials
18 restConfig, err := rest.InClusterConfig()
19 if err != nil {
20 klog.ErrorS(err, "get rest config")
21 os.Exit(1)
22 }
23 client, err := kubernetes.NewForConfig(restConfig)
24 if err != nil {
25 klog.ErrorS(err, "get rest client")
26 os.Exit(1)
27 }
28
29 // get a unique identity for ourselves
30 hostname, err := os.Hostname()
31 if err != nil {
32 klog.ErrorS(err, "get hostname")
33 os.Exit(1)
34 }
35
36 ctx := context.Background()
37 ctx, cancel := context.WithCancel(ctx)
38
39
40 // runs in a leader election loop
41 // panics on failure
42 leaderelection.RunOrDie(ctx, leaderelection.LeaderElectionConfig{
43 // this is the lease we create/update if we win the leader
44 Lock: &resourcelock.LeaseLock{
45 LeaseMeta: metav1.ObjectMeta{
46 Namespace: "default",
47 Name: "test-lease",
48 },
49 Client: client.CoordinationV1(),
50 LockConfig: resourcelock.ResourceLockConfig{
51 Identity: hostname,
52 },
53 },
54 // recommended defaults
55 LeaseDuration: 15 * time.Second,
56 RenewDeadline: 10 * time.Second,
57 RetryPeriod: 2 * time.Second,
58 // TODO, ensure exit from critical work before canceling context
59 ReleaseOnCancel: true,
60 Callbacks: leaderelection.LeaderCallbacks{
61 // main work should happen here
62 OnStartedLeading: func(ctx context.Context) {
63 dur := 5 * time.Second
64 klog.InfoS("leading", "tick_dur", dur)
65 tick := time.NewTicker(dur)
66 defer tick.Stop()
67 leadLoop:
68 for {
69 select {
70 case <-tick.C:
71 klog.InfoS("still leading")
72 case <-ctx.Done():
73 klog.InfoS("leading cancelled")
74 break leadLoop
75 }
76 }
77 },
78 OnStoppedLeading: func() {
79 // TODO: ensure work loop exit before canceling leaderelection ctx
80 cancel()
81 klog.InfoS("stopped leading")
82 },
83 OnNewLeader: func(identity string) {
84 // just notifications
85 klog.InfoS("new leader", "id", identity)
86 },
87 },
88 })
89}
And a simple skaffold config to deploy into a kind cluster
1apiVersion: skaffold/v4beta1
2kind: Config
3metadata:
4 name: test-leader
5build:
6 artifacts:
7 - image: go.seankhliao.com/test-leaderelection/cmd/leader
8 ko:
9 main: ./cmd/leader
10 local:
11 push: false
12 tagPolicy:
13 sha256: {}
14manifests:
15 rawYaml:
16 - deploy/manifests/*
17deploy:
18 kubectl: {}
19 kubeContext: kind-kind
20 logs:
21 prefix: podAndContainer
1$ skaffold run --tail
2Generating tags...
3 - go.seankhliao.com/test-leaderelection/cmd/leader -> go.seankhliao.com/test-leaderelection/cmd/leader:latest
4Checking cache...
5 - go.seankhliao.com/test-leaderelection/cmd/leader: Not found. Building
6Starting build...
7Found [kind-kind] context, using local docker daemon.
8Building [go.seankhliao.com/test-leaderelection/cmd/leader]...
9Target platforms: [linux/amd64]
10Using base gcr.io/distroless/static:nonroot@sha256:bce851161342b0c9d19d0d56e3d37b4787cc6b53a164ec21432e0e1755d08e17 for go.seankhliao.com/test-leaderelection/cmd/leader
11Using build config go.seankhliao.com/test-leaderelection/cmd/leader for go.seankhliao.com/test-leaderelection/cmd/leader
12Building go.seankhliao.com/test-leaderelection/cmd/leader for linux/amd64
13Loading go.seankhliao.com/test-leaderelection/cmd/leader:266dddc70cc019db6657b05dd884b16d59882f98947c1e2833dc5274cca5ecac
14Loaded go.seankhliao.com/test-leaderelection/cmd/leader:266dddc70cc019db6657b05dd884b16d59882f98947c1e2833dc5274cca5ecac
15Adding tag latest
16Added tag latest
17Build [go.seankhliao.com/test-leaderelection/cmd/leader] succeeded
18Starting test...
19Tags used in deployment:
20 - go.seankhliao.com/test-leaderelection/cmd/leader -> go.seankhliao.com/test-leaderelection/cmd/leader:5963bd657b4fabc4e4dcb69b57ef08569e25a3b8df132c8c67339801cac25d9c
21Starting deploy...
22Loading images into kind cluster nodes...
23 - go.seankhliao.com/test-leaderelection/cmd/leader:5963bd657b4fabc4e4dcb69b57ef08569e25a3b8df132c8c67339801cac25d9c -> Loaded
24Images loaded in 1.455 second
25 - deployment.apps/test-leader configured
26 - role.rbac.authorization.k8s.io/test-leader configured
27 - rolebinding.rbac.authorization.k8s.io/test-leader unchanged
28 - serviceaccount/test-leader unchanged
29Waiting for deployments to stabilize...
30 - deployment/test-leader is ready.
31Deployments stabilized in 3.068 seconds
32Press Ctrl+C to exit
33[test-leader-6d7766bbcc-8sddv leader] I0107 12:50:38.714950 1 leaderelection.go:248] attempting to acquire leader lease default/test-lease...
34[test-leader-6d7766bbcc-8sddv leader] I0107 12:50:38.718334 1 main.go:76] "new leader" id="test-leader-7d76b8f7df-7m2x8"
35[test-leader-6d7766bbcc-pdp5j leader] I0107 12:50:37.666586 1 leaderelection.go:248] attempting to acquire leader lease default/test-lease...
36[test-leader-6d7766bbcc-pdp5j leader] I0107 12:50:37.671308 1 main.go:76] "new leader" id="test-leader-7d76b8f7df-7m2x8"
37[test-leader-6d7766bbcc-r7qq8 leader] I0107 12:50:36.889879 1 leaderelection.go:248] attempting to acquire leader lease default/test-lease...
38[test-leader-6d7766bbcc-r7qq8 leader] I0107 12:50:36.894878 1 main.go:76] "new leader" id="test-leader-7d76b8f7df-7m2x8"
39[test-leader-6d7766bbcc-pdp5j leader] I0107 12:50:52.958867 1 leaderelection.go:258] successfully acquired lease default/test-lease
40[test-leader-6d7766bbcc-pdp5j leader] I0107 12:50:52.958957 1 main.go:76] "new leader" id="test-leader-6d7766bbcc-pdp5j"
41[test-leader-6d7766bbcc-pdp5j leader] I0107 12:50:52.958985 1 main.go:57] "leading" tick_dur="5s"
42[test-leader-6d7766bbcc-8sddv leader] I0107 12:50:54.270478 1 main.go:76] "new leader" id="test-leader-6d7766bbcc-pdp5j"
43[test-leader-6d7766bbcc-r7qq8 leader] I0107 12:50:54.749086 1 main.go:76] "new leader" id="test-leader-6d7766bbcc-pdp5j"
44[test-leader-6d7766bbcc-pdp5j leader] I0107 12:50:57.959756 1 main.go:64] "still leading"
45[test-leader-6d7766bbcc-pdp5j leader] I0107 12:51:02.960722 1 main.go:64] "still leading"
46[test-leader-6d7766bbcc-pdp5j leader] I0107 12:51:07.959616 1 main.go:64] "still leading"
47[test-leader-6d7766bbcc-pdp5j leader] I0107 12:51:12.959551 1 main.go:64] "still leading"
48[test-leader-6d7766bbcc-pdp5j leader] I0107 12:51:17.959866 1 main.go:64] "still leading"