skaffold kaniko docker caching

the convoluted setup i go through to get proper caching

skaffold

skaffold is a great dev tool to detect code changes and manage the build & deploy phases. I use it with kaniko to build images in my cluster. Unfortunately, this comes with a set of problems: caching.

baseline

Here's our starting setup, using a random project I had lying around. A standard multistage dockerfile and skaffold pipeline.

 1FROM golang:alpine AS build
 2ENV CGO_ENABLED=0
 3WORKDIR /workspace
 4COPY . .
 5RUN go build -trimpath -ldflags='-s -w' -o /bin/feed-agg
 6
 7FROM gcr.io/distroless/static
 8COPY conf.yaml /etc/feed-agg/conf.yaml
 9COPY --from=build /bin/feed-agg /bin/feed-agg
10ENTRYPOINT ["/bin/feed-agg"]

 1apiVersion: skaffold/v2beta16
 2kind: Config
 3metadata:
 4  name: feed-agg
 5build:
 6  artifacts:
 7    - image: europe-north1-docker.pkg.dev/com-seankhliao/kluster/feed-agg
 8      kaniko:
 9        reproducible: true
10        singleSnapshot: true
11        skipUnusedStages: true
12        useNewRun: true
13        image: gcr.io/kaniko-project/executor:latest
14  cluster:
15    pullSecretName: kaniko-secret
16    pullSecretPath: kaniko-secret
17    namespace: skaffold
18deploy:
19  kubeContext: kind-cluster30
20  kustomize:
21    paths:
22      - kustomize/overlays/cluster30

There are a few problems with this:

it downloads dependencies from a remote proxy every time
it rebuilds everything from scratch every time

I could split out COPY go.mod go.sum ./ and RUN go mod download and use docker's layering to cache the dependency download, but that's meh: it changes every time any dependency changes and bloats the image repo.

private proxy

Instead of downloading from som remote proxy, we could instead use a local-ish one. I'm running athens, and we can conditionally pass in the proxy using build args, preserving the defaults when we build elsewhere.

note: ARG needs to be specified after FROM to take effect inside the image.

This is only very marginally faster than using the public proxy but does solve one other problem: private dependencies. This way no credentials need to be passed to the build image, since access is implicit.

1FROM golang:alpine AS build
2ARG GOPROXY=https://proxy.golang.org,direct
3...

 1apiVersion: skaffold/v2beta16
 2kind: Config
 3metadata:
 4  name: feed-agg
 5build:
 6  artifacts:
 7  - image: europe-north1-docker.pkg.dev/com-seankhliao/kluster/feed-agg
 8    kaniko:
 9      ...
10      buildArgs:
11        GOPROXY: http://athens.athens.svc.cluster.local

mounted caches

To really speed things up, we need to make use of go's native caching, namely its local filesystem build and module caches. We can use kaniko's --whitelist-var-run, excluding it from the build image to mount the caches.

note: I'm using hostPath because I'm running a single node using kind and couldn't be bothered to set a CSI provider that could provision ReadWriteMany persistent volumes.

1FROM golang:alpine AS build
2ARG GOMODCACHE=/go/pkg/mod
3ARG GOCACHE=/root/.cache/go-build
4...

 1apiVersion: skaffold/v2beta16
 2kind: Config
 3metadata:
 4  name: feed-agg
 5build:
 6  artifacts:
 7  - image: europe-north1-docker.pkg.dev/com-seankhliao/kluster/feed-agg
 8    kaniko:
 9      ...
10      buildArgs:
11        GOCACHE: /var/run/gobuildcache
12        GOMODCACHE: /var/run/gomodcache
13      volumeMounts:
14        - name: modcache
15          mountPath: /var/run/gomodcache
16        - name: buildcache
17          mountPath: /var/run/gobuildcache
18  cluster:
19    ...
20    volumes:
21      - name: modcache
22        hostPath:
23          path: /opt/kind/cluster30/kaniko-gomodcache
24      - name: buildcache
25        hostPath:
26          path: /opt/kind/cluster30/kaniko-gobuildcache

final setup

So here's everything, taking an initial build from 149sec down to 25sec.

 1FROM golang:alpine AS build
 2ARG CGO_ENABLED=0
 3ARG GOPROXY=https://proxy.golang.org,direct
 4ARG GOMODCACHE=/go/pkg/mod
 5ARG GOCACHE=/root/.cache/go-build
 6WORKDIR /workspace
 7COPY . .
 8RUN go build -trimpath -ldflags='-s -w' -o /bin/feed-agg
 9
10FROM gcr.io/distroless/static
11COPY conf.yaml /etc/feed-agg/conf.yaml
12COPY --from=build /bin/feed-agg /bin/feed-agg
13ENTRYPOINT ["/bin/feed-agg"]

 1apiVersion: skaffold/v2beta16
 2kind: Config
 3metadata:
 4  name: feed-agg
 5build:
 6  artifacts:
 7    - image: europe-north1-docker.pkg.dev/com-seankhliao/kluster/feed-agg
 8      kaniko:
 9        reproducible: true
10        singleSnapshot: true
11        skipUnusedStages: true
12        useNewRun: true
13        whitelistVarRun: true
14        image: gcr.io/kaniko-project/executor:latest
15        registryMirror: mirror.gcr.io
16        buildArgs:
17          GOPROXY: http://athens.athens.svc.cluster.local
18          GOCACHE: /var/run/gobuildcache
19          GOMODCACHE: /var/run/gomodcache
20        volumeMounts:
21          - name: modcache
22            mountPath: /var/run/gomodcache
23          - name: buildcache
24            mountPath: /var/run/gobuildcache
25  cluster:
26    pullSecretName: kaniko-secret
27    pullSecretPath: kaniko-secret
28    namespace: skaffold
29    volumes:
30      - name: modcache
31        hostPath:
32          path: /opt/kind/cluster30/kaniko-gomodcache
33      - name: buildcache
34        hostPath:
35          path: /opt/kind/cluster30/kaniko-gobuildcache
36deploy:
37  kubeContext: kind-cluster30
38  kustomize:
39    paths:
40      - kustomize/overlays/cluster30