benchmark kaniko go

building go apps in containers faster

kaniko

kaniko a container builder from google, intended to be run as a container. kaniko wants to be root but does not need privileges, different from others.

docker with buildkit was not tested because clean builds with both --cache-from and --build-arg BUILDKIT_INLINE_CACHE=1 specified don't work.

results

tldr: multistage builds with vendored deps have the fasted build times, produce the slimmest images but at the cost of bloating up the source code repo with checked in deps.

--use-new-run did not appear to have appreciable impact for multi-vendored.

build fails were kaniko complaining about unlinkat / device busy, did not appear to be transient failures.

1error building image: error building stage: failed to execute command: extracting fs from image: removing whiteout .wh.workspace: unlinkat //workspace: device or resource busy

setup

Repo: GitHub CI: Google Cloud Build, triggered by app

Dockerfile

Dockerfiles for the different tests, uncomment as necessary

All tests used tidied go.mod/go.sum. The vendored tests used the same setup as basic but with a checked in vendor/ directory.

 1FROM golang:alpine AS build
 2
 3WORKDIR /workspace
 4RUN apk add --update --no-cache ca-certificates
 5
 6# go.mod/sum + go mod download
 7# COPY go.mod .
 8# COPY go.sum .
 9# RUN go mod download
10
11# go.mod/sum + go mod vendor
12# COPY go.mod .
13# COPY go.sum .
14# RUN go mod download
15
16COPY . .
17RUN CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /bin/app
18
19# multistage
20# FROM scratch AS app
21# COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
22# COPY --from=build /bin/app /bin/
23
24ENTRYPOINT [ "/bin/app" ]

cloudbuild.yaml

cloudbuild config for testing. only the _IMG variable differed, also --use-new-run for the test.

 1substitutions:
 2  _IMG: test-multi-basic
 3  _REG: us.gcr.io
 4tags:
 5  - $SHORT_SHA
 6  - $COMMIT_SHA
 7steps:
 8  - id: build-push
 9    name: gcr.io/kaniko-project/executor:latest
10    args:
11      - -c=.
12      - -f=Dockerfile
13      - -d=$_REG/$PROJECT_ID/$_IMG:latest
14      - -d=$_REG/$PROJECT_ID/$_IMG:$SHORT_SHA
15      - --cache=true
16      - --reproducible
17      # - --use-new-run

code

The following code has a total of 112 dependencies (!!) weighing in at 17M (vendored).

go.mod:

module go.seankhliao.com/testrepo-176

go 1.16

require (
        go.seankhliao.com/usvc v0.8.8
        golang.org/x/net v0.0.0-20200822124328-c89045814202
)

main.go:

 1package main
 2
 3import (
 4        "flag"
 5        "os"
 6
 7        "go.seankhliao.com/usvc"
 8        "golang.org/x/net/context"
 9)
10
11var (
12        cacheBust = 0
13)
14
15func main() {
16        usvc.Exec(context.Background(), &Server{}, os.Args)
17}
18
19type Server struct{}
20
21func (s *Server) Flags(fs *flag.FlagSet)                        {}
22func (s *Server) Setup(ctx context.Context, u *usvc.USVC) error { return nil }