go mod basics

is this a faq on modules?

SEAN K.H. LIAO

go mod basics

is this a faq on modules?

modules

Go modules: the dependency management system for the Go ecosystem.

What it optimizes for: automatic and stable version selection over time. This means that if your code builds now with a set of recorded dependencies, it will continue to build in the future no matter what your dependencies publish (or unpublish). To do this, it relies on the ecosystem following semantic versioning and on shared infrastructure such as a caching proxy and transparency log. Recommended reading: semantic import versioning minimal version selection reproducible, verifiable, verified builds

lightning round

How do I
start
go mod init example.com/some/module

Modules are units of versioning, and in most cases you will want a single module at the root of your repository.

Module names are / separated ascii, the first segment must contain a dot (names without a dot are reserved for the stdlib with the exception of example and test). If you host your code remotely, it should match the code host; if it's only available locally, feel free to use one of the reserved tlds like some-name.local. Versions 2+ need a /vN suffix.

get dependencies

go get some-package@version or after writing import "..." in youf code, go mod tidy

The first gives you more control, the second will also trim out unused dependencies.

@version, one of:

note: indirect references like @latest and @branch are cached by proxies. use GOPROXY=direct to skip the proxy.

update dependencies

update all dependencies:

go get -u ./...

update all dependencies, but only to patch versions:

go get -u=patch ./...`

update only direct dependencies:

go get $(go list -f '{{if not (or .Main .Indirect)}}{{.Path}}{{end}}' -m ./...)
list the chosen dependencies

Things you actually use:

go list -deps -f '{{ if not .Standard }}{{ .Module }}{{ end }}' ./path/to/package | sort | uniq

# or

go version -m $executable

Why a module is somewhere in the dependency graph

go mod why some.module/dependency

# or

go mod graph
printf debug a dependency

go mod vendor and edit the files in vendor/. The next run of go mod vendor will wipe away those changes.

If you need more control or expect to contribute a fix upstream, clone the repo somewhere and use replace: go mod edit -replace dependency.module/path=../path/to/cloned/repo.

fork a dependency

Is this a temporary or permanent fork?

Temporary: go mod edit -replace dependency.module/path=forked.module/path@version

Permanent: rename all instances of the import path in the forked module and treat it like any other dependency.

private code

You want to write private code, also as other blog post.

init

If you use one of the more fully featured code hosting software that responds with <meta go-import="..."> or you have a vanity domain setup:

go mod init gitlab.corp.example/path/to/repo

go mod init github.com/your/repo

go mod init vanity.example/repo

Otherwise (eg with plain git), use .git as part of your module path (also import paths):

go mod init git.host.example/your/repo.git
git config

You'll also want to have git use ssh instead of https:

git config --global url."git@github.com:".insteadOf https://github.com/

git config --global url."you@git.host.example".insteadOf https://git.host.example

or if you prefer editing .gitconfig by hand:

[url "git@github.com:"]
    insteadOf = https://github.com/
[url "you@git.host.example:"]
    insteadOf = https://git.host.example/
go command

Go will use a proxy by default. If you don't have a private one setup, exclude module path prefixes from lookup from proxies:

GOPRIVATE=github.com/you,git.host.example
# persist with
go env -w GOPRIVATE=github.com/you,git.host.example

local only code

You never want to share or host your code anywhere.

go mod init example.local/app1

if you still want to use multiple modules:

code
├── app1
│  ├───main.go
│  └── go.mod
│        # module example.local/app1
│        #
│        # go 1.16
│        #
│        # require example.local/some-lib v0.0.0
│        #
│        # replace example.local/some-lib => ../some-lib
│
└── some-lib
   ├───lib.go
   └── go.mod
         # module example.local/app1
         #
         # go 1.16

ci and docker

This is complicated How to optimize this depends on 2 things: are your worker nodes stateful (can they retain a cache/volume between builds) and what do you use to build containers (docker, docker buildx, kaniko, ...).

stateful workers

Your workers have persistent volumes you can use between builds.

stateful worker docker buildx

This shares a module download cache and a build cache between all docker builds. Docker currently doesn't allow control over the cache location.

#syntax=docker/dockerfile:1.2
FROM golang:alpine AS build
WORKDIR /workspace
COPY . .
RUN --mount=type=cache,id=gomod,target=/go/pkg/mod \
    --mount=type=cache,id=gobuild,target=/root/.cache/go-build \
    go build -o app

FROM scratch
COPY --from=build /workspace/app /app
ENTRYPOINT ["/app"]
stateful worker kaniko

This takes advantage of kaniko skipping /var/run, so we can put our mutable cache there and share it between runs.

docker run --rm -it \
  -v $(pwd):/workspace \
  -v /path/to/mod/cache:/var/run/go-mod \
  -v /path/to/build/cache:/var/run/go-build \
  -w /workspace \
  gcr.io/kaniko-project/executor:latest \
  -c=. \
  -f=Dockerfile \
  -d=your.docker/registry/image

with dockerfile:

FROM golang:alpine AS build
ENV GOCACHE=/var/run/go-build \
    GOMODCACHE=/var/run/go-mod
WORKDIR /workspace
COPY . .
RUN go build -o app

FROM scratch
COPY --from=build /workspace/app /app
ENTRYPOINT ["/app"]
stateless workers

This is complicated, you have to trade off between downloading and restoring a cache, and just doing the work.

if you just want the download step to be cacheable as a layer:

FROM golang:alpine AS build
WORKDIR /workspace
COPY go.mod go.sum .
RUN go mod download

COPY . .
RUN go build -o app

and run with the below if you're using multistage builds

docker buildx \
  --cache-from type=registry,ref=your.registry/image \
  --cache-to   type=registry,ref=your.registry/image,mode=max \

If you also want to share the build cache as a layer, the best way might be to build a base image with your code once, and update the base image every time you update dependencies.

FROM golang:alpine AS base-image
WORKDIR /workspace
COPY . .
RUN go build ./... && \
    rm -rf *

questions

The better places to ask questions:

#modules on gophers slack, invite link

Mailing list: go-nuts

places i looked for questions

reports

summary / comments