go mod basics

is this a faq on modules?

SEAN K.H. LIAO

go mod basics

is this a faq on modules?

modules

Go modules: the dependency management system for the Go ecosystem.

What it optimizes for: automatic and stable version selection over time. This means that if your code builds now with a set of recorded dependencies, it will continue to build in the future no matter what your dependencies publish (or unpublish). To do this, it relies on the ecosystem following semantic versioning and on shared infrastructure such as a caching proxy and transparency log. Recommended reading: semantic import versioning minimal version selection reproducible, verifiable, verified builds

lightning round

How do I
start
1go mod init example.com/some/module

Modules are units of versioning, and in most cases you will want a single module at the root of your repository.

Module names are / separated ascii, the first segment must contain a dot (names without a dot are reserved for the stdlib with the exception of example and test). If you host your code remotely, it should match the code host; if it's only available locally, feel free to use one of the reserved tlds like some-name.local. Versions 2+ need a /vN suffix.

get dependencies

go get some-package@version or after writing import "..." in youf code, go mod tidy

The first gives you more control, the second will also trim out unused dependencies.

@version, one of:

note: indirect references like @latest and @branch are cached by proxies. use GOPROXY=direct to skip the proxy.

update dependencies

update all dependencies:

1go get -u ./...

update all dependencies, but only to patch versions:

1go get -u=patch ./...`

update only direct dependencies:

1go get $(go list -f '{{if not (or .Main .Indirect)}}{{.Path}}{{end}}' -m ./...)
list the chosen dependencies

Things you actually use:

1go list -deps -f '{{ if not .Standard }}{{ .Module }}{{ end }}' ./path/to/package | sort | uniq
2
3# or
4
5go version -m $executable

Why a module is somewhere in the dependency graph

1go mod why some.module/dependency
2
3# or
4
5go mod graph
printf debug a dependency

go mod vendor and edit the files in vendor/. The next run of go mod vendor will wipe away those changes.

If you need more control or expect to contribute a fix upstream, clone the repo somewhere and use replace: go mod edit -replace dependency.module/path=../path/to/cloned/repo.

fork a dependency

Is this a temporary or permanent fork?

Temporary: go mod edit -replace dependency.module/path=forked.module/path@version

Permanent: rename all instances of the import path in the forked module and treat it like any other dependency.

private code

You want to write private code, also as other blog post.

init

If you use one of the more fully featured code hosting software that responds with <meta go-import="..."> or you have a vanity domain setup:

1go mod init gitlab.corp.example/path/to/repo
2
3go mod init github.com/your/repo
4
5go mod init vanity.example/repo

Otherwise (eg with plain git), use .git as part of your module path (also import paths):

1go mod init git.host.example/your/repo.git
git config

You'll also want to have git use ssh instead of https:

1git config --global url."git@github.com:".insteadOf https://github.com/
2
3git config --global url."you@git.host.example".insteadOf https://git.host.example

or if you prefer editing .gitconfig by hand:

[url "git@github.com:"]
    insteadOf = https://github.com/
[url "you@git.host.example:"]
    insteadOf = https://git.host.example/
go command

Go will use a proxy by default. If you don't have a private one setup, exclude module path prefixes from lookup from proxies:

1GOPRIVATE=github.com/you,git.host.example
2# persist with
3go env -w GOPRIVATE=github.com/you,git.host.example

local only code

You never want to share or host your code anywhere.

1go mod init example.local/app1

if you still want to use multiple modules:

 1code
 2├── app1
 3│  ├───main.go
 4│  └── go.mod
 5│        # module example.local/app1
 6│        #
 7│        # go 1.16
 8│        #
 9│        # require example.local/some-lib v0.0.0
10│        #
11│        # replace example.local/some-lib => ../some-lib
12│
13└── some-lib
14   ├───lib.go
15   └── go.mod
16         # module example.local/app1
17         #
18         # go 1.16

ci and docker

This is complicated How to optimize this depends on 2 things: are your worker nodes stateful (can they retain a cache/volume between builds) and what do you use to build containers (docker, docker buildx, kaniko, ...).

stateful workers

Your workers have persistent volumes you can use between builds.

stateful worker docker buildx

This shares a module download cache and a build cache between all docker builds. Docker currently doesn't allow control over the cache location.

 1#syntax=docker/dockerfile:1.2
 2FROM golang:alpine AS build
 3WORKDIR /workspace
 4COPY . .
 5RUN --mount=type=cache,id=gomod,target=/go/pkg/mod \
 6    --mount=type=cache,id=gobuild,target=/root/.cache/go-build \
 7    go build -o app
 8
 9FROM scratch
10COPY --from=build /workspace/app /app
11ENTRYPOINT ["/app"]
stateful worker kaniko

This takes advantage of kaniko skipping /var/run, so we can put our mutable cache there and share it between runs.

1docker run --rm -it \
2  -v $(pwd):/workspace \
3  -v /path/to/mod/cache:/var/run/go-mod \
4  -v /path/to/build/cache:/var/run/go-build \
5  -w /workspace \
6  gcr.io/kaniko-project/executor:latest \
7  -c=. \
8  -f=Dockerfile \
9  -d=your.docker/registry/image

with dockerfile:

 1FROM golang:alpine AS build
 2ENV GOCACHE=/var/run/go-build \
 3    GOMODCACHE=/var/run/go-mod
 4WORKDIR /workspace
 5COPY . .
 6RUN go build -o app
 7
 8FROM scratch
 9COPY --from=build /workspace/app /app
10ENTRYPOINT ["/app"]
stateless workers

This is complicated, you have to trade off between downloading and restoring a cache, and just doing the work.

if you just want the download step to be cacheable as a layer:

1FROM golang:alpine AS build
2WORKDIR /workspace
3COPY go.mod go.sum .
4RUN go mod download
5
6COPY . .
7RUN go build -o app

and run with the below if you're using multistage builds

1docker buildx \
2  --cache-from type=registry,ref=your.registry/image \
3  --cache-to   type=registry,ref=your.registry/image,mode=max \

If you also want to share the build cache as a layer, the best way might be to build a base image with your code once, and update the base image every time you update dependencies.

1FROM golang:alpine AS base-image
2WORKDIR /workspace
3COPY . .
4RUN go build ./... && \
5    rm -rf *

questions

The better places to ask questions:

#modules on gophers slack, invite link

Mailing list: go-nuts

places i looked for questions

reports

summary / comments