k8s cluster setup

starting a kubernetes cluster

kubernetes

tldr: kind for local testing, k3s for dead easy cluster setup, kubeadm for a customized install

all are tested with a Arch Linux host created with:

1gcloud compute instances create k0 \
2    --image-project arch-linux-gce --image-family arch \
3    --machine-type e2-medium

single machine

kind

use extraPortMappings to map host ports onto a node, then use an ingresscontroller with hostports

 1sudo pacman -Syu docker kubectl
 2sudo systemctl enable --now docker
 3sudo usermod -aG docker $(whoami)
 4newgrp docker
 5
 6sudo curl -Lo /usr/local/bin/kind https://kind.sigs.k8s.io/dl/v0.9.0/kind-linux-amd64
 7sudo chmod +x /usr/local/bin/kind
 8kind create cluster
 9
10kubectl get --all-namspaces all

minikube

separate command to tunnel into LoadBalancer services

 1sudo pacman -Syu docker kubectl
 2sudo systemctl enable --now docker
 3sudo usermod -aG docker $(whoami)
 4newgrp docker
 5
 6sudo curl -Lo /usr/local/bin/minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
 7sudo chmod +x /usr/local/bin/minikube
 8minikube start
 9
10kubectl get --all-namspaces all

microk8s

This needs snap... too much of a PITA to install

multi machine

k3s

starts traefik with hostports 80, 443, 8080
Klipper LB (node hostports) for Service LoadBalancer
installs itself as a systemd service

1sudo pacman -Syu which
2
3curl -sfL https://get.k3s.io | sh -
4
5kubectl get --all-namspaces all

kubeadm

provide your own stuff, this is "just" responsible for half the config
using the containerd runtime, --cni-bin-dir isn't supported, either:
- configure containerd to match the kubelet args /etc/containerd/config.toml
- make sure kubelet uses the default /opt/cni/bin

 1sudo pacman -Syu containerd kubeadm kubelet kubectl
 2sudo rm /etc/sysctl.d/60-gce-network-security.conf
 3sudo sysctl --system
 4sudo modprobe br_netfilter
 5sudo mkdir -p /etc/containerd
 6cat << EOF | sudo tee /etc/containerd/config.toml
 7version = 2
 8[plugins."io.containerd.grpc.v1.cri".cni]
 9  bin_dir = "/usr/lib/cni"
10EOF
11sudo systemctl enable --now containerd
12sudo systemctl enable kubelet
13
14sudo kubeadm init --pod-network-cidr=10.244.0.0/16
15kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
16
17sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get --all-namespaces all

cilium

system requirements is really important, ex that note about systemd and rp_filter

 1sudo pacman -Syu containerd kubeadm kubelet kubectl
 2sudo rm /etc/sysctl.d/60-gce-network-security.conf
 3echo 'net.ipv4.conf.lxc*.rp_filter = 0' | sudo tee /etc/sysctl.d/99-override_cilium_rp_filter.conf
 4sudo systemctl restart systemd-sysctl
 5sudo modprobe br_netfilter
 6sudo mkdir -p /etc/containerd
 7echo 'KUBELET_ARGS=' | sudo tee /etc/kubernetes/kubelet.env
 8sudo systemctl enable --now containerd
 9sudo systemctl enable kubelet
10
11sudo kubeadm init
12kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.9/install/kubernetes/quick-install.yaml
13
14sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get --all-namespaces all

k0s

doesn't do much for networking / services

1sudo pacman -Syu kubectl
2echo 127.0.0.1 localhost | sudo tee -a /etc/hosts
3
4sudo curl -Lo /usr/local/bin/k0s https://github.com/k0sproject/k0s/releases/download/v0.8.1/k0s-v0.8.1-amd64
5sudo chmod +x /usr/local/bin/k0s
6sudo k0s server
7
8sudo kubectl --kubeconfig /var/lib/k0s/pki/admin.conf get --all-namespaces all

kops

kOps

This is for creating your self managed kubernetes clusters in cloud environments, not exactly what I want to testing today. Short setup though.

kubespray

Similar to kOps but uses ansible and slightly more generic? Very long setup.