k8s cluster setup

starting a kubernetes cluster

k8s cluster setup

starting a kubernetes cluster

kubernetes

tldr: kind for local testing, k3s for dead easy cluster setup, kubeadm for a customized install

all are tested with a Arch Linux host created with:

gcloud compute instances create k0 \
    --image-project arch-linux-gce --image-family arch \
    --machine-type e2-medium

single machine

kind

use extraPortMappings to map host ports onto a node, then use an ingresscontroller with hostports

sudo pacman -Syu docker kubectl
sudo systemctl enable --now docker
sudo usermod -aG docker $(whoami)
newgrp docker

sudo curl -Lo /usr/local/bin/kind https://kind.sigs.k8s.io/dl/v0.9.0/kind-linux-amd64
sudo chmod +x /usr/local/bin/kind
kind create cluster

kubectl get --all-namspaces all

minikube

separate command to tunnel into LoadBalancer services

sudo pacman -Syu docker kubectl
sudo systemctl enable --now docker
sudo usermod -aG docker $(whoami)
newgrp docker

sudo curl -Lo /usr/local/bin/minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo chmod +x /usr/local/bin/minikube
minikube start

kubectl get --all-namspaces all

microk8s

This needs snap... too much of a PITA to install

multi machine

k3s

starts traefik with hostports 80, 443, 8080
Klipper LB (node hostports) for Service LoadBalancer
installs itself as a systemd service

sudo pacman -Syu which

curl -sfL https://get.k3s.io | sh -

kubectl get --all-namspaces all

kubeadm

provide your own stuff, this is "just" responsible for half the config
using the containerd runtime, --cni-bin-dir isn't supported, either:
- configure containerd to match the kubelet args /etc/containerd/config.toml
- make sure kubelet uses the default /opt/cni/bin

sudo pacman -Syu containerd kubeadm kubelet kubectl
sudo rm /etc/sysctl.d/60-gce-network-security.conf
sudo sysctl --system
sudo modprobe br_netfilter
sudo mkdir -p /etc/containerd
cat << EOF | sudo tee /etc/containerd/config.toml
version = 2
[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/usr/lib/cni"
EOF
sudo systemctl enable --now containerd
sudo systemctl enable kubelet

sudo kubeadm init --pod-network-cidr=10.244.0.0/16
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get --all-namespaces all

cilium

system requirements is really important, ex that note about systemd and rp_filter

sudo pacman -Syu containerd kubeadm kubelet kubectl
sudo rm /etc/sysctl.d/60-gce-network-security.conf
echo 'net.ipv4.conf.lxc*.rp_filter = 0' | sudo tee /etc/sysctl.d/99-override_cilium_rp_filter.conf
sudo systemctl restart systemd-sysctl
sudo modprobe br_netfilter
sudo mkdir -p /etc/containerd
echo 'KUBELET_ARGS=' | sudo tee /etc/kubernetes/kubelet.env
sudo systemctl enable --now containerd
sudo systemctl enable kubelet

sudo kubeadm init
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.9/install/kubernetes/quick-install.yaml

sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get --all-namespaces all

k0s

doesn't do much for networking / services

sudo pacman -Syu kubectl
echo 127.0.0.1 localhost | sudo tee -a /etc/hosts

sudo curl -Lo /usr/local/bin/k0s https://github.com/k0sproject/k0s/releases/download/v0.8.1/k0s-v0.8.1-amd64
sudo chmod +x /usr/local/bin/k0s
sudo k0s server

sudo kubectl --kubeconfig /var/lib/k0s/pki/admin.conf get --all-namespaces all

kops

kOps

This is for creating your self managed kubernetes clusters in cloud environments, not exactly what I want to testing today. Short setup though.

kubespray

Similar to kOps but uses ansible and slightly more generic? Very long setup.