SEANK.H.LIAO

managing k8s manifests

too much yaml

k8s manifests

Kubernetes manifests are great, defining a common, declarative, API-first approach to managing resources. Just too bad that the canonical format is YAML with all its oddities, especially around types. And the fact that there isn't much to help you deal with repitition of same or similar blocks.

So what other options do we have?

helm

Ah, Helm, CNCF Graduated, and the closest thing to a standard package manager for kubernetes. What you get is using Go text/template to generate YAML (not even structured data), feeding in variables through another YAML file, with all the typing problems that comes with it. It sort-of helps with duplication in a single chart, or if you have the stomach for it, a single chart that you can cram all your microservices into. It's a (not very smart) templating engine with a deployment tool on top, and your customization points are limited to what the chart authors came up with, and if you use upstream charts, they're all of varying quality. you have to deal with varying levels of quality from upstream charts.

kustomize

A step up from raw kubernetes manifests, you get some common functionality, like label and annotations for everything, in a short form with patches and transformers for a fully customizable output. It does require more sprawl in directory and file setups but it can manage your entire cluster. You're still writing YAML though... and the correct docs for it are a bit hard to find (hint: last one):

terraform kubernetes provider

Kubernetes provider: you write the same manifest, but in HCL / terraform. Good: less YAML to worry about. Bad: HCL is more verbose in lines-of-code (screen real estate), blocks don't have a good story for reuse in variables, and you're limited to the resource versions they've defined.

terraform kubernetes alpha provider

Kubernetes Alpha provider: write any k8s resource in HCL / terraform, which sort of makes you question why, other than you really hate YAML.

grafana tanka

Tanka, write your manifests in json(net), it's not as bad as it sounds, with some functions and templating blocks and overrides, but it is a bit exotic? If you're fully into the Grafana ecosystem, jsonnet is probably really worth it (also for Grafana dashboards).

cue

CUE, more a language than a fully featured tool, the core idea seems to be having a schema you can validate / generate against. The comparison sections in its usecases go into more detail, but in general, probably not as good for managing config sprawl.