We've settled on Argo CD for our continuous deployments. Some other parts of our stack: helm for packaging kubernetes manifests and helm-gcs (bash version) for a private helm repository.
So a colleague was tasked with setting it up. Everything worked more or less as expected, well almost everything, they said the helm-gcs plugin didn't work. Sprint planning comes around and now there's a ticket for solving the issue with 3 options: fixing it as an upstream bug (linking to #4439), using an alternate chart repository, and something else i forgot. Side note: an option would also be to expose the bucket as a static site. Noises were made about it being a difficult problem, I ask if they tried the other helm-gcs and it was promptly dumped on me.
Ah, what do we do? Try to reproduce the problem of course.
gsutilnot installed :facepalm:
What next? Try to see what's happening.
found in Chart.yaml, but missing in charts/ directory
helm dependency buildto our helm wrapper script (we need it for helm-secrets)
Unable to move current charts to tmp dir: rename charts tmpcharts: file exists
tmpchartsis from when helm is killed
helm dependency buildafter the first failure
That was a dead end, time to get my hands dirty
time helm dependency build
1m33sso borderline so it sometimes works and sometimes (mostly) doesn't
The helm-gcs we use is a bash script that wraps
gsutil which is itself written in python.
Argo itself is written in Go, so the resources were sized accordingly,
25/50m cpu and
Python is hungry and fat.
I believe my original proposition of using the other helm-gcs (written in Go) might also have worked, but wouldn't have uncovered this issue of resource starvation.