i’m trying to come up with a good way to set up cpu/memory options for our kubernetes-based drone setup.
Here’s the problem:
All runner-pods run on dedicated (via tains/toleratinos) runner nodes, and we want these to scale up and down on demand. In order to do that, the pod needs some settings informing kubernetes on the amount of memory and cpu it is likely to consume. Here’s the issue: Unlike a lot of other (non-k8s) build systems, drone ci creates a single pod containing all containers (steps) the build pipeline needs. In Kuberenets, cpu/memory is a per-container thing, not a pod-thing.
This means that if I set “1 cpu, 1GB” for a pipeline, every single container in the pipeline get those settings. If the pipeline has 10 steps, that means 10 cores and 10 gigs of ram. If there happens to be an available node with capable of running that, fine - but if not the build pod will just be pending indefinetely.
On the other side, if I don’t set any cpu/memory limits, kubernetes’ cluster-autoscaler won’t know when it’s time to add more nodes to the autoscaling group, since it doesn’t have enough information about the requirements of the “upcoming” pod.
All in all, I would conclude because of the way drone ci uses runner pods, kubernete’s built-in scaling mechanisms are not adequate to build a robust autoscaling build environment for Drone CI. I would be interested in learning what other users are thinking about this.