Is there an option to spin-up agents dynamically as the jobs come and then kill them when they are done. More like jenkins slaves like https://github.com/jenkinsci/kubernetes-plugin . We are using this option dynamically scale of CI pipeline by adding containers / nodes during the week when there is a lot of activity in Git, but over the weekend things die down when there isn’t much activity.
This would help because the builds don’t have to wait in the queue. When nodes can’t take the container capacity request new nodes are dynamically added by kubernetes auto scaling which would help a lot.