[drone-runner-kube] Not correctly setting resource requests for Steps

[Kash]

I’ve noticed some unusual behaviour when setting Resource Requests for Pipeline Steps. In our runner we provide the following two env vars:

• DRONE_RESOURCE_REQUEST_CPU | Drone
• DRONE_RESOURCE_REQUEST_MEMORY | Drone

DRONE_RESOURCE_REQUEST_CPU: 200m
DRONE_RESOURCE_REQUEST_MEMORY: 100MiB

I assumed (perhaps wrongly) that this is the default CPU & Memory Request applied to each individual Step, in the absence of Resource Requests being defined in the Pipeline. Instead what it seems, is that this is the Maximum Requests which are applied to the entire Pipeline (which is not overridable). So every Step in the Pipeline ends up with:

    Requests:
      cpu:     1m
      memory:  4Mi

But the first Step (usually the git clone) will get whatever is remaining from the total after the above values were subtracted. So if you have 4 more Steps after the git clone, your first clone Step would reserve:

    Requests:
      cpu:     196m
      memory:  84Mi

If this is intentional behaviour then the docs should probably be updated with more of an explanation that setting these vars on the runner means it’s the total reserved resources for the entire Pod that is scheduled (overriding any specification for individual Steps in a users Pipeline), split across each Container.

I’ll test removing both env variables to make sure each step can define their own Requests. It would still be nice to set a minimum applied for all Steps (at the runner level), so that they are given a buffer of reserved resources and reduce the likelihood of over-allocation of Jobs on a single Node (another problem we hit).

[Kash]

It’s related to this logic which was merged in December but not officially tagged: drone-runner-kube/compiler.go at master · drone-runners/drone-runner-kube · GitHub

[Kash]

I’ve also tested the following two env vars (in isolation) introduced in that commit:

• DRONE_RESOURCE_MIN_REQUEST_CPU
• DRONE_RESOURCE_MIN_REQUEST_MEMORY

Their behaviour overrides the Resource Requests for each Step, regardless of what the step has defined. I would expect that it would set the higher value of either the env vars above, or what has been defined in the Step. For example if the Env Var has 200MiB for Memory and the Step has 250MiB, it should set the latter in the Spec.

[bradrydzewski]

This thread explains the changes we are making to how the kubernetes runner works:

Strategy for pod/cpu settings on kubernetes runner

Similar to Tekton, we could specify resource request as a top-level item and then specify individual container limits as shown below: kind: pipeline type: kubernetes +resources: + request: + cpu: 2000 + memory: 2000MiB steps: - name: en image: alpine commands: - echo hello resources: limit: cpu: 1000 memory: 1000MiB - request: - cpu: 2000 - memory: 2000MiB - name: es image: alpine commands: - echo hola - name…

Please keep in mind the Kubernetes runner is in Beta and we do not have a stable specification. We are still making breaking changes which may result in inconsistencies in the docs (despite our best efforts to keep them up to date). If you need something more stable, we recommend the Docker runner.