When multiple events for the same repository hit Drone in quick succession, is there a way to force Drone to queue builds for said repository? Our goal is to avoid the overlapped (simultaneous) execution of any pipeline step across revisions.
In more concrete terms, if a (GitHub) repository sends events
push 1 and
push 2 in the space of a couple of minutes, we would like all pipelines for event
push 1 to complete execution before the first pipeline for event
push 2 is scheduled for execution. It would be ideal if this queuing behaviour could be specified at the repository layer so that other repositories – not subject to unusual mutual exclusion constraints – are free to maximise concurrency and make full use of available resources on execution nodes.
We use Drone to execute some of our Terraform pipelines. I’ll try to summarise this problem in a general way, so that it will be understandable without any prior exposure to this specific tooling. Terraform is a doohickey that transforms plain text into live IaaS resources in some remote cloud. By necessity, the tool must interact with remote resources; these remote resources are obviously not contained within the ephemeral build context, and so, our builds are not perfectly hermetic. As a safety precaution, the tool relies upon a remote mutex to ensure that only one instance of itself is interacting with remote resources at any given time. You can probably see where this is going.
Our Drone builds work fine if there is only one build executing at any given time. Terraform is able to lock its mutex, do its work, then unlock its mutex in preparation for the next build.
Unfortunately, our Drone builds fail – with false negatives – if the Drone scheduler attempts to execute for multiple revisions at the same time. All concurrent builds subsequent to the first will be unable to lock that mutex. These builds would have succeeded had they been queued.
Our builds are wide and short.
.drone.yml contains a number of pipelines (cardinality increases over time) that we endeavour to execute concurrently for a single revision. Each pipeline contains only a couple of steps. (And each pipeline is associated with a unique terraform mutex.) We have only one (docker) runner at the moment configured with
I could perhaps work around this problem with a step that blocks on the mutex lock, but that would tie up an executor slot for no good reason. Other repositories may not be able to build in the meantime.