Drone

Large (20-60GB) cache

Hi, I’m looking for a CI system to build Yocto images. The build directory is large and runs from 20-60GB. The build directory must be preserved between builds (cached). Will a drone volume-cache (I assume it is just a docker volume) be practical for something this large?

We currently use containers to do builds, but we just map in the build directory from the host file system into the container.

Cliff

I cannot speak to performance characteristics of existing plugins, since they typically tar or gzip the directory when they cache on the host machine. However, you always have the option to create your own custom plugin that is optimized for your use case. Creating a plugin is pretty simple [1]. The volume plugin, for example, is under 100 lines of bash code [2].

[1] https://docs.drone.io/plugins/examples/bash/
[2] https://github.com/Drillster/drone-volume-cache/blob/master/cacher.sh

Thanks Brad – I’ll set things up and start learning.

For this use case, copying or zipping the cache won’t work – seems like it has to be a docker volume or bind mount to the host. Ideally, I’d like to be able to bind mount a directory on the host into the container running a task.

To mount host machine volumes see https://docs.drone.io/user-guide/pipeline/volumes/

Note that you need to host your own drone installation to mount volumes. For security reasons we do not allowing mounting host machine volumes in Drone Cloud. Also our servers would not support such high disk space usage for a single pipeline.

its working:

Still some details to work through, but I’m liking Drone a lot.

Thanks for the great support.

The exec runner (https://exec-runner.docs.drone.io/configuration/) is also a good fit for this type of build. I think it was released after our initial discussion above.