Long pending times on cloud.drone.io

#1

Hi,

I’ve already mentioned this multiple times in the Gitter chat but never got a response. First let me say I don’t want to blame someone, but I would like to get expectations for cloud.drone.io clarified.

During the last weeks I noticed very long pending times for ARM jobs especially for ARM64. This happens multiple times per day and such jobs were pending for 1-2 hours before they get picked up. For a job that has a full runtime of ~ 5 minutes that’s kinda useless. I understand that it’s a free service and there are limited resource. Limiting parallel runs per user or/repo would be fine but those long pending jobs even if there is just a single active job for the current user is really a productivity blocker. So some questions that comes to my mind:

  • Is cloud.drone.io expected to be used as a reliable production CI for open source projects or a more public tech demo of Drone?
  • Are (optional) paid subscriptions planned for cloud.drone.io?
  • What about resource limit/fair use documentation?

I’m aware of the possibility to self-host Drone but that’s often also not that simple for part-time open source projects. Again, this post is not about to make a claim for something, just to find out the current state and future plans.

#2

Hi @xoxys, Thanks for reaching out! Going forward please log your issue here in this forum and please be assured that you will get a response for sure here as our support team always keep an eye on this queue.

It surprising to see the job is on hold for hours, can you confirm the connectivity of Drone runner in your infrastructure to cloud.drone.io. It would be helpful if you can share Drone runner logs to assist further.

On your last question, Yes we do have enterprise support for Drone please feel free to check Drone Enterprise and let us know if you have any questions.

Regards,
Harness Support Team

#3

Thanks for your response. Now I’m confused. I’m talking about the free cloud.drone.io instance, so I don’t have access nor control over the used runners. Example:

This job finished in < 5 min: Drone | Continuous Integration

Other Jobs needed up to 2 hours to finish as the ARM64 pipeline was pending for more than an hour:

grafik
grafik1130×571 52.3 KB

I know about the Drone Enterprise support, but the Enterprise support does not scale for private persons like me or small open source projects without a budget. The Enterprise support works also just for on-prem servers right? What I was asking for is something like a payable subscription/extended plan for the hosted cloud.drone.io e.g. Circle CI offers an extended plan for around $30 per month to get more concurrent jobs/better performance and to help you to continue offering the free plan for cloud.drone.io

#4

Sadly still the same. As I don’t know how long this situation will remain I have to look for alternatives :frowning:

#5

FWIW I’m seeing this in SIMDe, too. Yesterday the waits were between ~ 2 and 4 hours.

As I don’t know how long this situation will remain I have to look for alternatives

If you find anything please let me know. AFAIK the only providers which have AArch64 are Drone and Travis CI, and Travis CI’s new limitations have been crippling for us; within the last week or two I’ve removed all builds from Travis. Even if this gets fixed I’d like to spread the load around a bit if possible.

One option might be the “Works on Arm Cluster” (WorksOnArm/cluster on GitHub; sorry, the software limits new users to a single link per post…), which provides free access to Arm hardware for open source development. It might even be possible to install a Drone runner to provide access to multiple repos. I don’t know enough about Drone’s software to volunteer to manage that, but I’d be interested in taking advantage of someone else’s hard work ;).

#6

Drone cloud runs on donated infrastructure and we have fixed capacity that allows us to run 200+ concurrent pipelines on bare metal instances. However, we sometimes experience high build volume which causes congestion and results in long wait times.

As @nemequ mentions, the best option to guarantee short queue time is to install Drone on your own infrastructure. The Works on Arm program donates Arm machines to open source projects, and you can also get free open source credits for AWS which also supports Arm. There are also plenty of providers that you can use for low cost amd64 machines.

#7

Alright, thanks for the clarification. I guess running own infrastructure for single user projects often doesn’t make sense and not all developers can manage own servers. That’s why cloud.drone.io as managed services is a wonderful thing. Limited donated resources is understandable but that’s why I asked for any plans to allow monthly donations. Personally I would like to pay for the hosted cloud service and I think others would do as well if the price is “fair”.

#8

Edit: Never mind, I just found Drone - Armv8 Collaboration, it looks like they already have Drone set up and I should request access using that GitHub project.

#9

Harness is a multi-million dollar company (at least in VC funding terms) and should not require donations. That said, I’d also probably pay per-hour for the hosted instances, too.

#10

Is cloud.drone.io expected to be used as a reliable production CI for open source projects or a more public tech demo of Drone?

While the first question still lacks a clear answer you can find one in another thread Unstable performance on ARM64 - #2 by bradrydzewski