Drone

Intermittent Drone Server/Nginx Timeout Outages

Hello, we are seeing intermittent outages with the drone UI (showing as Nginx timeout and are having some trouble debugging the root cause. The drone agent containers look OK. I have provided the below log. “[Recovery] panic recovered” appears multiple times. Any feedback is appreciated

Log:

error: cannot find parent proc with id 77317: dial tcp 172.18.4.16:5432: getsockopt: connection timed out

error: rpc.update: cannot find pproc with id 77317: dial tcp 172.18.4.16:5432: getsockopt: connection timed out

error: cannot find proc with id 77317: dial tcp 172.18.4.16:5432: getsockopt: connection timed out

error: rpc.update: cannot find pproc with id 77332: dial tcp 172.18.4.16:5432: getsockopt: connection timed out

error: cannot find parent proc with id 77332: dial tcp 172.18.4.16:5432: getsockopt: connection timed out

error: rpc.update: cannot find pproc with id 77332: dial tcp 172.18.4.16:5432: getsockopt: connection timed out

time=“2019-11-20T19:19:24Z” level=error msg=“Error #01: sql: no rows in result set\n” ip=10.254.0.59 latency=3.099827ms method=GET path="/api/repos/XXXlogs/34/2" status=404 time=“2019-11-20T19:19:24Z” user-agent=“Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36”

time=“2019-11-20T19:19:25Z” level=error msg=“Error #01: sql: no rows in result set\n” ip=10.254.0.59 latency=3.750217ms method=GET path="/api/repos/XXX/logs/34/2" status=404 time=“2019-11-20T19:19:25Z” user-agent=“Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:71.0) Gecko/20100101 Firefox/71.0”

time=“2019-11-20T19:33:18Z” level=error msg=“error setting commit status for XXX/34: Post https://XXX/rest/build-status/1.0/commits/e515b5dca56558d4fbede922f880a7f588d02af8: Post https://XXX/rest/build-status/1.0/commits/e515b5dca56558d4fbede922f880a7f588d02af8: read tcp 192.168.199.11:45060->172.18.3.9:443: read: connection timed out”

2019/11/20 19:53:11 [Recovery] panic recovered:

POST /hook?access_token=XXX

Host: XXX

Connection: close

Accept: application/json

Accept-Encoding: gzip,deflate

Connection: close

Content-Length: 555

Content-Type: application/json

As an aside, it looks like you are using Drone version 0.8 which is end of life. The latest stable version of Drone is 1.6. You should consider upgrading as this version is no longer supported.

1 Like