Clone: fatal: Invalid refspec '+refs/heads/:'

hi all,

my working setup recently starts throwing erros on git clone. My drone setup is running in docker on drone:latest (updated on 26th of august). drone is connected to gitea (also running on latest docker image). I have a repo which is triggered to deploy when a tag is created. My last working deploy was 9 days ago on 23rd of august. Since then I didn’t changed anything on the drone.yaml but the only things changed are updates to drone and gitea. So the gitea webhook still lokks quite similar. Maybe my setup was wrong before and just worked by accident or there might be something broken.

My yaml lokks like this:

name: deployment
kind: pipeline

steps:
  - name: copy_release
    image: appleboy/drone-scp
    settings:
      host: some.server.net
      username: some_user
      password: some_password
      port: 22
      target: /target
      source: /source
      rm: true

  - name: activate_release
    image: appleboy/drone-ssh
    settings:
      host: some.server.net
      username: some_user
      password: some_password
      port: 22
      command_timeout: 10m
      script:
        - some_commands

trigger:
  event:
    - tag

Webhooks look like (containing refs to the tag):

{
  "secret": "secret",
  "ref": "refs/tags/1.1.21",
  "before": "0000000000000000000000000000000000000000",
  "after": "d2af71c7c3058ac2725848b91c27271cdcb7ea29",
  "compare_url": "https://some.server.net:3000/",
  "commits": [],
  "head_commit": null,
  "repository": {
    "id": 2,
    "owner": {
      "id": 2,
      "login": "private",
      "full_name": "",
      "email": "",
      "avatar_url": "https://some.server.net:3000/avatars/2",
      "language": "",
      "is_admin": false,
      "last_login": "1970-01-01T00:00:00Z",
      "created": "2018-03-15T18:16:13Z",
      "username": "private"
    },
    "name": "some.project",
    "full_name": "private/some.project",
    "description": "some description",
    "empty": false,
    "private": false,
    "fork": false,
    "parent": null,
    "mirror": false,
    "size": 7310,
    "html_url": "https://some.server.net:3000/private/some.project",
    "ssh_url": "git@isome.server.net:private/some.project.git",
    "clone_url": "https://some.server.net:3000/private/some.project.git",
    "original_url": "",
    "website": "",
    "stars_count": 0,
    "forks_count": 0,
    "watchers_count": 1,
    "open_issues_count": 2,
    "default_branch": "master",
    "archived": false,
    "created_at": "2018-03-14T15:56:04Z",
    "updated_at": "2019-09-01T12:07:56Z",
    "permissions": {
      "admin": false,
      "push": false,
      "pull": false
    },
    "has_issues": true,
    "has_wiki": false,
    "has_pull_requests": false,
    "ignore_whitespace_conflicts": false,
    "allow_merge_commits": false,
    "allow_rebase": false,
    "allow_rebase_explicit": false,
    "allow_squash_merge": false,
    "avatar_url": ""
  },
  "pusher": {
    "id": 1,
    "login": "login",
    "full_name": "Full Name",
    "email": "someone@mail.net",
    "avatar_url": "https://secure.gravatar.com/avatar/111222?d=identicon",
    "language": "en-US",
    "is_admin": true,
    "last_login": "2019-08-30T18:10:23Z",
    "created": "2018-03-14T14:58:54Z",
    "username": "user"
  },
  "sender": {
    "id": 1,
    "login": "login",
    "full_name": "Full Name",
    "email": "someone@mail.net",
    "avatar_url": "https://secure.gravatar.com/avatar/111222?d=identicon",
    "language": "en-US",
    "is_admin": true,
    "last_login": "2019-08-30T18:10:23Z",
    "created": "2018-03-14T14:58:54Z",
    "username": "user"
  }
}

And git clone error:

Initialized empty Git repository in /drone/src/.git/
usage: git remote add [<options>] <name> <url>

    -f, --fetch           fetch the remote branches
    --tags                import all tags and associated objects when fetching
                          or do not fetch any tag at all (--no-tags)
    -t, --track <branch>  branch(es) to track
    -m, --master <branch>
                          master branch
    --mirror[=<push|fetch>]
                          set up remote as a mirror to push to or fetch from

+ git fetch origin +refs/heads/:
fatal: Invalid refspec '+refs/heads/:'

Not sure what goes wrong here and why it suddenly starts throwing.

Thanks for your help!

the last change to how Drone handles Gitea webhooks were over two months ago, so I do not believe there has been any regression in the Drone codebase. I did notice the webhook body you posted does not match our test samples [1]. Have you checked with the Gitea folks to see if they have made any changes we need to be aware of?

[1] https://github.com/drone/go-scm/blob/master/scm/driver/gitea/testdata/webhooks/tag_create.json

No - I first thought there were changes - but I posted the wrong webhook payload :frowning: that one was from the push and not the create - sorry for any troubled created!

The right one looks quite similar:

{
  "secret": "secret",
  "sha": "d2af71c7c3058ac2725848b91c27271cdcb7ea29",
  "ref": "1.1.21",
  "ref_type": "tag",
  "repository": {
    "id": 2,
    "owner": {
      "id": 2,
      "login": "private",
      "full_name": "",
      "email": "",
      "avatar_url": "https://some.server.net:3000/avatars/2",
      "language": "",
      "is_admin": false,
      "last_login": "1970-01-01T00:00:00Z",
      "created": "2018-03-15T18:16:13Z",
      "username": "private"
    },
    "name": "some.project",
    "full_name": "private/some.project",
    "description": "some description",
    "empty": false,
    "private": false,
    "fork": false,
    "parent": null,
    "mirror": false,
    "size": 7310,
    "html_url": "https://some.server.net:3000/private/some.project",
    "ssh_url": "git@some.server.net:private/some.project.git",
    "clone_url": "https://some.server.net:3000/private/some.project.git",
    "original_url": "",
    "website": "",
    "stars_count": 0,
    "forks_count": 0,
    "watchers_count": 1,
    "open_issues_count": 2,
    "default_branch": "master",
    "archived": false,
    "created_at": "2018-03-14T15:56:04Z",
    "updated_at": "2019-09-01T12:07:56Z",
    "permissions": {
      "admin": false,
      "push": false,
      "pull": false
    },
    "has_issues": true,
    "has_wiki": false,
    "has_pull_requests": false,
    "ignore_whitespace_conflicts": false,
    "allow_merge_commits": false,
    "allow_rebase": false,
    "allow_rebase_explicit": false,
    "allow_squash_merge": false,
    "avatar_url": ""
  },
  "sender": {
    "id": 1,
    "login": "login",
    "full_name": "Full Name",
    "email": "mail@server.net",
    "avatar_url": "https://secure.gravatar.com/avatar/112233?d=identicon",
    "language": "en-US",
    "is_admin": true,
    "last_login": "2019-08-30T18:10:23Z",
    "created": "2018-03-14T14:58:54Z",
    "username": "user"
  }
}

are push hooks for tags new? I do not recall this being supported before.

Honestly, I can’t tell you - I just saw that inside the db:

I just have a normal webhook on gitea:

I think I could remove the push and delete flag…

But I didn’t changed anything on the settings for months and all was working well…

I can confirm we have no code in Drone to account for Gitea sending push webhooks for tags (we have such logic in place for other providers), which leads me to believe this is a new feature in Gitea (likely) or that Drone has been broken since March (less likely). Either way we will require a patch to go-scm to ignore push hooks for tags for Gitea.

Hm - I changed the webhook to just fire on Create of tag or branch. Nothing happend - no build/deployment was fired in drone. Looks like I have a deep misunderstanding on how things are working together with drone and gitea… I just don’t understand how it was working before…

But the scenario build/deploy on tag created is supported by drone? Could you leed me to some docs as starting point?

Drone currently relies on the tag creation webhook from Gitea for tag events. Based on what you have posted, it appears that Gitea changed their webhooks in the latest version and is also sending push hooks for tag creation. This behavior is new. Any change to webhook behavior (payload, event type, etc) has the potential to negatively impact systems that process webhooks.

no actually I think there is a communication mismatch :slight_smile: I think they are sending both of them - push and create.

grafik

So I think drone should handle the create hook.

yes, we are saying the same thing. GitHub and GitLab send both a push hook and a tag create hook when you push a tag. Drone handles this. Previously Gitea only sent a single tag create hook. Sending two hooks is new behavior. Drone does not have any logic in place to account for this new behavior and ignore the push hook.

honestly, I think that was just due to my setup of custom events. I changed my settings in gitea:

Now its just firing the create webhook

And that event data looks quite similar to that one you pointed to in your first reply.

I am not sure I agree. The custom events should never need to be changed and Push webhooks should always be enabled, otherwise Drone will not receive webhooks when you push regular commits (e.g. git push origin master).

I am confident Drone previously did not received a Push hook for tag creation, otherwise we would have accounted for this in the code, just like we do for GitHub and GitLab. Based on what you have posted this is new behavior and requires a patch to the Drone code.

Instead of speculating, however, I think we should just ask @techknowlogick who is a Gitea maintainer and Drone contributor to confirm whether or not there are changes and how we should proceed.

1 Like

your the owner - and you know the system for sure a trillion times better than me. I just didn’t wanted to point you into wrong directions. But if your sure - as I said - you know better than me :slight_smile:

I think in this case @techknowlogick knows best and will be able to advise further :slight_smile:. Thanks for providing a detailed report and we will take it from here.

Yes, as @bradrydzewski has said, this is likely something to do with Gitea. I’m currently travelling, but can look into this more when I get back Tuesday.

The last change to webhook has been Aug 11. Which version of Gitea are you on (I see you are using :latest for Drone, is same for Gitea, if so then I’ll need specific commit to verify against)?

Gitea is also running on gitea:latest - which is also regularyly updated bei watchtower. Currently running on build Gitea Version: 1.10.0+dev-232-gf1c414882. Thanks for diving into…

ok. So I’ve gone back to v1.8.3 and confirmed that the webhooks are the same from there all the way to the most recent commit. I’ve confirmed that two webhooks are indeed sent (but that this behaviour is still the same for versions going back to 1.8.3), one for tag create and one for tag push.

Push webhook has following for a tag:

"ref": "refs/tags/v1.0.1",
  "before": "0000000000000000000000000000000000000000",
  "after": "3045cf09fe0f35412e1007201e29a3afec9a78ae",

and the create webhook has following for tag:

  "sha": "d184ac85ba01f077b7634eb293c870418b31cda8",
  "ref": "v1.0.1",
  "ref_type": "tag",

notice how the ref has a different format in both (this is likely what is cause invalid refspec as IIRC the ref is split on / and if the create doesn’t have a / then it might use an empty variable).

I think suggestion from above of changes to go-scm might be in order. I will review what has been done for GitHub/GitLab, and send a PR, likely it’ll be to ignore the “create” webhook.

@techknowlogick thanks for the effort you put into this!