Drone

Drone-migrate update-repos fails with "no such file or directory"

Hi there

I’m trying to migrate from Drone 0.8 to 1.2 using the drone/drone-migrate tool and in the last step I keep getting this error:

[ec2-user@ip-10-0-0-123 ~]$ ./drone-migrate.sh --debug update-repos
DEBU[0000] target database driver: mysql                
DEBU[0000] target database datasource: drone:...@tcp(drone....rds.amazonaws.com:3306)/drone?parseTime=true 
DEBU[0000] scm driver: stash                            
DEBU[0000] scm server: https://bitbucket.example.com/repos 
FATA[0000] open : no such file or directory             
[ec2-user@ip-10-0-0-123 ~]$

All the previous steps worked fine, only this one fails.

How can I fix it or work around it?

We’re using BitBucket Server 5.11 (i.e. self-hosted, not cloud).

Thanks!

did you provide the migration utility with the private key file? The particular step is the only step that uses the private key file, and open : no such file or directory tells me there was a problem reading it.

 SCM_DRIVER=stash 
 SCM_SERVER=https://stash.company.com
 STASH_CONSUMER_KEY=OauthKey
+STASH_PRIVATE_KEY_FILE=/path/to/private/key.pem

If you set the private key file path, and you are using the drone-migrate docker image, also make sure you mount the private key file (using a docker volume) otherwise it cannot be accessed from inside the container.

--volume=/path/to/private/key.pem:/path/to/private/key.pem

Good point, I put the key inline at first as I’ve taken it from the drone config DRONE_STASH_CONSUMER_RSA_STRING='-----BEGIN RSA PRIVATE KEY-----\nMIICXAIBAAK...'

Now I put the key in a file but it’s still the same. To make sure it’s correctly mounted in the container I start it from the built-in bash:

[laptop] ~ $ docker run -it --rm \
-e SOURCE_DATABASE_DRIVER=mysql \
-e TARGET_DATABASE_DRIVER=mysql \
-e 'SOURCE_DATABASE_DATASOURCE=drone:.../dronedev?parseTime=true' \
-e 'TARGET_DATABASE_DATASOURCE=drone:.../drone2?parseTime=true' \
-e DRONE_SERVER="http://drone-2.{example.com}" \
-e SCM_DRIVER=stash \
-e SCM_SERVER="https://bitbucket.{example.com}/repos" \
-e STASH_CONSUMER_KEY=... 
-e STASH_PRIVATE_KEY_FILE=/private-key-file.pem \
-v ${PWD}/private-key-file.pem:/private-key-file.pem \
--entrypoint /bin/sh drone/migrate

## Make sure the key is readable
[docker-migrate] ~ $ cat /private-key-file.pem
-----BEGIN RSA PRIVATE KEY-----
MIICXAIBAAK...
...
-----END RSA PRIVATE KEY-----

## Run drone-migrate
[docker-migrate] ~ $ /bin/drone-migrate --stash-private-key-file /private-key-file.pem \
                                      --debug update-repos 
DEBU[0000] target database driver: mysql                
DEBU[0000] target database datasource: drone:.../drone2?parseTime=true 
DEBU[0000] scm driver: stash                            
DEBU[0000] scm server: https://bitbucket..../repos 
FATA[0000] open : no such file or directory           
  
[docker-migrate] ~ $

What now?

BTW it would be great if it printed the filename it couldn’t open…

I think the next step would be to look at the code to triage further. If you see an issue please consider sending a pull request. Or a pull request to update the docs if they can be improved.

I managed to fix this issue - the code was using c.String("stash-private-key-file") which returned empty string. When changed to c.GlobalString(...) it works.
Here is the pull request: https://github.com/drone/drone-migrate/pull/14

However I’m still seeing some issues:

DEBU[0000] private key file: /home/ec2-user/private-key-file.pem
INFO[0000] updating repository metadata
ERRO[0000] failed to get remote repository   error="%!v(PANIC=runtime error: index out of range)" owner=xyz repo=CCC/ami-ec2
ERRO[0000] failed to get remote repository   error="Not Authorized" owner=abc repo=CCC/ami-ecs
...
INFO[0000] repository metadata update complete

Ok, fixed that PANIC too. Apparently all the URL paths in go-scm/scm/driver/stash/repo.go do not start with the leading / and that makes the requests fail. This helps:

index e4d4e37..3fb3852 100644
--- a/scm/driver/stash/repo.go
+++ b/scm/driver/stash/repo.go
@@ -99,7 +99,7 @@ type repositoryService struct {
 // Find returns the repository by name.
 func (s *repositoryService) Find(ctx context.Context, repo string) (*scm.Repository, *scm.Response, error) {
        namespace, name := scm.Split(repo)
-       path := fmt.Sprintf("rest/api/1.0/projects/%s/repos/%s", namespace, name)
+       path := fmt.Sprintf("/rest/api/1.0/projects/%s/repos/%s", namespace, name)
        out := new(repository)
        res, err := s.client.do(ctx, "GET", path, nil, out)
        return convertRepository(out), res, err

It must be done across all the Sprintf() calls in the file. Maybe there is a better way to do that. Perhaps check in scm/driver/stash/stash.go -> do() whether the path starts with / and if not prefix it?

I won’t make a pull request for this as I don’t know what’s the right way to fix it.

Hmm, no leading slash in the path is required for me to use Drone successfully with Bitbucket Server. Note that github, gogs, gitea, etc do not use a leading slash either [1].

Perhaps check in scm/driver/stash/stash.go -> do() whether the path starts with / and if not prefix it?

The code accounts for a slash in the path [2][3] when the client is created. If the base path is empty or does not finish with a trailing slash one is added. The base path (trailing slash) is joined with the endpoint path (no leading slash).

I did notice your scm server address includes /repos in the path which seems abnormal. Perhaps this should be removed?

-https://bitbucket.{example.com}/repos
+https://bitbucket.{example.com}

[1] https://github.com/drone/go-scm/blob/master/scm/driver/github/repo.go#L66
[2] https://github.com/drone/go-scm/blob/master/scm/driver/stash/stash_test.go
[3] https://github.com/drone/go-scm/blob/master/scm/driver/stash/stash.go#L30