Problem using docker plugin with drone v1.0.0

#1

Using docker plugin with drone v1.0.0, error message is:

+ /usr/local/bin/docker push ********/myimage:latest
The push refers to repository [docker.io/********/myimage]
b1d96c528f3f: Preparing
... [more lines like this]
denied: requested access to the resource is denied

My complete .drone.yml

kind: pipeline
name: build_publish

steps:
- name: publish
  pull: always
  image: plugins/docker:linux-amd64
  settings:
    auto_tag: true
    repo: myorg/myimage
    username: 
      from_secret: docker_username
    password:
      from_secret: docker_password

docker_username and docker_password are configured as repo secrets and I’ve checked and recreated them multiple times, and manually logged in with the same credentials to check.

This job is triggered by a push to master branch.

I’m completely stumped, so please point out my mistake.

Cheers

#2

Can you please post the full output for the Docker plugin step. The plugin outputs additional information for debugging purposes that can be used to triage.

#3

This is OP - I posted the log but it was blocked or something? Could you please check? thanks.

#4
+ /usr/local/bin/dockerd --data-root /var/lib/docker
Registry credentials not provided. Guest mode enabled.
+ /usr/local/bin/docker version
Client: Docker Engine - Community
Version: 18.09.3
API version: 1.39
Go version: go1.10.8
Git commit: 774a1f4
Built: Thu Feb 28 06:32:01 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.3
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: 774a1f4
Built: Thu Feb 28 06:40:51 2019
OS/Arch: linux/amd64
Experimental: false
+ /usr/local/bin/docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 18.09.3
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: e6b3f5632f50dbc4e9cb6288d911bf4f5e95b18e
runc version: 6635b4f0c6af3810594d2770f662f34ddc15b40d
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.15.0-46-generic
Operating System: Alpine Linux v3.9 (containerized)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.852GiB
Name: f7e1563eeed2
ID: TUUY:KIBR:AANM:6S6Z:KTT4:T6LX:6E2V:NGLQ:VSCB:ETWN:SIS7:432W
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: No swap limit support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
+ /usr/local/bin/docker build --rm=true -f Dockerfile -t c4389ffeb3a05107e5117c1f1a7463f83b578ddf . --pull=true --label org.label-schema.schema-version=1.0 --label org.label-schema.build-date=2019-03-29T08:33:56Z --label org.label-schema.vcs-ref=c4389ffeb3a05107e5117c1f1a7463f83b578ddf --label org.label-schema.vcs-url=https://gitlab.com/********rg/myapp.git
Sending build context to Docker daemon 119.8kB
Step 1/13 : FROM node:8.11.4-stretch
8.11.4-stretch: Pulling from library/node
05d1a5232b46: Pulling fs layer
5cee356eda6b: Pulling fs layer
89d3385f0fd3: Pulling fs layer
80ae6b477848: Pulling fs layer
28bdf9e584cc: Pulling fs layer
0fce7df8ad89: Pulling fs layer
f79e9df6a92d: Pulling fs layer
a4eb593de75c: Pulling fs layer
7a04b6827691: Pulling fs layer
80ae6b477848: Waiting
28bdf9e584cc: Waiting
0fce7df8ad89: Waiting
f79e9df6a92d: Waiting
a4eb593de75c: Waiting
7a04b6827691: Waiting
89d3385f0fd3: Verifying Checksum
89d3385f0fd3: Download complete
5cee356eda6b: Verifying Checksum
5cee356eda6b: Download complete
05d1a5232b46: Verifying Checksum
05d1a5232b46: Download complete
0fce7df8ad89: Verifying Checksum
0fce7df8ad89: Download complete
80ae6b477848: Verifying Checksum
80ae6b477848: Download complete
f79e9df6a92d: Verifying Checksum
f79e9df6a92d: Download complete
a4eb593de75c: Verifying Checksum
a4eb593de75c: Download complete
7a04b6827691: Verifying Checksum
7a04b6827691: Download complete
28bdf9e584cc: Verifying Checksum
28bdf9e584cc: Download complete
05d1a5232b46: Pull complete
5cee356eda6b: Pull complete
89d3385f0fd3: Pull complete
80ae6b477848: Pull complete
28bdf9e584cc: Pull complete
0fce7df8ad89: Pull complete
f79e9df6a92d: Pull complete
a4eb593de75c: Pull complete
7a04b6827691: Pull complete
Digest: sha256:3a45a0c2e53aefa5f93555101b354c4603971e06d973ceb3d2d6b32922068bb8
Status: Downloaded newer image for node:8.11.4-stretch
---> a74cf1832f44
Step 2/13 : ENV BUILD_PACKAGES="build-essential python gcc g++ git curl tar bzip2" NODE_ENV=production
---> Running in 026858eeedb4
Removing intermediate container 026858eeedb4
---> bd57b94ddb53
Step 3/13 : WORKDIR /
---> Running in 97c93805cf45
Removing intermediate container 97c93805cf45
---> 30c9359bfc6e
Step 4/13 : RUN apt-get -y update && apt-get -y install ${BUILD_PACKAGES} && npm install -g node-gyp && node-gyp install
---> Running in 46cf8a2dde3f
Ign:1 http://deb.debian.org/debian stretch InRelease
Get:2 http://security.debian.org/debian-security stretch/updates InRelease [94.3 kB]
Get:3 http://deb.debian.org/debian stretch-updates InRelease [91.0 kB]
Get:4 http://deb.debian.org/debian stretch Release [118 kB]
Get:5 http://deb.debian.org/debian stretch Release.gpg [2434 B]
Get:6 http://security.debian.org/debian-security stretch/updates/main amd64 Packages [481 kB]
Get:7 http://deb.debian.org/debian stretch-updates/main amd64 Packages [11.1 kB]
Get:8 http://deb.debian.org/debian stretch/main amd64 Packages [7084 kB]
Fetched 7881 kB in 2s (3679 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
bzip2 is already the newest version (1.0.6-8.1).
g++ is already the newest version (4:6.3.0-4).
gcc is already the newest version (4:6.3.0-4).
python is already the newest version (2.7.13-2).
python set to manually installed.
tar is already the newest version (1.29b-1.1).
tar set to manually installed.
The following additional packages will be installed:
libcurl3 libcurl4-openssl-dev
Suggested packages:
gettext-base git-daemon-run | git-daemon-sysvinit git-doc git-el git-email
git-gui gitk gitweb git-arch git-cvs git-mediawiki git-svn libcurl4-doc
libcurl3-dbg libidn11-dev libldap2-dev librtmp-dev libssh2-1-dev
libssl1.0-dev | libssl-dev
Recommended packages:
less rsync
The following NEW packages will be installed:
build-essential
The following packages will be upgraded:
curl git libcurl3 libcurl4-openssl-dev
4 upgraded, 1 newly installed, 0 to remove and 68 not upgraded.
Need to get 5068 kB of archives.
After this operation, 33.8 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian stretch/main amd64 git amd64 1:2.11.0-3+deb9u4 [4167 kB]
Get:2 http://deb.debian.org/debian stretch/main amd64 build-essential amd64 12.3 [7346 B]
Get:3 http://deb.debian.org/debian stretch/main amd64 libcurl4-openssl-dev amd64 7.52.1-5+deb9u9 [374 kB]
Get:4 http://deb.debian.org/debian stretch/main amd64 curl amd64 7.52.1-5+deb9u9 [227 kB]
Get:5 http://deb.debian.org/debian stretch/main amd64 libcurl3 amd64 7.52.1-5+deb9u9 [292 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 5068 kB in 0s (36.3 MB/s)
(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 29649 files and directories currently installed.)
Preparing to unpack .../git_1%3a2.11.0-3+deb9u4_amd64.deb ...
Unpacking git (1:2.11.0-3+deb9u4) over (1:2.11.0-3+deb9u3) ...
Selecting previously unselected package build-essential.
Preparing to unpack .../build-essential_12.3_amd64.deb ...
Unpacking build-essential (12.3) ...
Preparing to unpack .../libcurl4-openssl-dev_7.52.1-5+deb9u9_amd64.deb ...
Unpacking libcurl4-openssl-dev:amd64 (7.52.1-5+deb9u9) over (7.52.1-5+deb9u6) ...
Preparing to unpack .../curl_7.52.1-5+deb9u9_amd64.deb ...
Unpacking curl (7.52.1-5+deb9u9) over (7.52.1-5+deb9u6) ...
Preparing to unpack .../libcurl3_7.52.1-5+deb9u9_amd64.deb ...
Unpacking libcurl3:amd64 (7.52.1-5+deb9u9) over (7.52.1-5+deb9u6) ...
Setting up build-essential (12.3) ...
Setting up libcurl3:amd64 (7.52.1-5+deb9u9) ...
Processing triggers for libc-bin (2.24-11+deb9u3) ...
Setting up libcurl4-openssl-dev:amd64 (7.52.1-5+deb9u9) ...
Setting up git (1:2.11.0-3+deb9u4) ...
Setting up curl (7.52.1-5+deb9u9) ...
/usr/local/bin/node-gyp -> /usr/local/lib/node_modules/node-gyp/bin/node-gyp.js
+ node-gyp@3.8.0
added 97 packages in 4.775s
gyp info it worked if it ends with ok
gyp info using node-gyp@3.8.0
gyp info using node@8.11.4 | linux | x64
gyp http GET https://nodejs.org/download/release/v8.11.4/node-v8.11.4-headers.tar.gz
gyp http 200 https://nodejs.org/download/release/v8.11.4/node-v8.11.4-headers.tar.gz
gyp http GET https://nodejs.org/download/release/v8.11.4/SHASUMS256.txt
gyp http 200 https://nodejs.org/download/release/v8.11.4/SHASUMS256.txt
8.11.4
gyp info ok
Removing intermediate container 46cf8a2dde3f
---> acfaa2cc92ca
Step 5/13 : COPY scripts /scripts
---> 12674048750e
Step 6/13 : ONBUILD COPY .build /
---> Running in b5d0b3342386
Removing intermediate container b5d0b3342386
---> 395634078782
Step 7/13 : ONBUILD RUN /scripts/build.sh && /scripts/rebuild_bin_npm_modules.sh && /scripts/clean.sh
---> Running in 28ebefe26ac5
Removing intermediate container 28ebefe26ac5
---> 2a6be3b0a0c5
Step 8/13 : EXPOSE 80
---> Running in 6658cc3b5416
Removing intermediate container 6658cc3b5416
---> 379a1b1b3943
Step 9/13 : CMD ["/bin/bash", "/scripts/run.sh"]
---> Running in 84df053da9ea
Removing intermediate container 84df053da9ea
---> 2dd213047f5b
Step 10/13 : LABEL org.label-schema.build-date=2019-03-29T08:33:56Z
---> Running in 0bd88ab11ebb
Removing intermediate container 0bd88ab11ebb
---> 51de1e6c3301
Step 11/13 : LABEL org.label-schema.schema-version=1.0
---> Running in a45800b0af4f
Removing intermediate container a45800b0af4f
---> a7054bc07433
Step 12/13 : LABEL org.label-schema.vcs-ref=c4389ffeb3a05107e5117c1f1a7463f83b578ddf
---> Running in fd291fbcfb67
Removing intermediate container fd291fbcfb67
---> dc3c4e11ba3b
Step 13/13 : LABEL org.label-schema.vcs-url=https://gitlab.com/********rg/myapp.git
---> Running in 93acdce5d4d3
Removing intermediate container 93acdce5d4d3
---> ffb477950b8e
Successfully built ffb477950b8e
Successfully tagged c4389ffeb3a05107e5117c1f1a7463f83b578ddf:latest
+ /usr/local/bin/docker tag c4389ffeb3a05107e5117c1f1a7463f83b578ddf ********/myapp:latest
+ /usr/local/bin/docker push ********/myapp:latest
The push refers to repository [docker.io/********/myapp]
3dd9222d691e: Preparing
1577b3fc1b5f: Preparing
c863f17383b1: Preparing
eaa5f62d5199: Preparing
b1d96c528f3f: Preparing
52e3da8961fa: Preparing
a19cb627cc73: Preparing
ab016c9ea8f8: Preparing
2eb1c9bfc5ea: Preparing
0b703c74a09c: Preparing
b28ef0b6fef8: Preparing
52e3da8961fa: Waiting
a19cb627cc73: Waiting
ab016c9ea8f8: Waiting
2eb1c9bfc5ea: Waiting
0b703c74a09c: Waiting
b28ef0b6fef8: Waiting
denied: requested access to the resource is denied
time="2019-03-29T08:34:59Z" level=fatal msg="exit status 1"
#5

this is the key line in the logs, indicating the credentials are not provided. The root cause for this is typically related to configuration. Please provide the following details to help pinpoint the issue:

  1. version of Drone you are using
  2. the output of drone secret ls <repo> for your repository.
  3. provide the output of drone build info <repo> <build_number> for the failing build.
#8

Hi @bradrydzewski - thanks for the help on this.

I have no idea what happened but while poking around trying to debug, it’s now pushed successfully. I probably screwed up somewhere along the way…

#9

I have this problem “sometimes”. The workaround is to restart this build and 30% chance to success push.

  1. version of Drone you are using
    1.1.0
  2. the output of drone secret ls <repo> for your repository.

google_credentials
Pull Request Read: false
Pull Request Write: false

  1. provide the output of drone build info <repo> <build_number> for the failing build.
    Failed:
+ /usr/local/bin/dockerd --data-root /var/lib/docker
Registry credentials not provided. Guest mode enabled.
+ /usr/local/bin/docker version
Client: Docker Engine - Community
Version: 18.09.4
API version: 1.39
Go version: go1.10.8
Git commit: d14af54
Built: Wed Mar 27 18:33:40 2019

Success

+ /usr/local/bin/dockerd --data-root /var/lib/docker
+ /usr/local/bin/docker version
Client: Docker Engine - Community
Version: 18.09.4
API version: 1.39
Go version: go1.10.8
Git commit: d14af54
Built: Wed Mar 27 18:33:40 2019
#10

@hawkhsieh your post is still missing the following:

  1. the results of drone build info <repo> <build_number> when the step fails
  2. your yaml file

Once the requested information is available, we can hep you debug. This is almost certainly a configuration issue. We know this because the following error indicates the secrets are empty:

Registry credentials not provided. Guest mode enabled.

This could be caused by yaml misconfiguration or secret misconfiguration, or both. The two items I requested above will help determine which.

#11

@bradrydzewski
Number: 65
Status: failure
Event: push
Commit: 21ffc11d4c655971d95ad4e23d4091dcdb519113
Branch: master
Ref: refs/heads/master
Author: hawkhsieh hawk.hsieh@gmail.com
Message: fix problem

ymlfile

pipeline:
  build_image:
    image: plugins/gcr
    registry: asia.gcr.io
    repo: mygoogle/test
    tags:
      - latest
      - "${DRONE_COMMIT_SHA}"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    secrets: [google_credentials]
    when:
      event: push
      branch: master
#12

In your example, you repo name must be the fully qualified name. For example asia.gcr.io/mygoogle/test

#13

According to this http://plugins.drone.io/drone-plugins/drone-gcr/, the repo name does not include asia.gcr.io.

BTW. Once I replace the
secrets: [google_credentials]
to
from_secret: google_credentials

This problem seems not to happen anymore.