External/encrypted secrets not working

Hi all

I am stuck! :joy:

I’ve probably gone through most relevant forum threads and I have surely gone through the documentation many times over (to the point I started noticing typos and made a couple of PRs).

I am writing a plugin for Kubernetes because existing solutions don’t cover my use case (my need was for 100% kubectl apply -f multi-config.yaml parity from inside the pipelines)

And while for my personal use case (in-cluster) works fine, I thought since there is no other plugin at the moment to solve the issue, make it work for out-of-cluster scenarios too (note: there is one open PR -adding drone-kubernetes-apply #264- that while it works, setting up a docker build containing kubectl is an overkill, while using Kubernetes API and Server Side Apply feels more proper solution).

Out-of-cluster scenarios need to use secrets (to store kubeconfig) and for the life of me I can not make them work!
I’d like to stress that I have the same problem for some time now and is not related to my plugin or current k8s cluster version. The only secrets I have ever managed to use are ones created via GUI. Even encrypted secrets created with drone CLI don’t work. The don’t work in the settings section of plugin using from_secret plus they don’t work with env variables either.

For example if I run a pipeline like this

kind: pipeline
type: kubernetes
name: default

steps:
- name: dron8s
  image: bh90210/dron8s:v0.0.16
  environment:
    PLUGIN_KUBECONFIG:
      from_secret: kc
  settings:
    yaml: ./k8s-deployment.yaml
    kubeconfig:
      from_secret: kc
  commands:
    - echo $PLUGIN_KUBECONFIG
    - echo $PLUGIN_YAML
---
kind: secret
name: kc
data: ZZFNHUI489053iojrgfg56df65df857sg[...]

I get

default — dron8s
00:02
+ echo $PLUGIN_KUBECONFIG

+ echo $PLUGIN_YAML
./k8s-deployment.yaml

My setup is entirely on Kubernetes. Server, runner & secrets are operating in the same cluster. Everything installed via provided Helm charts.

Relevant drone-runner-kube helm chart values

env:
  DRONE_RPC_SECRET: <password>
  DRONE_RPC_HOST: some.url.com
  DRONE_RPC_PROTO: https
  DRONE_NAMESPACE_DEFAULT: drone
  DRONE_SECRET_PLUGIN_ENDPOINT: drone-kubernetes-secrets.drone.svc.cluster.local:3000
  DRONE_SECRET_PLUGIN_TOKEN: <password>

and relevant drone-kubernetes-secrets helm chart values

rbac:
  enabled: true
  secretNamespace: default
env:
  SECRET_KEY: <password>
  KUBERNETES_NAMESPACE: default

The secret was created by running kubectl create secret generic dron8s --from-file=kubeconfig=$HOME/.kube/config

though if the error is in kube runner/kube secrets configs, this does not explain why plain encrypted option is not working either

and logs from each pod
server:

{"level":"info","msg":"main: internal scheduler enabled","time":"2020-10-20T23:35:18Z"}
{"interval":"30m0s","level":"info","msg":"starting the cron scheduler","time":"2020-10-20T23:35:18Z"}
{"interval":"24h0m0s","level":"info","msg":"starting the zombie build reaper","time":"2020-10-20T23:35:18Z"}
{"acme":false,"host":"https://some.url.com","level":"info","msg":"starting the http server","port":":80","proto":"https","time":"2020-10-20T23:35:18Z","url":"https://some.url.com"}

runner:

time="2020-10-21T01:21:16Z" level=info msg="starting the server" addr=":3000"
time="2020-10-21T01:21:16Z" level=info msg="successfully pinged the remote server"
time="2020-10-21T01:21:16Z" level=info msg="polling the remote server" capacity=100 endpoint="https://some.url.com" kind=pipeline type=kubernetes

secrets:

time="2020-10-21T00:33:31Z" level=info msg="server listening on address :3000"

everything seems normal.

I figured if this was a bug somewhere other people would complain here or in github issues but it seems to be just me. Am I missing something obvious? Please help.

Thanks in advance,
Byron

How did you encrypt the secret? Did you follow these instructions? Also note that encrypted secrets are not exposed to pull requests.

Also please enable trace logs for your runner. As you can see from the code, we have very detailed trace logs that can help you understand why encrypted secrets are not being exposed to your pipeline step:

1 Like

thanks for the fast reply!

I believe you are referring to https://docs.drone.io/server/storage/encryption/
and not the command bellow?

drone encrypt user/repo $(printf “%s” “$(<~/.kube/config)”)

In any case after enabling debug/trace I get:

level=debug msg="secret: encrypted: cannot decrypt" error="cipher: message authentication failed" kind=secret name=kubeconfig thread=75

I tried upgrading the chart with DRONE_DATABASE_SECRET but still the same error.

I had some success though, I managed to get Kubernetes Secrets working. I just generated new openssl passwords, replaced the old used ones, helm upgrade first the server chart (to pass DRONE_DATABASE_SECRET), then runner chart then the secrets chart and it just worked. I have no idea why.

EDIT: I didn’t mention it originally but for the record, before re-applying (helm charts via helm upgrade) everything I noticed that I was using the same password for both RPC and DRONE_SECRET_PLUGIN_TOKEN, dunno if this had to do with anything but feel I should mention it.

I am referring to https://docs.drone.io/secret/encrypted/. If you want to encrypt a secret in your yaml (as shown in your previous example) you must encrypt the secret using the drone encrypt command. This command sends your secrets to the Drone server and returns the encrypted value. Drone uses a per-repository encryption key to encrypt the secret. The per-repository encryption key is internally generated by Drone, at runtime.

the DRONE_DATABASE_SECRET is used to encrypt columns in the database. It is not related to using encrypted or external secrets and would not be relevant to what we are discussing in this thread.

I’m sorry if I wasn’t super clear

this is how I encrypted the secret
drone encrypt user/repo $(printf “%s” “$(<~/.kube/config)”)

and this is the relevant log

level=debug msg="secret: encrypted: cannot decrypt" error="cipher: message authentication failed" kind=secret name=kubeconfig thread=75

hmm, not sure, I was unable to reproduce the error. Here are the steps I took:

first I encrypted the secret from a file:

$ drone encrypt drone/hello-world @/path/to/secret.txt
qT0IGWVhUz7RAxYo5pXSEYOiTqMqjbVs9BHQ8LY49s7oRoj4H+6JPA==

then I added the encrypted secret to my yaml [1]:

then I executed a build and I can see the secret was correctly injected when I print to the console (we know it was injected because the secret is masked when printed) [2]

this feature is quite mature and has no known issues. The only reasons I can think of for failure to decrecypt would be:

  1. copy / paste issues
  2. encrypting a secret for repository A, then trying to use with repository B
  3. encrypting a secret with one Drone instance (e.g drone.company.com), and then trying to use with a different Drone instance (e.g. cloud.drone.io)

[1] https://github.com/drone/hello-world/blob/eb84da152fc9980db58ef2e74ab22d91e9922faf/.drone.yml#L16
[2] https://cloud.drone.io/drone/hello-world/192/1/2

1 Like

omg you are (obvsly) correct!
I was, again mindlessly creating the secret for wrong repo!
solved, and I am so sorry for opening another thread.

EDIT: I guess the take away for future people bumping on this thread is don’t be like me, pay some attention at what you type. :joy: