Kubernetes External Secrets not work


#1

Hi, so glad that the 1.0.0-rc.5 released, I tired to enable the global secrets feature in my project but not work, please let me know if I missed something.

Here is the k8s deploy file:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: drone-secrets
  namespace: default
  labels:
    app: drone-secrets
spec:
  replicas: 1
  selector:
    matchLabels:
      app: drone-secrets
  template:
    metadata:
      labels:
        app: drone-secrets
    spec:
      containers:
        - name: drone-secrets
          image: drone/kubernetes-secrets
          env:
            - name: SECRET_KEY
              value: hex_string
          ports:
            - name: http
              containerPort: 3000
      restartPolicy: Always

---
kind: Service
apiVersion: v1
metadata:
  name: drone-secrets-service
  namespace: default
spec:
  selector:
    app: drone-secrets
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000
      name: http

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: drone-server
  namespace: default
  labels:
    app: drone-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: drone-server
  template:
    metadata:
      labels:
        app: drone-server
    spec:
      containers:
        - name: drone-server
          image: drone/drone:1.0.0-rc.5
          env:
            - name: DRONE_KUBERNETES_ENABLED
              value: "true"
            - name: DRONE_KUBERNETES_NAMESPACE
              value: default
            - name: DRONE_SECRET_SECRET
              value: hex_string
            - name: DRONE_SECRET_ENDPOINT
              value: http://drone-secrets-service

Here is the k8s secrets:

---
apiVersion: v1
kind: Secret
type: Opaque
data:
  username: base64_string
  password: base64_string
metadata:
  name: drone-secrets

Here is the drone.yml

kind: pipeline
name: deploy

steps:
- name: pre-check
  image: docker
  environment:
    USERNAME:
      from_secret: username
    PASSWORD:
      from_secret: password
  commands:
    - echo $USERNAME
    - echo $PASSWORD

---
kind: secret

external_data:
  username:
    path: drone-secrets
    name: username
  password:
    path: drone-secrets
    name: password

Finally, I didnt get the secret print on console, output is blank

+ echo $USERNAME

+ echo $PASSWORD


#2

I checked the drone/kubernetes-secrets container output, log as

kubectl logs drone-secrets-7f4469dcf5-kqr5x 
time="2019-01-25T03:05:18Z" level=info msg="server listening on address :3000"

it looks like no request received.


Drone on k8s with k8s secrets plugin
#3

Finally it works on v1.0.0-rc.6, greate job!


#4

@AlloVince: Was there anything you changed or only the version? I have the same problem.

I have a very similar setup and everything in the “drone” namespace. Builds are working, but for secrets… they are empty and as I see it does not even try to fetch.

Drone: drone/drone:1.0.0-rc.6
drone-kubernetes-secrets: drone/kubernetes-secrets:latest

I tried if drone can even see the secret service:
> kubectl exec -n drone -it drone-prod-with-random-letters /bin/ash
/ # wget -O - http://drone-secrets
Connecting to drone-secrets (10.245.128.192:80)
wget: server returned error: HTTP/1.1 400 Bad Request

And if I do that, I can see:
time="2019-03-12T17:31:18Z" level=debug msg="secrets: invalid or missing signature in http.Request"

but I see only errors I triggered with wget and nothing else.

relevant env for drone:

- name: DRONE_SECRET_SECRET
  valueFrom:
    secretKeyRef:
      name: drone-config
      key: DRONE_SECRET_SECRET
- name: DRONE_SECRET_ENDPOINT
  value: http://drone-secrets

Env for drone-kubernetes-secrets:

  env:
    - name: KUBERNETES_NAMESPACE
      value: drone
    - name: DEBUG
      value: "1"
    - name: SECRET_KEY
      valueFrom:
        secretKeyRef:
          name: drone-config
          key: DRONE_SECRET_SECRET

everything is under namespace drone.

Edit:
I tried without setting KUBERNETES_NAMESPACE to drone but got the same results and I think it does not even asks the service so does not matter what is the value.

Meanwhile I checked all the envs for job pods too and they have the same settings (with kubectl describe pod).

Edit no2:
I tried to run drone plugins secret with port-forwarded secrets. I set up both (export) DRONE_SECRET_ENDPOINT and DRONE_SECRET_SECRET with a simple copy-paste logic (to be sure it’s the same).

kubectl port-forward -n drone drone-secrets-5487f5cb6-rqdmr 3000:3000

I tried to request a totally random secret bcos why not:

time="2019-03-12T18:09:17Z" level=debug msg="secrets: cannot find secret : invalid or missing secret name"

I got a request. So somehow drone does not even try to connect there (not even with wrong SECRET).


#5

yes, there were a couple of syntax changes described in the release notes 1.0.0-rc.6 release notes

we also published a new kubernetes docker image. The previous docker image was outdated and had not received weeks (months?) of changes.

wget -O - http://drone-secrets
Connecting to drone-secrets (10.245.128.192:80)
wget: server returned error: HTTP/1.1 400 Bad Request

also fwiw this is expected. The endpoint is protected for security reasons and is not going to respond to a curl request that lacks authentication credentials and signatures, and lacks a valid body.


#6

Oh that makes sense XD I tried to read all the stuff I found… .except the release note and it’s kinda important change under “Breaking Changes”

Thanks you, when I’m done at work, i’ll update my files based on the release note…

An important part to highlight (if anyone lands on this thread later):

Before

kind: secret

external_data:
  docker_username:
    path: secret/data/docker
    name: username
  docker_password:
    path: secret/data/docker
    name: password

After

---
kind: secret
name: docker_username
get:
  path: secret/data/username
  name: username

---
kind: secret
name: docker_password
get:
  path: secret/data/docker
  name: password

For the curl response: Yeah, I was sure it’s the expected response without secret_key


#7

and it works \o/ the solution was to use the right format for secrets in .drone.yml