Kubernetes External Secrets not work

#1

Hi, so glad that the 1.0.0-rc.5 released, I tired to enable the global secrets feature in my project but not work, please let me know if I missed something.

Here is the k8s deploy file:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: drone-secrets
  namespace: default
  labels:
    app: drone-secrets
spec:
  replicas: 1
  selector:
    matchLabels:
      app: drone-secrets
  template:
    metadata:
      labels:
        app: drone-secrets
    spec:
      containers:
        - name: drone-secrets
          image: drone/kubernetes-secrets
          env:
            - name: SECRET_KEY
              value: hex_string
          ports:
            - name: http
              containerPort: 3000
      restartPolicy: Always

---
kind: Service
apiVersion: v1
metadata:
  name: drone-secrets-service
  namespace: default
spec:
  selector:
    app: drone-secrets
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000
      name: http

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: drone-server
  namespace: default
  labels:
    app: drone-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: drone-server
  template:
    metadata:
      labels:
        app: drone-server
    spec:
      containers:
        - name: drone-server
          image: drone/drone:1.0.0-rc.5
          env:
            - name: DRONE_KUBERNETES_ENABLED
              value: "true"
            - name: DRONE_KUBERNETES_NAMESPACE
              value: default
            - name: DRONE_SECRET_SECRET
              value: hex_string
            - name: DRONE_SECRET_ENDPOINT
              value: http://drone-secrets-service

Here is the k8s secrets:

---
apiVersion: v1
kind: Secret
type: Opaque
data:
  username: base64_string
  password: base64_string
metadata:
  name: drone-secrets

Here is the drone.yml

kind: pipeline
name: deploy

steps:
- name: pre-check
  image: docker
  environment:
    USERNAME:
      from_secret: username
    PASSWORD:
      from_secret: password
  commands:
    - echo $USERNAME
    - echo $PASSWORD

---
kind: secret

external_data:
  username:
    path: drone-secrets
    name: username
  password:
    path: drone-secrets
    name: password

Finally, I didnt get the secret print on console, output is blank

+ echo $USERNAME

+ echo $PASSWORD

#2

I checked the drone/kubernetes-secrets container output, log as

kubectl logs drone-secrets-7f4469dcf5-kqr5x 
time="2019-01-25T03:05:18Z" level=info msg="server listening on address :3000"

it looks like no request received.

Drone on k8s with k8s secrets plugin
#3

Finally it works on v1.0.0-rc.6, greate job!

#4

@AlloVince: Was there anything you changed or only the version? I have the same problem.

I have a very similar setup and everything in the “drone” namespace. Builds are working, but for secrets… they are empty and as I see it does not even try to fetch.

Drone: drone/drone:1.0.0-rc.6
drone-kubernetes-secrets: drone/kubernetes-secrets:latest

I tried if drone can even see the secret service:
> kubectl exec -n drone -it drone-prod-with-random-letters /bin/ash
/ # wget -O - http://drone-secrets
Connecting to drone-secrets (10.245.128.192:80)
wget: server returned error: HTTP/1.1 400 Bad Request

And if I do that, I can see:
time="2019-03-12T17:31:18Z" level=debug msg="secrets: invalid or missing signature in http.Request"

but I see only errors I triggered with wget and nothing else.

relevant env for drone:

- name: DRONE_SECRET_SECRET
  valueFrom:
    secretKeyRef:
      name: drone-config
      key: DRONE_SECRET_SECRET
- name: DRONE_SECRET_ENDPOINT
  value: http://drone-secrets

Env for drone-kubernetes-secrets:

  env:
    - name: KUBERNETES_NAMESPACE
      value: drone
    - name: DEBUG
      value: "1"
    - name: SECRET_KEY
      valueFrom:
        secretKeyRef:
          name: drone-config
          key: DRONE_SECRET_SECRET

everything is under namespace drone.

Edit:
I tried without setting KUBERNETES_NAMESPACE to drone but got the same results and I think it does not even asks the service so does not matter what is the value.

Meanwhile I checked all the envs for job pods too and they have the same settings (with kubectl describe pod).

Edit no2:
I tried to run drone plugins secret with port-forwarded secrets. I set up both (export) DRONE_SECRET_ENDPOINT and DRONE_SECRET_SECRET with a simple copy-paste logic (to be sure it’s the same).

kubectl port-forward -n drone drone-secrets-5487f5cb6-rqdmr 3000:3000

I tried to request a totally random secret bcos why not:

time="2019-03-12T18:09:17Z" level=debug msg="secrets: cannot find secret : invalid or missing secret name"

I got a request. So somehow drone does not even try to connect there (not even with wrong SECRET).

#5

yes, there were a couple of syntax changes described in the release notes 1.0.0-rc.6 release notes

we also published a new kubernetes docker image. The previous docker image was outdated and had not received weeks (months?) of changes.

wget -O - http://drone-secrets
Connecting to drone-secrets (10.245.128.192:80)
wget: server returned error: HTTP/1.1 400 Bad Request

also fwiw this is expected. The endpoint is protected for security reasons and is not going to respond to a curl request that lacks authentication credentials and signatures, and lacks a valid body.

1 Like
#6

Oh that makes sense XD I tried to read all the stuff I found… .except the release note and it’s kinda important change under “Breaking Changes”

Thanks you, when I’m done at work, i’ll update my files based on the release note…

An important part to highlight (if anyone lands on this thread later):

Before

kind: secret

external_data:
  docker_username:
    path: secret/data/docker
    name: username
  docker_password:
    path: secret/data/docker
    name: password

After

---
kind: secret
name: docker_username
get:
  path: secret/data/username
  name: username

---
kind: secret
name: docker_password
get:
  path: secret/data/docker
  name: password

For the curl response: Yeah, I was sure it’s the expected response without secret_key

#7

and it works \o/ the solution was to use the right format for secrets in .drone.yml

#8

If this can help someone, I just spent 2h to figure out this:

  • you need to add a drone/kubernetes-secrets service in your Drone v1 deployment if you want to access K8s secret. Drone server doesn’t read K8s secrets out of the box
  • on this pod, you need to specify KUBERNETES_NAMESPACE=cicd if your not in default
  • the drone controller K8s job will contact this service to retrieve the K8s secret
  • you need to inform drone server (which will provide this info to the build controller) the DRONE_SECRET_ENDPOINT. This field needs to be in the form http://[kubernetes-secrets-service]. http:// needs to be here.
  • If you print your secret on the build log, don’t be surprise if you see *******, that’s actually a nice feature of the UI for obfuscation. But if you jump in the container, your secret is here ! :slight_smile:
2 Likes
#9

actually, it doesn’t work. what i miss?
here is my post.

#11

Hi,

the drone controller K8s job will contact this service to retrieve the K8s secret

The secret service is contacted by the Drone Job at the run time, not by Drone server itself.

This won’t work:

            - name: DRONE_SECRET_ENDPOINT
              value: http://localhost:3000

You need to expose the secret port via a K8s service, and make sure the Drone Job will be set to contact this service/port.

Based on your manifests, it seems you are running k8s-secret manager in the same pod of drone itself, which “should” be ok.

#14

as your suggestion, i split the drone manifests into 2 files.
drone-secrets.yaml

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: drone
subjects:
  - kind: ServiceAccount
    name: default
    namespace: cd
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: drone-secrets
  namespace: cd
  labels:
    app: drone-secrets
spec:
  replicas: 1
  selector:
    matchLabels:
      app: drone-secrets
  template:
    metadata:
      labels:
        app: drone-secrets
    spec:
      containers:
      - name: drone-secrets
        image: drone/kubernetes-secrets:latest
        imagePullPolicy: Always
        env:
        - name: SECRET_KEY
          value: 558f3eacbfd5928157cbfe34823ab921
        ports:
        - name: http
          containerPort: 3000
      restartPolicy: Always

---
kind: Service
apiVersion: v1
metadata:
  name: drone-secrets
  namespace: cd
spec:
  selector:
    app: drone-secrets
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000
      name: http

drone.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: drone
  namespace: cd
  labels:
    app: drone
spec:
  replicas: 1
  selector:
    matchLabels:
      app: drone
  template:
    metadata:
      labels:
        app: drone
    spec:
      containers:
      - name: drone
        image: drone/drone:1
        imagePullPolicy: Always
        env:
        - name: DRONE_GIT_ALWAYS_AUTH
          value: "false"
        - name: DRONE_RUNNER_CAPACITY
          value: "2"
        - name: DRONE_TLS_AUTOCERT
          value: "false"
        - name: DRONE_KUBERNETES_ENABLED
          value: "true"
        - name: DRONE_KUBERNETES_NAMESPACE
          value: cd
        - name: DRONE_GITLAB_CLIENT_ID
          value: dabde6f82c009320483267cf565e32a1ec82318310ced47fa6574eee0bccfa5f
        - name: DRONE_GITLAB_SERVER
          value: http://gitland.your.com
        - name: DRONE_GITLAB_CLIENT_SECRET
          value: b9cf0f5c4206a1c19fbfec40c84056e3a5e6432a71450866fc7f345bc2642664
        - name: DRONE_SERVER_HOST
          value: <forget it, just because i am the new user.>
        - name: DRONE_SERVER_PROTO
          value: http
        - name: DRONE_DATABASE_DRIVER
          value: sqlite3
        - name: DRONE_DATABASE_DATASOURCE
          value: "/drone/drone.sqlite"
        - name: DRONE_USER_CREATE
          value: username:huangkaibin,admin:true
        - name: DRONE_SECRET_SECRET
          value: 558f3eacbfd5928157cbfe34823ab921
        - name: DRONE_SECRET_ENDPOINT
          value: http://drone-secrets
        - name: DRONE_RPC_SECRET
          value: c02e33d34bf9775d716f663434b7b31c
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        volumeMounts:
        - name: drone-pvc
          mountPath: "/drone"
      volumes:
      - name: drone-pvc
        persistentVolumeClaim:
          claimName: drone-pvc
      restartPolicy: Always

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: drone-pvc
  namespace: cd
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: standard

---
kind: Service
apiVersion: v1
metadata:
  name: drone
  namespace: cd
spec:
  selector:
    app: drone
  ports:
  - protocol: TCP
    port: 80
    name: http

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: drone-ingress
  namespace: cd
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/frontend-entry-points: http,https
spec:
  rules:
  - host: drone.your.com
    http:
      paths:
      - path: /
        backend:
          serviceName: drone
          servicePort: 80

.drone.yaml

---
kind: pipeline
name: deploy

steps:
  - name: print-env
    image: plugins/docker
    environment:
      username:
        from_secret: username
      password:
        from_secret: password
    commands:
    - echo $username
    - echo $password

---
kind: secret
name: username
get:
  path: docker
  name: username

---
kind: secret
name: password
get:
  path: secret/data/docker
  name: password

and it does not work as the same. the secret service still cannot recieve any request.

$ kubectl -n cd logs -f drone-secrets-7c78c86d9d-cpjwh 
time="2019-05-05T07:31:14Z" level=info msg="server listening on address :3000"
#15

What about this ?

« on this pod, you need to specify KUBERNETES_NAMESPACE=cicd if your not in default »

Does that help ?

I am not sure, but based on my memory, you don’t see any logs on the secret pod.

#16
        - name: DRONE_KUBERNETES_NAMESPACE
          value: cd

base on the offical suggestion, the env DRONE_KUBERNETES_NAMESPACE had been set to cd
@johnmarcou

#17

I am talking about the secret pod env var. Add the env var with your appropriate value for the NS.

Here comes a snippet of helm chart template for the secret pod:

  containers:
    - name: secret-manager
      image: "{{ .Values.k8sSecretImage.repository }}:{{ .Values.k8sSecretImage.tag }}"
      imagePullPolicy: {{ .Values.k8sSecretImage.pullPolicy }}
      env:
      - name: KUBERNETES_NAMESPACE
        value: {{ .Release.Namespace | quote }}
      - name: DEBUG
        value: {{ .Values.config.debug | quote }}
      - name: SECRET_KEY
        valueFrom:
          secretKeyRef:
            name: {{ print $.Release.Name }}-secret
            key: "DRONE_SECRET_SECRET"
      ports:
        - name: http
          containerPort: 3000
          protocol: TCP