Drone on k8s with k8s secrets plugin


#1

Hello, drone is awesome.

I’m attempting to use drone on k8s, and use k8s secrets in my build.

I have used this yaml to deploy drone and drone/kubernetes-secrets:

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: drone-data
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: drone
  labels:
    app: drone
spec:
  replicas: 1
  selector:
    matchLabels:
      app: drone
  template:
    metadata:
      labels:
        app: drone
    spec:
      containers:
      - name: drone
        image: drone/drone:1.0.0-rc.4
        livenessProbe:
          httpGet:
            path: /healthz
            port: 80
          initialDelaySeconds: 3
          periodSeconds: 3
        env:
        - name: DRONE_LOGS_DEBUG
          value: "true"
        - name: DRONE_KUBERNETES_ENABLED
          value: "true"
        - name: DRONE_KUBERNETES_NAMESPACE
          value: default
        - name: DRONE_GITEA_SERVER
          value: https://git.example.com
        - name: DRONE_RPC_SECRET
          valueFrom:
            secretKeyRef:
              name: drone-secret
              key: rpc
        - name: DRONE_SERVER_HOST
          value: drone.example.com
        - name: DRONE_SERVER_PROTO
          value: https
        - name: DRONE_SECRET_SECRET
          valueFrom:
            secretKeyRef:
              name: drone-secret
              key: secret
        - name: DRONE_SECRET_ENDPOINT
          value: http://localhost:3000
        ports:
        - containerPort: 80
          name: drone
        volumeMounts:
        - mountPath: /data
          name: drone-data
      - name: drone-secrets
        image: drone/kubernetes-secrets
        env:
        - name: DEBUG
          value: "true"
        - name: SECRET_KEY
          valueFrom:
            secretKeyRef:
              name: drone-secret
              key: secret
      volumes:
      - name: drone-data
        persistentVolumeClaim:
          claimName: drone-data
---
kind: Service
apiVersion: v1
metadata:
  name: drone
spec:
  selector:
    app: drone
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
  name: drone
spec:
  rules:
  - host: drone.example.com
    http:
      paths:
      - backend:
          serviceName: drone
          servicePort: 80
  tls:
  - hosts:
    - drone.example.com
    secretName: drone-cert

A secret I want to use:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: drone-example-registry
  namespace: default
data:
  password: xxxxx
  username: xxxxx

And then this .drone.yml in my build:

kind: pipeline
name: default

steps:
- name: build-image
  image: banzaicloud/drone-kaniko
  settings:
    registry: registry.example.net
    repo: registry.example.net/hello-k8s
    tags:
    - ${DRONE_COMMIT_SHA}
    - latest
    username:
      from_secret: username
    password:
      from_secret: password
  when:
    branch:
    - master

---
kind: secret

external_data:
  username:
    path: drone-example-registry
    name: username
  password:
    path: drone-example-registry
    name: password

As you can see I’ve turned on the debug logging on both containers, but the drone/kubernetes-secrets show no attempts by the other container to access it (I tested it was logging requests that I made with curl) and the drone logs don’t show any errors.

I spend a few minutes looking at the drone source and couldn’t figure out how the env var DRONE_SECRET_ENDPOINT did anything. It seemed like perhaps that code had been removed?

Thanks!


#2

Hey there, sorry about that, the Kubernetes runtime does not yet support global secrets or registry credentials, but I’m actively working on this in the latest sprint. Kubernetes was not originally planned for 1.0, but I gave into scope creep and am doing my best to bring the implementation up to speed as quickly as possible before a 1.0 final :slight_smile:

I documented some of the known gaps in Kubernetes in the latest release notes, in the Roadmap section. Once implements, the Kubernetes runtime should have parity with the Docker runtime, or will at least be very close. See 1.0.0-rc.4 release notes

In the meantime, hopefully you can use the built-in secrets as a temporary workaround until the next release is available.

edit: just wanted to follow up that I hope to complete this sprint end of next week, at which point I will tag the next release.


#3

@bradrydzewski thank you for that quick update. I’ll be watching for the next release.


#4

Met same issue, looking forward to the k8s global secret feature release


#5

Hey @bradrydzewski I tried this again, but with drone/drone:1.0.0-rc.5 and it’s still not working. Any thoughts on what I might be doing wrong or ideas on where to look?

I’m guessing the k8s global registry stuff works the same way with a microservice answering requests? I’m basing that off of https://github.com/drone/drone/issues/1998

I’m a developer working on my homelab, not making any money. Are there going to be features like this locked off in the final release?

Thanks!


#6

as same as @myers, I have same problem:

Am I missing something or this feature not release?


#7

hey there, sorry, I am unable to reproduce any issues. I will be publishing the full source code later next week so that everyone has access to the code and can self-troubleshoot.


#8

Looking forward to it.


#9

I have also deployed the drone/kubernetes-secrets plugin according to documentation and I see zero requests to it. I was expecting the plugins/docker to push the built image to the repository with credentials out of the k8s secret. Doesn’t seem to work yet in rc-5.

Docker push log tail:

unauthorized: The client does not have permission to push to the repository.
time="2019-02-19T15:44:02Z" level=fatal msg="exit status 1"

kubernetes-secrets:

$ kubectl logs -f -n drone drone-drone-server-5b576884bc-l9smf -c secrets
time="2019-02-19T15:35:49Z" level=info msg="server listening on address :3000"

#10

@rafi I provided some notes [1] for troubleshooting the Vault plugin. You should be able to use a similar approach to troubleshoot the Kubernetes secret plugin. I also recommend inspecting the source code to get more details.

[1] Drone-vault plugin not quite working


Drone-cli: plugins "unsupported protocol scheme"