Drone

Drone.io How to control multi-module build pipeline where modules are interdependent for CI

Hi guys,

I am working on my drone CI pipeline facing following issue:

My git repo looks like this:

  myrepo
     -> .drone.yml
     -> images/
           -> a1/
                 -> a1script
                 -> Dockerfile
           -> a2/
                 -> a2script
                 -> Dockerfile
           -> a3/
                 -> a3script
                 -> Dockerfile
            -> a-final/
                 -> Dockerfile

a-final depends on a3 to build and publish it’s image on our artifactory and same for others - a3 depends on a2 and a2 depends on a1.

Now, in my .drone.yml what I want is if a module has been changed in git repo - then only that module and all it’s dependent modules should be build and published (there is only 1 “step” for each module in my .drone.yml file to publish) one after another and not all of them. For example, If I push some code in a2, then only a2, a3 and a-final should be built and published in order a2, a3, a-final.

I tried doing that in my .drone.yml but can’t control the behavior even with using conditions like status:success OR environment variables.

This is how the first 2 modules (a1 and a2) of my .drone.yml looks like:

pipeline:

 # Get the latest git diff and see what directories changed.
check_clone:
  image: plugins/git
  commands:
    - git diff-tree --no-commit-id --name-only -r `git log | head -1 | awk '{print $2}'` > git_latest_delta
    - cat ./git_latest_delta
    - echo "Done!"

# Check if a1 is in the diff list - if not then exit 1 
a1_build:
  image: ubuntu:bionic
  commands:
     - ./is_build_required a1
    

 # Publish a1 on some condition - like if previous step is successful or if an env variable is set.
a1_publish:
  image: plugins/docker:17.05
  registry: docker.mycompany.com
  repo: docker.mycompany.com/app/a1
  dockerfile: images/a1/Dockerfile

  tags:
    - "${DRONE_COMMIT:0:8}"
  when:
    status: success
  username: myuser
  secrets:
    - source: artifactory_password
      target: plugin_password

a2_build:
  image: ubuntu:bionic
  commands:
    - echo $$DRONE_TAG
    - ./is_build_required a2
  when:
    status: [ success, failure ] 


 # Don't know how to get at this step as if a1 publish is skipped  then pipeline fails and the below step doesn't execute. :(

a2_publish:
  image: plugins/docker:17.05
  registry: docker.mycompany.com
  repo: docker.mycompany.com/app/a2
  dockerfile: images/a2/Dockerfile
  tags:
    - "${DRONE_COMMIT:0:8}"
  when:
    status: success
  username: svcpejkn001
  secrets:
    - source: artifactory_password
      target: plugin_password

Same for a3 and a-final. This how I need to control the pipeline build process.
I tried with custom environment variables but couldn’t control the “when:” field in step because it doesn’t accept our defined variables - it only goes with fixed things like status, branch, repo, etc.

So, please help me.

Thanks,
Neo

It sounds like you want to run certain pipeline steps based on whether or not files in subdirectories were changed. For this you should consider installing a plugin which can be used to add custom configuration options to Drone. Note that these plugins only work with Drone 1.3 or higher. Here are a few third party plugins that might be useful:

  1. https://github.com/microadam/drone-config-changeset-conditional
  2. https://github.com/bitsbeats/drone-tree-config

Please note these are third party plugins and are not maintained or supported by the Drone project. If you have any questions or need any help with a plugin you should create an issue in the plugin repository.

@bradrydzewski Thank you for the reply and suggesting the third-party plugins.
Unfortunately, I am using Drone 0.8 version.

  1. Isn’t there any other way I can achieve this? I was able to create global variables and was successful changing there values between the steps too. But, for publish steps, I couldn’t add them to “when” tag.

  2. May be is there something I can use specifically on publish sections in my above .drone.yml file where it will decide weather or not to run this publish step - without failing the entire pipeline OR any other solution if you have in mind.

  3. Also, as you have seen my git repo structure and dependencies above… I am open to a complete design change too for my git repo which can fit into Drone builds and I can achieve the same result.

So, please feel free to suggest anything that you know.
Again thank you so much for the help.

Thanks,
Neo

sorry, I am not aware of any way to solve this in 0.8, which is why we were excited to add this capability in 1.x