Using Job DSL to Change Jenkins Jobs

In my previous post, we looked at the pain of changing 100 different Jenkins jobs manually when most of them need the exact same change.  In this post, we will explore a way of preventing that from happening by using some infrastructure.  Namely, a domain specific language called Job DSL to create and manage our jobs.

Job DSL Description and Example

 

Rather than configure a Jenkins job through the web GUI, Job DSL allows you to write code in order to create your jobs.  Here’s an example straight from the job-dsl-plugin documentation:

Job DSL scripts are written in Groovy (a dynamic language built on top of Java).  The above shows an example of creating four jobs:

  • PROJ-unit-tests
  • PROJ-sonar
  • PROJ-integration-tests
  • PROJ-release

All four get code from the same Git URL.  The unit tests job is triggered by a commit.  The sonar and integration tests build periodically.  The release job must be built manually.  All of them have different build steps.

Suppose the Git URL changes.  If you changed everything through the web GUI, you would have four places to have to click through and change.  With this script, you have one location to change, then all four jobs will get code from the updated location.  That’s significant if you have hundreds or thousands of jobs to maintain.

One Job to Rule Them All

Lord of the Rings Ring

Do you remember how the Lord of the Rings has one ring that rules over all the other rings?  In a similar manner, I have used the Job DSL plugin in Jenkins in order to have one job to ‘rule over them all’.  This is somewhat similar to the above example, except I have one job whose sole responsibility is to run that Job DSL script.  In addition, my approach was much more modular in order to create about 30 jobs.

Here’s the list of steps I recommend:

  1. Install the Job DSL plugin.
  2. Create one job in Jenkins that reads from a DSL script.  I recommend these scripts live in some project in your source code repository for Jenkins libraries.  Jenkins then references that Job DSL script through source control.
    1. I make the DSL script replace any existing jobs with the same name.  This forces everyone to use the Job DSL scripts to manage their jobs (you could manually go into a job and alter something without going through the Job DSL scripts… but it’ll be replaced the next time the Job DSL script runs).
    2. Make this job creator job run once every night (so local changes to the job get overwritten – so it forces everyone to use the Job DSL scripts) and also once a commit is made on the Job DSL script itself.
  3. Make one common file for Job DSL to start from.  In general, this has involved saving off a bunch of functions (for example, set a pointer to the “job” function and any other functions you need).  Access to these functions is global (because once the main groovy script calls another groovy script, those Job DSL functions are no longer in the namespace of that file).
  4. Create separate groovy files that build either one job or a certain type of job.  I usually split these up into a file that holds the variables – the few things that change in this type of job – and another file that holds the job’s main configuration (which references a few variables from the config file)/
    1. For example, buildcpp.groovy could build a job that compiles C++ code.
    2. You also need a configuration file – buildcppconfig.groovy.  Perhaps that file holds 10 different SVN source repository URL locations.  buildcpp.groovy imports this file and uses its variables.
    3. The main groovy script would call a function in buildcpp.groovy which loops through the config file to create however many jobs the buildcppconfig.groovy file specifies.
  5. Once you have that configuration and it loops through each configuration with each job type, you have one source that creates all your other jobs.

Pros and ConsThumbs Up and Down

Pros:

  • These scripts can be very custom.
  • They can maintain hundreds or thousands of jobs with ease.  If you have written your Groovy script well, you should only need to change a few configuration file scripts (in general) to create new jobs or alter existing ones.
  • Can create all jobs at once.  In order to replicate all the jobs on your Jenkins server, you just have to replicate one job, which then can create every other job.

Cons:

  • Job DSL has a bit of a learning curve.  It certainly is not nearly as easy as just configuring the web GUI.
  • This configuration I have described overwrites local changes.  This is intentional, but it may cause some to lose their work if they aren’t disciplined enough to add their changes to the Job DSL script before it gets run.

Conclusion

This was a really interesting use of the Job DSL plugin in my opinion.  It allowed me to have a whole lot of control over the jobs in Jenkins.  It was really easy to replicate several tens of jobs for a separate branch in SVN just by adding one line of configuration in one of my config files.  It also verified all those jobs were the same.  Although there is a learning curve, it is definitely worth considering in a large-scale Jenkins deployment.  Having that type of control over all your jobs saves a lot of time down the road.

6 comments:

  1. hello, I find what you do super cool, I’m doing exactly the same thing, but I can not do what you say in point 4, that is having two groovy script one with the configuration that should always go and another what is changed, but I do not know how to call them, you can help me, I would really appreciate it.

    1. It’s going to be hard to tell you without knowing what you’ve tried.

      The Jenkins job should be able to call a top-level groovy script. Usually in that script, I import from the configuration file and also import from the file that creates the job.

      I generally save a variable that holds the functions themselves, then pass those variables into the Groovy functions that do the actual job creation (along with the configuration).

      Here’s two sites that explain how to import one groovy script from another:
      https://gangmax.me/blog/2015/10/15/include-a-groovy-script-in-another/
      http://mrhaki.blogspot.com/2011/06/groovy-goodness-add-imports.html

      I think I personally did something similar to the mrhaki link, using the “package” and “import” keywords.

      Hopefully that helps.

      1. Thanks Tscott.
        I have another question, I have now configured and created DSL for jenkins without problems, however I have a big question, I am taking the information in this page https://jenkinsci.github.io/job-dsl-plugin/#, but I have a Plugins installed Dimensions, and I need to choose a new source source source configuration, is the part where git, subversion, among others, my question is How can I mark something that is not in the api, something like keep my own dsl?

        1. There’s several things I can think of.

          1) Ask the maintainers of the Dimensions plugin to add support to Job DSL.
          2) I don’t have a Jenkins server available, but I think I remember being able to choose a job that has the Job DSL from the local workspace (rather than from a repo). You could have the prebuild step of checking code out from Dimensions and then use the local DSL script from your workspace.
          3) Job DSL also has the ability to use XML. All jobs in Jenkins are created from XML files, and I think Job DSL has the ability to use that XML.

          Personally, I would try steps 2 or 3.

          Good luck!

          1. Hi,
            I have jobs in my production jenkins server and would like to write the groovy script manually for the existing jobs.

            could you tell me the process, how to proceed and write?
            if any links also ok.
            thanks in advance.

          2. I don’t know if there is an automated way to do it – you probably have to manually write the script.

            But I would start here: https://github.com/jenkinsci/job-dsl-plugin

            I would also create a Jenkins job who can create jobs based on DSL and do a lot of testing with that job and Googling.

            I forget if you can use XML in the DSL (if you can, it’s not the ideal way to do it – only do that as a last resort), but if you can, each job has it’s own XML file to hold the configuration for that job. If memory serves, it is located somewhere in /var/lib/jenkins/[job_name]/[something].xml. That XML might be a starting point to finding the right equivalent DSL code if you can’t find it with Google.

            Good luck!

Leave a Reply

Your email address will not be published. Required fields are marked *