In my previous post, we looked at the pain of changing 100 different Jenkins jobs manually when most of them need the exact same change. In this post, we will explore a way of preventing that from happening by using some infrastructure. Namely, a domain specific language called Job DSL to create and manage our jobs.
Job DSL Description and Example
Rather than configure a Jenkins job through the web GUI, Job DSL allows you to write code in order to create your jobs. Here’s an example straight from the job-dsl-plugin documentation:
def gitUrl = 'git://github.com/jenkinsci/job-dsl-plugin.git'
scm('*/15 * * * *')
maven('-e clean test')
cron('15 13 * * *')
cron('15 1,13 * * *')
maven('-e clean integration-test')
// no trigger
// limit builds to just Jack and Jill
maven('-B release:prepare release:perform')
Job DSL scripts are written in Groovy (a dynamic language built on top of Java). The above shows an example of creating four jobs:
All four get code from the same Git URL. The unit tests job is triggered by a commit. The sonar and integration tests build periodically. The release job must be built manually. All of them have different build steps.
Suppose the Git URL changes. If you changed everything through the web GUI, you would have four places to have to click through and change. With this script, you have one location to change, then all four jobs will get code from the updated location. That’s significant if you have hundreds or thousands of jobs to maintain.
One Job to Rule Them All
Do you remember how the Lord of the Rings has one ring that rules over all the other rings? In a similar manner, I have used the Job DSL plugin in Jenkins in order to have one job to ‘rule over them all’. This is somewhat similar to the above example, except I have one job whose sole responsibility is to run that Job DSL script. In addition, my approach was much more modular in order to create about 30 jobs.
Here’s the list of steps I recommend:
- Install the Job DSL plugin.
- Create one job in Jenkins that reads from a DSL script. I recommend these scripts live in some project in your source code repository for Jenkins libraries. Jenkins then references that Job DSL script through source control.
- I make the DSL script replace any existing jobs with the same name. This forces everyone to use the Job DSL scripts to manage their jobs (you could manually go into a job and alter something without going through the Job DSL scripts… but it’ll be replaced the next time the Job DSL script runs).
- Make this job creator job run once every night (so local changes to the job get overwritten – so it forces everyone to use the Job DSL scripts) and also once a commit is made on the Job DSL script itself.
- Make one common file for Job DSL to start from. In general, this has involved saving off a bunch of functions (for example, set a pointer to the “job” function and any other functions you need). Access to these functions is global (because once the main groovy script calls another groovy script, those Job DSL functions are no longer in the namespace of that file).
- Create separate groovy files that build either one job or a certain type of job. I usually split these up into a file that holds the variables – the few things that change in this type of job – and another file that holds the job’s main configuration (which references a few variables from the config file)/
- For example, buildcpp.groovy could build a job that compiles C++ code.
- You also need a configuration file – buildcppconfig.groovy. Perhaps that file holds 10 different SVN source repository URL locations. buildcpp.groovy imports this file and uses its variables.
- The main groovy script would call a function in buildcpp.groovy which loops through the config file to create however many jobs the buildcppconfig.groovy file specifies.
- Once you have that configuration and it loops through each configuration with each job type, you have one source that creates all your other jobs.
Pros and Cons
- These scripts can be very custom.
- They can maintain hundreds or thousands of jobs with ease. If you have written your Groovy script well, you should only need to change a few configuration file scripts (in general) to create new jobs or alter existing ones.
- Can create all jobs at once. In order to replicate all the jobs on your Jenkins server, you just have to replicate one job, which then can create every other job.
- Job DSL has a bit of a learning curve. It certainly is not nearly as easy as just configuring the web GUI.
- This configuration I have described overwrites local changes. This is intentional, but it may cause some to lose their work if they aren’t disciplined enough to add their changes to the Job DSL script before it gets run.
This was a really interesting use of the Job DSL plugin in my opinion. It allowed me to have a whole lot of control over the jobs in Jenkins. It was really easy to replicate several tens of jobs for a separate branch in SVN just by adding one line of configuration in one of my config files. It also verified all those jobs were the same. Although there is a learning curve, it is definitely worth considering in a large-scale Jenkins deployment. Having that type of control over all your jobs saves a lot of time down the road.