Jenkins Jobs: When Automation Goes Wrong

So You Created Your First Jenkins Job…

Butler
Well, he kind of looks like Jenkins, right? He’s a butler…

For those that are unfamiliar with Jenkins, it is software that makes it easy to setup a continuous integration server with a web front-end.  A ‘job’ or ‘project’ on Jenkins is something that usually builds or tests code, although it can do a lot of things you normally could on a terminal or command prompt.  It usually checks out code from some source code repository, builds that code, tests it, then reports some information about it (such as unit tests passing/failing, code coverage, sending emails, etc.).

Getting your first job created on Jenkins just the way you want it can be challenging.  You have to get the source repository setup properly (with credentials), make sure the build step command works properly, execute tests properly, put them in the proper directory… the list goes on and on.

However, once you finish that first job, there usually is not much to change for the next job.  For example, suppose your code consists of ten different packages of C++ code.  Once you finish one package, the next package’s only difference is the location of it within the source code repository.  You still use the same make command, the same executable to run tests, and the same method to report test results.

The pitfall of this happens when you copy-paste jobs over and over within Jenkins.  Eventually you are going to need to change something.  Then things can get hairy.

Introducing the Jenkins Job Update Nightmare

Maintenance

Let’s say you have copied that job and now have 100 different versions of it.  Perhaps you have 10 packages, but each package has 10 different branches with a different job for it.

Since each job is different, you may have done one-off small changes in each job.  Although they should be common, they no longer are common.  There are very small changes within each job.

Now you need to change the make command from “make all” to “make source”.  You now have to go to 100 different web pages, click the configuration link, scroll down to the build step, then make that change.  At the very least, this is extremely tedious.  At worst, it can consume a few hours for something very trivial.  It’s also very error-prone.

There must be a better way!

Solutions

IdeaThere are smarter ways than updating every individual job through Jenkin’s web GUI.  I will list them and briefly describe them in this post.  In future posts, I will elaborate on all these solutions.

Use regex to replace the job’s XML

Each job within Jenkins has an associated XML file that defines that job.  If you have access to the build server, you can navigate to the jobs directory and perform a regular expression replacement.  Hopefully you can find one that will replace all those 100 different jobs correctly.

Pros:

  • Works even if you don’t have existing way of creating all the jobs from a script.

Cons:

  • Regexes can be tricky and error-prone.
  • Requires access to the build server.

Use the Job DSL plugin

Jenkins has a way of defining jobs in a DSL (domain specific language) format using the Job DSL plugin.  I have used this before to create one job that creates all the other jobs.  You can make it as custom as you wish.  You also have a single point that can create every job on your server if you are diligent.

Pros:

  • Can be very custom
  • Can create all jobs at once, overwriting all local changes – since they should be managed from the common script anyway

Cons:

  • The DSL has a bit of a learning curve.
  • If you do make local changes, they can be overwritten by this.  Some may consider that a good thing, but it could be really bad if you needed those changes and just lost them.

Use Jenkinsfiles

Jenkinsfiles are configuration files that are similar to the Job DSL plugin scripts.  These configuration files live directly within your project’s code.  If used in conjunction with shared libraries, you can have very trivial Jenkinsfiles that all refer to a common configuration.

Pros:

  • Configuration lives with source code.  Developers not familiar with Jenkins may still be able to change the job without knowing much about Jenkins.
  • Can share common configuration with multiple jobs if using shared libraries.
  • Can be very custom.

Cons:

  • Has a bit of a learning curve.

Conclusion

If you already have an existing Jenkins installation with several jobs that do almost the same thing, you can either invest some time into making it better with the Job DSL plugin or regex replacing XML files.  Keep in mind that you don’t have to manually replace one line of configuration in 100 web pages.

If you are just starting out with Jenkins, attempt to learn the Jenkinsfile format.  Not only is it a more maintainable way of maintaining your jobs, but it also allows developers not familiar with the server to change how a job is built.  It’s a lower barrier to entry for those who don’t want to learn the Jenkins platform (although higher for the person originally implementing them).  I believe this approach is the most maintainable method for keeping multiple jobs up-to-date.

5 comments:

  1. I’ve also found the configuration slicer to be useful when changing a bunch of similar jobs that you inherited. I also have found it useful in figuring out how that set of jobs varies and putting things back to a common configuration

    1. That looks like an interesting plugin. I really like the idea of using it to explore how existing jobs work and how they are related to each other. Thanks for pointing me to it!

Leave a Reply

Your email address will not be published.