The Importance of Automation

The Importance of Automation

Automation makes a world of difference in how effective you are as a developer.  I’ve worked in a few places now, and I can tell a huge difference between the type of developer that focuses on development alone versus the developer that focuses on a lot more…

  • Developing
  • Planning
  • Deploying
  • Testing

Developer

The ‘Normal’ Development Process

The ‘Normal’ Steps
  • Get requirements
  • Plan out the architecture (maybe)
  • Write some or all of the code
  • Compile (if necessary) and test.  (Multiple loops of writing code and testing for those that develop in a smarter way without waiting until the end to test everything)
You Just THOUGHT It Was Finished…

Once all the manual testing finishes and the code works as expected, it gets checked into source control.  End of story, everybody lives happily-ever-after, right?

Wrong.

BugIf you are lucky, a bug turns up a day or two after the source code was committed to source control.  The more likely case is that weeks pass before that bug shows up.  So what happens then?

 

  • Manually reproduce the issue
  • Write code to fix it
  • Manually verify the issue is gone
  • Commit to source control

So… end of story, everybody lives happily-ever-after, right?

Round 2

BugWrong.Bug

Congratulations, now you have two problems.  This pattern repeats, over and over.  Eventually, it is time to ship code and you cannot wait any longer.  Even worse, every time a new problem occurs, the same process is used to fix those bugs.   What is going to change?  How are you going to make everything better?

Round 3

AH!  We can rewrite all the code!  It is obvious those previous people writing the code did not know what they were doing.  They were obviously idiots.  So you re-architect the whole thing.  It’s a masterpiece.  So… end of story, everybody lives happily-ever-after, right?

My Thoughts

On a more personal note, I hate this approach.  A lot of the developers I’ve worked with still work with tools/methods that were popular in the 90’s.  This is one of the big problems with developers that do not keep up with current trends.  You end up working in a much less efficient way due to not keeping your skills up to date.  For those who are not familiar with better methods, it is the norm.  They believe it is an inevitable part of development.

It does not have to be that way.  In addition, it drives those of us who know better approaches to go crazy (and causes potential to lose your good developers!).  It is usually difficult to teach people new ways of doing things.  At times, these developers are not willing to learn something new.  On one side of the spectrum, it is a bit uncomfortable.  On the other side of the spectrum, you are stuck in an environment where your code works relatively well, and you spend most of your time working on other people’s bugs/bad design.  And don’t get me started on deadlines pushed by management that cause some to take shortcuts…

I think you get the point by now.  Unfortunately, despite our best efforts or design, bugs do crop up.  We need a smarter way to stop the whole develop -> manually test -> bug turns up a lot later -> develop -> manually test -> etc. process.  This is where automation can make a huge difference.

Automation Makes Us More Efficient Developers

Unit Testing

Suppose that rather than manually testing everything every time a change is made, there is a set of automated tests that run within seconds that test most of the functionality.  Such a concept exists called unit tests.  Let’s see how the cycle goes with bugs now.

  • Write an automated test to capture the issue (versus manually reproducing the issue).  This means the test should fail.
  • Write code to fix the failing test.
  • Integrate the code (it’s a good practice to also manually verify the fix worked).
  • Commit to source control.

So what happens now?  Perhaps your changes cause another bug, but if that automated test is run automatically every time a code commit occurs, that same bug will not occur again (if you don’t ignore a failing test).  So rather than create new bugs every time, the code becomes more and more stable as more and more automated tests get run every time a code change is made.

Continuous Integration

Running those automated tests after a code commit can happen automatically (as well as email alerts when the code fails those tests).  This is called continuous integration.  Tools such as Jenkins, TeamCity, and GitCI can help automate that testing.  Yay!  We’ve stopped the cycle of insanity (the code -> manual test -> check-in code cycle) and now have an actual process that will stop many bugs occurring over time (versus what we could call the ‘hope’ method).

Integration

Deployment

If you need to deploy your code to a different machine, you can potentially use something like Docker or a VM.  Why would you do this?  If you completely duplicate the environment, you are eliminating almost all chances of potential bugs cropping up from environment issues.  I can’t tell you how many times I or someone on a team I am working on has been bitten by a different version of Python, Linux, or something similar.  If you can develop in the same environment as what gets deployed, you minimize the risk of some issue occurring in deployment.  Just one less thing to have to worry about.

Want to Learn More?

Posts in this blog will look at these different types of automation and more.  Learning these things can make you a much more efficient developer.  You can be a developer that others depend on to get things done (I have heard this from my co-workers several times).  Others will know your software will just work most of the time without having to track down very many issues.  In short, you will be a reliable developer that others enjoy working with.

6 comments:

  1. Hi Tim!

    Thanks for sharing your insights!

    I find that I am often a victim of that never-ending loop you mentioned. This is especially true when working in collaboration with others on source code that either have no tests or requires manual testing. As a result, continuous additions are made to the code base while the success of past code is uncertain.

    It is agonizing to watch the mess that is unfolding. Continuous integration sure does sound very appealing haha. Do you have any advice for projects that have already began with no tests and is continuously changing?

    1. I completely understand. I used to work in an environment that was agonizing in that way – you ‘fix’ something only to break several other legacy pieces of functionality that weren’t tested well enough.

      I’m actually currently working on a blog post that will talk a little bit about this that will be published tomorrow. That being said, I’ll ask a few questions and hopefully point you in the right direction.

      1) The way you feel about the code development… it being agonizing… is that a feeling that you and a few others alone share, or is it a feeling shared by most of the developers and even management? If not very many people feel there is a problem, there is a culture problem. Fixing culture can be very challenging. Not only do you have to change people’s attitudes, but you will get very little buy-in from management (for the time you need) or other developers (for them to follow those new processes) to help implement processes that need to be put in place for continuous integration.
      2) If it is not a culture problem, then it’s a matter of learning the new tools and processes and taking the time to implement them. My recommendation:
      * Get a server you can designate as your build server. This will check out your code and do things with it (such as build and test it. Perhaps even deploy it/package it). So it needs to be hooked up to the same network as your source control. If you are not familiar with Jenkins and don’t have other tools to use, install it on that server and create a job that does something stupid, like echoing “Hello World”. This will get you slightly familiar with Jenkins.
      * Write a very, very simple unit test against your code. Find a very, very simple function that is very small, and wrap it in a unit test. I don’t know what language you use, but there are several frameworks out there: PyUnit (Python), cppunit or GoogleTest (C++), Qtest (Qt), JUnit (Java), etc. If you are unfamiliar with unit testing, try writing a simple test in whatever language you are using that makes sure 1 + 1 is 2. Then write one for your code.
      * Setup the environment on your build server so it has the right packages/dependencies it needs.
      * Create a job that checks out your codebase and runs that very simple unit test (which also will need to be checked into source control).

      Once you have that setup, you have a framework for adding stuff over time. Over time, you add more and more unit tests, and your CI server is more and more useful in detecting bugs automatically.

      Personally in that work environment, we had management buy-in to write a suite of unit tests against existing code. That means we took a time hit in the short-term, because we weren’t producing code that directly made money for the company. However, over time, we were much, much, much, much, much (did I say that enough times? 🙂 ) more efficient developing in that section of the code.

      1. Hey Tim,

        Thanks for the detailed response and your most recent blog post. The latest entry speaks to a lot of my concerns haha.

        I believe that several co-workers have the same sense of uncertainty and concern as I do but as you said, it would be difficult to convince others of the benefits. Especially because of the up-front investment. However, it may be good to discuss the ever increasing technical debt that will result in a greater expense eventually.

        I think that your second question is also a barrier to entry for us as we do not have experience with the necessary tools. This leads us back to having a larger frontal investment before the returns are seen. Nonetheless, I truly believe with you that having such a system set up will prove greatly beneficial in the future.

        Looking forward to reading new posts and implementing some of your ideas in personal and work projects!

        1. You may can mitigate that up-front cost if you want to outsource it. You would need to make sure you have at least one or two people that could maintain it once it is in place. In addition, your team would need training on how to use the tools. I (and I’m sure others) could provide services as a consultant if that is something your group/company is interested in.

          In the meantime, if there is a post you see here or something you think of that would help to learn more about, let me know. Best of luck!

Leave a Reply

Your email address will not be published.