Slow Feedback Leads to Waste
In one of my previous positions, we took a lot of time waiting to be able to test our code changes.
The code was in C++. Testing a change first required compilation. If it was a clean build, this took anywhere from 10-30 minutes. For a non-clean build, the quickest build time was about 2 minutes.
After the code was built, it took about 3-5 minutes to load that compiled code onto a hardware debugger.
So if I want to test even the simplest change (such as a comma showing up in some output), I would have to wait at minimum 5 minutes.
What happens during that 5 minutes of downtime?
- You lose focus.
- Boredom sets in.
- You surf the web.
So if you could eliminate that downtime and make it seconds instead of minutes, this would be a huge time-savings.
In addition, consider where the test setup is very involved. It requires hundreds, thousands, or hundreds of thousands of dollars of test equipment. Since that level of testing is only available in a resource-limited higher-level of testing, it takes a month for code to get there. That’s a nightmare!
Reducing the Test Cycle Time with Unit Tests
All the C++ code was in a huge “main” branch. This was a big reason why it took so long to compile everything. That one branch was really big.
In order to reduce compile times, we had to reduce what was compiled. We did that by removing lots of dependencies. This made stand-alone pieces of code that could be compiled without a lot of the other dependencies. Hence, a software package.
The “main” function now ran unit tests (rather than the production main that executed the main functionality of the code). With this new, slimmer package, compile AND test times usually took under 10 seconds. At that speed, you no longer lose focus between builds. You also don’t have to manually test until you integrate that package back into the main codebase.
This was a huge win. By making the code-to-test cycle extremely quick, we increased focus and greatly improved quality. Rather than find issues 5 or 10 minutes later, you found them immediately… while you were still focused on the problem.
In addition, you don’t skip manual testing when you make a trivial change. Having to test so often after waiting so much tends to make developers skip testing for “easy” changes. When the building and testing takes seconds, this doesn’t happen nearly as often.
Reducing the Test Cycle Time by Replicating a Setup
In addition to speeding up the code-to-test cycle with unit testing, we also replicated the bigger test setup. This is because unit testing would not catch an error that a system test would catch.
By the time I put changes into the main branch, it would take about a month before I would hear about issues with that piece of code. This is because it took a large test setup to replicate (and getting to that large test setup meant going through many smaller tests before it got to that point).
A really good test engineer at the time was able to replicate the issue with a smaller setup. It took probably a day or two to replicate. That was a long time to setup, but it was well worth the effort. This is because it drastically reduced our code-to-test time.
Now I could be more confident in my changes. And I could test a lot more approaches. Thanks to being able to test in 10-15 minutes (rather than a month), I could come up with more aggressive solutions. I could test a lot more.
Once again, reducing that code-to-test cycle allowed everyone to come up with a much better end solution.
Neither of these solutions in making the cycle time shorter were “easy” fixes. They both took a lot of effort, both in creative problem solving and in time.
In both cases, however, the end goal (for the unit test and package approach) affected at least a hundred developers on a day-to-day basis. A savings of probably at least (very conservatively) 10 minutes for each of those developers each time they used that packaged code. This easily adds up very, very quickly.
As for the waiting for a month approach, this simpler test setup saved me multiple weeks in waiting for the test. Yes, I would move on to other things while I waited, but I was able to try to much more in a much shorter timeframe with the new test setup.
Reducing that code-to-test cycle time is such a huge benefit to everyone. Even if it takes a really long time to come up with the solution, I personally have found it to be worth it every time.