This Is My Opinion!
Before reading too much into this, this is simply my opinion of agile development. While it is flavored towards software, it could be used for other creative processes.
In addition, I am not an “expert” per se with Agile practices. I very well could misrepresent a fundamental concept. Either way, I feel there are good lessons to learn here.
Live by Waterfall, Die by Waterfall
For those unfamiliar with the waterfall cycle, it’s the philosophy of putting everything in a specific stage, with verifying everything in the previous stages at the very end.
For example, you first might determine how to build something (plan it out). Then you could do the work that you planned. Following that, you could test it.
The problem comes from the length of time between original concept to testing that concept. Usually there’s weeks, but usually months from the original requirements to the testing.
If you find something in the “building” part of the process during your validation part, you have to go back to the build phase.
But if you find something wrong with the requirements themselves, you have to go back even further.
This eats up a lot of time.
Let’s look at a perfect example.
The Dreaded Thermal Test
For embedded products, there are times when thermal temperature testing is done to verify it can work in different climates.
This type of testing can show issues with the hardware or software at one of these temperatures.
For example, when the processor is bogged down doing lots of processing, and the temperature is very hot, really bad things can happen.
When that same processor is bogged down and the temperature is at room temperature, everything is fine.
Based on a waterfall-like approach, a thermal test failure is discovered at the end of the development process for both hardware and software. So it is quite costly to find failures at this point.
This happens a lot during the waterfall development life. An issue is found in that ending validation phase, and you have to go back to a previous phase of the project.
Why Is It Costly to Find Problems Later?
There are many studies out there that show the cost of a project goes up as you find problems later in the development cycle.
As more hardware and software is developed, making changes costs more. For example, a software issue found after all the software is designed has the potential to cause an architecture redesign. That’s probably close to worst-case scenario, but that would cost a whole lot of time.
If that same issue was found earlier, perhaps the architecture could have been designed in a way to prevent that issue from occurring.
In addition, if you find a problem right after you design something, it is fresher in memory. You don’t have to spend a lot of time remembering something that was done in the past.
Due to the nature of creative work, you are going to have way too much stuff to hold in your head 100% of the time. So if you go back even as little as an hour/day/week at times, you have to remember what you did. The quicker you can find the problem from when it was created, the less time you spend remembering what’s going on.
We Want Shorter Cycles!
As a general rule of thumb, the quicker you get feedback based on something, the faster you can react and finish.
On a broader scale, this is why solving big problems like world hunger are hard. You can try a solution, but it is going to take a while before you get data to analyze how effective the solution is.
With hardware and software, the shorter development-to-test cycle is huge. When testing is very quick (seconds, as it would be for software unit tests), you can test very often. If something is found, your most recent change (most likely) caused it.
As your cycle grows longer, it takes more time to find failures. When it is found, there are many more things that could potentially have caused that failure.
For that thermal test, suppose we test multiple times (maybe once a day rather than after a month or two). Now the number of things that can cause a thermal failure go drastically down. If it passed yesterday but not today, only today’s changes are most likely to have caused the failure.
In addition, you have much less risk. If something fails at the end of a month project, 30 days have been verified. Only the 31st day failed. So risk is mitigated due to much more frequent testing (due to shorter development-to-test cycles).
Hopefully I’ve convinced you that these shorter cycles are desirable. How can we achieve them?
Agile Gives Us Short Cycles
I am most familiar with Scrum (which is a specific type of Agile development process), so my comments here are going to reflect that style of development.
Scrum has the idea of a “sprint”. A sprint is a time-boxed period of time during which features are developed, tested, documented… everything. At the end of each sprint, a feature could potentially be released right then to a customer. No ‘little tweaks’ before it gets shipped. It could be released right then (and has the appropriate container, executable, or whatever as the end product… it is already built).
Sprints should not be longer than 4 weeks, and most of the companies I’ve seen use 2-week sprints.
Remember those fast feedback times I talked about before?
Scrum is forcing your feedback window to fit into that sprint.
Examples of Questions that Arise from Sprints
Let’s take the example above for the thermal test. Rather than wait until the end to test thermal, it should be tested within each sprint.
This could mean all sorts of things…
- We don’t have enough thermal chambers! (buy more or use them more efficiently)
- This seems ridiculous! Do I really need to do this every single sprint? (yes, see the benefits of short cycles, and change your mindset)
- We don’t have time for thermal testing during our sprints! (reduce the scope of what you are working on, or improve your efficiency during a sprint with automation or more workers)
See how something as simple as saying we need that faster feedback can open up all sorts of questions?
But that’s just one example. The same can be applied to many, many different things.
- Where’s the time for documentation?
- Automate it
- Reduce scope so you make time for it
- Get more people so you improve how much work you can get done
- We don’t have time for testing!
- Are you developing too much within one sprint?
- Do your tests take too long to run?
- Perhaps more testing equipment would help?
- Maybe you need to virtualize hardware (with containers, VMs, unit tests, etc.) so software can be tested quicker.
You must keep focused on that fast feedback time. It’s the key to de-risking projects and delivering on time.
The waterfall development process promotes testing things at the end of a project. This leaves a lot of risk towards the end of a project. If something fails, it could be very costly to fix it. Which means a date is going to slip. Budgets might go through the roof.
Agile development helps mitigate this risk by having much quicker development-to-test cycle times. By continuously testing, risk is mitigated. Each test validates the previous development. So by the end of a project, there’s not very much risk left. Everything has already been validated previously.
Now… the concept is not that difficult to understand. However, the implementation can be extremely challenging. Company politics, following the process even under pressure, and several other factors get in the way of being consistent with Agile development.
A firm commitment to good development processes and following them even under pressure is what separates the most successful companies from those who fly by the seat of their pants (and are not as successful).