In one of my previous jobs, we had an issue with an FPGA that caused our product to not boot at cold temperatures. Ironically, this is not that different from my current job, where we also test our products over a varying degree of temperature ranges.
The process for the previous job went like this:
- Put system into temperature chamber (with serial cable to communicate with it coming out of chamber)
- Manually set temperature chamber to a low temperature (although it can be automated, there was no need for this test)
- Reboot device
- Verify a certain message doesn’t appear when it boots up, and verify system boots
When we rebooted the device, we verified it came back up by looking at its output over a telnet session. We used a tool called Procomm, which provides a way of using telnet to talk to a device.
A fix was put into the code and needed to be tested… but this is one of those fixes that isn’t ever really verified. So we decided to verify the fix by booting something like 100 times before we said the fix ‘worked’.
Rebooting the system took about 5-10 minutes. So you can imagine rebooting this thing manually 100 times would take a really, really long time. Do you really want to sit and wait 5 or 10 minutes for 100 times? That’s unreasonable, if you ask me.
Fortunately, we were able to automate this. Procomm allows a scripting language called ASPECT to run automated scripts over the telnet session. It used EXPECT-like syntax. So you can enter in a command, then expect a specific response and take actions depending on what response you get.
So the script went something like the following:
- reboot command
- expect debug message to appear
- Does it have error text? Stop script (display loop number). We are stuck and didn’t boot.
- No error text: proceed
- Wait for login prompt
- reboot, and repeat
Using this script in Procomm, we were able to save days or weeks of someone having to manually turn this system on and off again.
This experience shows that software automation doesn’t have to be used only for software. As a matter of fact, much more powerful and interesting things can happen when hardware and software automation intersect. Automating hardware tends to be a lot more interesting, because a lot more can happen. For example…
- Software can turn on a light (IoT)
- A power supply can automatically turn a system on or off
- A pickering switch can be used to create electrical connections in a circuit
- Thermal controllers can automatically be programmed to go to certain temperatures
Granted, using software automation for hardware can sometimes be quite pricey. But the results can be very rewarding.
Keep this in mind the next time you need to repeat some hardware task many times. It may be worth spending the extra money to get the tools needed to automate that task.