Patching software in a networked world is not enough: why flaky tests should not be ignored

28 November 2018

Dr. Greg Law, CTO, Undo

Dr. Greg Law, CTO, Undo

It is not uncommon that vulnerabilities are found once software is shipped. For example in September, Cisco reported it had exposed 13 vulnerabilities in its IOS and IOS XE switch and router operating software, and said this should be patched within weeks.

Network and router technology companies such as Cisco need to demonstrate a rapid evolution in innovation driven by the build out of 5G, the drive toward lowering capex with white boxes, lowering opex with automation, and growing rates of data consumption driven by new IoT devices, big data and AI, and ever more apps. They need to deliver on the big promises.

To do so, the answer so far has been to ship more and faster. But all too often, this leaves even the most touted ‘secure’ devices vulnerable to attacks of various kinds. Such defects could cost both Cisco and its customers orders of magnitude more to characterise and fix in a live network compared to being detected and fixed in development.

All of the Cisco vulnerabilities found were given a high impact security rating. No matter what we do, not all bugs will be fixed before software ships. However, the more bugs that can be caught earlier, the better. In so doing, the ability to record a failure or misbehaviour ‘in the act’, and replay the software execution that led up to the issue to understand the root cause of the defect, is invaluable. This enables engineers to truly understand the nature of any intermittent failures, both during testing and in production, and so better ensure software reliability.

This is most pertinent now. Cisco announced in March that it is to separate its underlying router and switch software from the hardware that hosts it. The move is designed to help the company meet growing demand for virtual network platforms from both hyper scale cloud providers such as AWS, Google and Microsoft, and the demand to support commodity white box hardware, driven by the world’s largest telecoms carriers.

As Cisco innovates to stay relevant within these markets, it will need to ensure that the foundation on which these technologies are built is constructed with software reliability in mind, and that when defects and vulnerabilities are found they can be solved and patched more quickly than in the past.