With some recent discussions online here about the general safety of Atomic energy and at the space about the perceived improvement in safety of self-driving cars, I thought this post today on Hack-a-day was appropo.
A fair and balanced article, but I do take exception to some of its conclusions, in particular
This conclusion seems to imply that with proper software and system engineering it is possible for flawless software to be created. I think this is patently false on a practical level. In distance history, I took a course that used a lot of math to establish that a particular piece of code was sound. And with small to modest complexity code it is possible to establish flawlessness with a mathematical certainty. However, as the code size increases, the complexity of those calculations grows exponentially.
Which might explain why such analysis is not performed on complete applications, and why we have accepted that software bugs are inevitable. Granted commercial software doesn’t receive the same QC/QA that say a medical device does (usually), but we still to this day have medical devices that get recalled, because of bugs. The reason is simple, we can not produce a new device with mathematically proven software in time frames we can live with, nor can we afford to perform that level of review.
Given that perfect software seems unattainable how much damage could a software bug cause if it is in a popular car model? Consider that our nation drives about 2.5 billion miles per day, with only about 100 deaths per day in car crashes… Consider the failure of just one model 0.01% of the time it is operated?
And in the event of such a catastrophe, we would see much the same response that we did in the Therac instance above, a search for an immediate cause and lawsuits. But like those, even when they found the specific cause for most of the Therac deaths, they failed to fix the underlying problems and yet another bug caused yet another death.
Which begs the question…who would be liable, since that is the entity in need of insuring. If two self-driving cars collide and serious property damage or bodily harm results, where might the liability reside in that case?
The THERAC-25 was deeply flawed, to a really terrifying extent. The code wasn’t even looked at by anyone other than the man who wrote it, & the people who built the hardware didn’t include any interlocks, because they assumed the software could never go wrong.
Mathematically validating every part of the system is one thing ; noticing an obvious race condition, or the fact that using the backspace key would result in random values being entered into the memory registers, is another thing entirely. If even the vaguest attempt at validation had been made, people’s lives would have been saved.
The conditions were hardly obvious. In the three years during which the events took place, more then 10,000 man hours were spent reviewing the code and testing the system. They found neither bug.
But the real point is that no complex system is bug free.
Have you read the Leveson/Turner paper from IEEE Computer (1993 July)?
It’s damning. There’s a big difference between the presumed impossibility of thoroughly validating a highly complex system, & the THERAC-25.
In Texas, you either have to have state minimum insurance or post a bond in lieu of insurance (note: that’s the law, but about 35% of my premium is for uninsured or under-insured drivers). As to who is liable? Your car, it’s in an accident, you will be named as a defendant. Whether you are ultimately liable is for a judge or jury to decide - but you will be paying for a defense attorney out of your pocket up front if you are not insured. You can course sue someone else and hope they are found liable.