There has been a lot of discussion lately about the risks posed by increasingly autonomous AI systems. Some predictions are unsettling, even catastrophic.
As I read them, I keep coming back to something from my years in engineering: failure-mode analysis.
Failure-mode analysis wasn't something we performed after a system was designed. It helped shape the design itself. We assumed equipment would fail, operators would make mistakes, and combinations of events would occur that no one intended.
That makes me wonder whether the same philosophy should be central to autonomous AI.
If an AI system can act at machine speed, perhaps its safety cannot depend solely on instructions embedded within that same AI. An independent control layer could monitor proposed actions in real time and enforce a predefined operating envelope before those actions reach the outside world.
Suppose an AI agent determines that accessing another computer system would help accomplish its objective. The safeguard shouldn't depend entirely on telling the AI, "Don't do that." An independent protection system could simply prevent communication with anything outside the systems the agent has been explicitly authorized to access.
Of course, that immediately raises the next engineering question:
How can the safety system itself fail?
Can the AI circumvent it? Can someone inadvertently change its constraints? Can two individually acceptable actions combine to create an unacceptable result? What happens if the protection system itself malfunctions?
And then the failure-mode analysis begins again.
Interestingly, I explored a version of this problem when I wrote my novel, The Heaven Dimension.
Two people independently compromise different safeguards on an experimental system. Each has his own reasons. Neither intends to create a disaster, and neither fully understands what the other has done.
But the safeguards were independent for a reason.
Remove both, and something becomes possible that neither person anticipated.
When I wrote those scenes, I was constructing a thriller. Today, I can't help seeing the parallel with the discussion surrounding autonomous AI.
Perhaps the most dangerous failure scenario isn't an AI suddenly deciding to become our enemy. It may be something much more familiar to engineers: a complex system in which individually reasonable decisions, unexpected interactions, or common assumptions quietly defeat the safeguards intended to protect us.
The objective isn't to design a system that can never fail. That standard has never been realistic for complex systems. The engineering challenge is to understand how it can fail, provide independent layers of protection, and ensure that no single failure, human or machine, can become catastrophic.
As AI systems become more autonomous, are we applying the same discipline to their failure modes that we have long applied to other systems whose failure can have serious consequences?
Add comment
Comments