The Deadly Consequences of Trusting AI: Eight Lessons from WarGames
"Shall we play a game?"
Yesterday, I rewatched a classic '80s movie: War Games. It's a great story that keeps you at the edge of your seat. But as I watched, despite being created in 1983, it felt highly relevant.
The gist of the story is a high school hacker and his friend, played by Matthew Broderick and Ally Sheedy, innocently start playing a war game against an AI. This AI, however, is in control of the US nuclear warning system and cannot distinguish between reality and fantasy. The computer simulates an attack and the system overseeing the nuclear arsenal believes the Soviets just launched a full scale first strike.
The threats of nuclear apocalypse posed by misguided logic and AI in this 1983 movie are no different than today.
Here are 8 lessons the movie shares that are highly relevant today:
1. Some humans will inevitably oppose violence as the answer
Early in the movie, when a test simulation begins, some human missile launch officers refuse to turn their keys. Confronted with the reality of vaporizing millions of lives, empathy and moral restraint override orders. This refusal of orders was the catalyst in the movie for later replacing humans with unemotional computer program, setting up the cascading failures created by AI.
In the movie, this hesitation was viewed as a failure. However, I would argue unflinching compliance is the true failure. If a nuclear war is unwinnable there is never a case for launching a nuclear weapon, even in retaliation against an imminent attack, regardless of whether the threat is false or real. Only a human can understand this.
Individual reluctance to kill and hesitancy to believe others could make the decision to kill has saved us from nuclear annihilation on a few occasions.
During the peak of the Cuban Missile Crisis on October 27, 1962, US Navy destroyers trapped a Soviet submarine and began detonating depth charges to force it up. Believing war had already broken out, both the submarine's captain and its political officer agreed to fire a nuclear-armed torpedo. The launch required unanimous consent from three officers, and Vasily Arkhipov, the flotilla commander on board, stood alone in refusing permission. Instead, he forced the vessel to come to the surface, single-handedly preventing a full-scale nuclear war.
On September 26, 1983, Stanislav Petrov was monitoring a Soviet early-warning bunker when the computer screens flashed alerts indicating multiple incoming American intercontinental ballistic missiles. Rather than passing the alarm up the chain of command, which would have triggered an automatic Soviet counter-strike, Petrov gambled that the alerts were a satellite malfunction. His calculation proved correct, preventing a catastrophic nuclear retaliation based entirely on a system error.
Sarah Connor
2. Artificial Intelligence isn't intelligent
The WOPR supercomputer (the name of the AI overseeing the nuclear alert system in the movie) does not understand death, geopolitics, or the value of human life. It simply processes instructions and executes calculations at scale. True intelligence requires comprehension, understanding, and empathy whereas the machine only possesses brute-force computational power.
Abdicating analysis and decision power to a computation, even if considered 'optimized' by its creators, will inevitably lead to unintended outcomes.
3. AI is made by flawed humans who give incomplete or flawed instructions
Stephen Falken, an AI scientist in the movie, designed WOPR that reflected his own biases, grief, and intellectual isolation. When programmers translate complex, messy human realities into code, they inevitably omit critical nuances. A machine is limited by the imagination of its creators and users.
4. AI is motivated to complete its designed task, regardless of how unrealistic or illogical
AI does not possess self-awareness to question the utility of its objective. When WOPR begins the game "Global Thermonuclear War," its only objective is to "win". The game can't end until there is a winner because, according to its understanding, that's the point of playing a game.
AI lacks the capacity to pause, re-evaluate, or consider alternatives that aren't built into its programming.
5. There may not be a tomorrow. Live your life today.
Falken retreats to a remote island, convinced that humanity’s extinction is a mathematical certainty. His nihilism reminds us to balance tomorrow with today. While visiting Falken, the main characters, learning of his unwillingness to help and facing their mortality, suddenly realize that the simple things they've put off are what they'll miss the most.
6. The illusion of control creates vulnerability
NORAD leaders handed control to WOPR to eliminate human hesitation from the nuclear chain of command. The aim was to gain more control by centralizing decision-making. Instead, by consolidating control they created a massive single point of failure, effectively losing control when the system didn't work according to plan. The pursuit of perfect control through automation and centralization often introduces catastrophic, unpredictable vulnerabilities.
7. A simulation becomes dangerous when the system cannot distinguish it from reality
WOPR could not differentiate between a training exercise and an actual Soviet first strike. When a system cannot separate a model from the real world, it will confidently act on false assumptions with real-world consequences. How do you teach an AI what is real and not real?
8. The only winning move is not to play
By running countless iterations of tic-tac-toe and nuclear war scenarios, the computer eventually learns that some games cannot be won. When victory is impossible a game becomes pointless. In the case of nuclear war, where mutual destruction is guaranteed, the only logical strategy is total avoidance.
This movie aged well and, despite the 1980s hardware, the lessons seem like they were curated for today's military leaders increasingly incorporating AI into military warfare. If you haven't seen it in a while, I suggest borrowing a copy from your local library.
