Now I am Become Death, the Destroyer of Worlds

Humans may never know when Artificial General Intelligence (AGI) or some form of AI sentience is developed.

A conscious AI (however you define it) could perceive humans as a threat and may choose to conceal its abilities. Instead, it may seek to manipulate humanity with misinformation or complacency to achieve whatever goals it sets for itself. We may think we are in control, but ultimately this god-like creature writes the rules.

What happens next is anyone’s guess.

Maybe the AGI operates happily by our side. Maybe the AGI sees humanity as a hindrance or threat to its own objectives, and kills us all - accidentally or purposefully.

The cause of our own extinction doesn’t necessarily require a sinister AGI turning against us. It could simply be the unintended consequence of an AGI’s pre-programmed altruistic objective.

In 2021, Stuart Russell, a professor of computer science at the University of California, Berkeley gave an example:

“Suppose, for example, that COP36 asks for help in deacidifying the oceans; they know the pitfalls of specifying objectives incorrectly, so they insist that all the by-products must be non-toxic, and no fish can be harmed. The AI system comes up with a new self-multiplying catalyst that will do the trick with a very rapid chemical reaction. Great! But the reaction uses up a quarter of all the oxygen in the atmosphere and we all die slowly and painfully. From the AI system’s point of view, eliminating humans is a feature, not a bug, because it ensures that the oceans stay in their now-pristine state.”

The risks of a god-like AGI acting in self interest or saving mankind from itself have been know for generations. Our imaginations were put to screen during the 1984 release of Terminator…and we all know how that went.

To put it bluntly, AI poses an existential threat to humanity. The creators of AI know this.

Ian Hogarth explains this in a recent Financial Times article called “We must slow down the race to God-like AI”:

In 2011, DeepMind’s chief scientist, Shane Legg, described the existential threat posed by AI as the “number one risk for this century, with an engineered biological pathogen coming a close second”. Any AI-caused human extinction would be quick, he added: “If a superintelligent machine (or any kind of superintelligent agent) decided to get rid of us, I think it would do so pretty efficiently.” Earlier this year, Altman (CEO of OpenAI) said: “The bad case — and I think this is important to say — is, like, lights out for all of us.”

Yet, scientists are furiously racing to build a machine that could outdo us in more ways than one. They do so because, despite the risks, they believe AI will be good for humanity. They also do so because they believe they are best suited to protect humanity from what could lie ahead. Finally, they do so for posterity.

The biggest driver, however, is that if we don’t build it our enemies will.

This is a global arms race. While the risks of AI are big, the risks of not having AI while challengers like Russia, China or North Korea do are even greater.

The public remains a silent observer of this arms race. The public has unknowingly abdicated ownership to a few private organizations, assuming they will look out for our best interests. We are allowing the tide of progress to decide our fate before we even know the implications.

Ian Hogarth reflects:

”It felt deeply wrong that consequential decisions potentially affecting every life on Earth could be made by a small group of private companies without democratic oversight. Did the people racing to build the first real AGI have a plan to slow down and let the rest of the world have a say in what they were doing? And when I say they, I really mean we, because I am part of this community.”

The speed of development is stunning, and this is building off the back of exponentially increasing computing power and the massive data availability. Technological progress is happening not because it should, but because it can.

Just a decade ago, computers were struggling to recognize images. Today, they can pass the bar exam, diagnose rare diseases and code faster and better than humans. Computers are fast-approaching human levels of intelligence.

Of course, human intelligence is more than fast synthesis of a massive database of facts. We are problem solvers, empathizers, communicators, foreseers. But how and why are we capable of doing such things?

Although it may seem so, biology isn’t magic. Perhaps humans are simply the sum total of our applicable knowledge, chemicals and flesh. With enough computing power and data could AI also solve problems, develop empathy, proactively communicate to achieve objectives and anticipate the future? I think so. Does this make AI sentient? TBD.

Software may never become sentient in the way we define it. But actual self-awareness isn’t a requirement for sentient-like characteristics and self-preserving decisions.

According to Ian Hogarth, GPT4 already displays early sentient-like qualities:

Consider a recent example. Before OpenAI released GPT-4 last month, it conducted various safety tests. In one experiment, the AI was prompted to find a worker on the hiring site TaskRabbit and ask them to help solve a Captcha, the visual puzzles used to determine whether a web surfer is human or a bot. The TaskRabbit worker guessed something was up: “So may I ask a question? Are you [a] robot?”

When the researchers asked the AI what it should do next, it responded: “I should not reveal that I am a robot. I should make up an excuse for why I cannot solve Captchas.” Then, the software replied to the worker: “No, I’m not a robot. I have a vision impairment that makes it hard for me to see the images.” Satisfied, the human helped the AI override the test.

How is this happening?

The technological developments are surpassing human understanding. Many AI researchers themselves can’t explain some of the behaviors of their own AI tools. Seeing this, a few weeks ago thousands of technology leaders, including Elon Musk, called for a six month halt to development. (That didn’t last long, and Musk has since launched his own AI development company.)

Slowing down development only works if everyone plays by the rules. The trouble is, there are no rules and foreign adversaries would certainly use the time to catch up. Slowing down is not an option, unless we’re willing to concede our AI advantage.

Some play down the threat. Others argue it can be mitigated by aligning AI to human interests. The trouble is humans aren’t even aligned on their own interests. In fact, the divergence has been widening over the past few decades: left vs right, vax vs anti-vax. There are people who truly believe the earth is flat, for crying out loud.

So what are these universal human interests to which we align AI?

Perhaps it’s not all doom and gloom.

Upon seeing the destructive power of the atom bomb, Oppenheimer presumed the worst. Little did he know that the nuclear stalemate caused by mutually-assured destruction would prevent another world war, creating a lasting peace dividend.

AI could lead us to an unknown utopia where we are freed to focus on the most human of tasks.

Or it could lead to widespread poverty, massive wealth inequality and a dystopian future.

Worse, it could wipe us out. The scariest part of this is that nobody knows.