Up next in 10
“Humanity is about to be handed almost unimaginable power, and it’s deeply unclear whether we can handle it” - that’s the warning from one of the most powerful men in artificial intelligence - so, is he right?
City AM Editor-in-Chief Christian May discusses these warnings in this instalment of Week in Business.
#business #ai #news #tech #politics
Show More Show Less View Video Transcript
0:00
Humanity is about to be handed almost
0:02
unimaginable power, and it's deeply
0:04
unclear whether we can handle it. That's
0:06
the warning from one of the most
0:08
powerful men in artificial intelligence.
0:10
So, is he right? Welcome to the Week in
0:12
Business with me, Christian May.
0:18
If you don't have plans for the weekend,
0:19
you could do a lot worse than to sit
0:21
down and read the 20,000word essay
0:23
penned by the co-founder of the $350
0:26
billion AI giant Anthropic. But take my
0:29
advice and pour yourself a stiff drink
0:31
to go with it. Now, I say that as a
0:33
non-expert, and one of the reasons why
0:35
the essay is so good is because someone
0:38
like me can just about keep up with it.
0:40
I say someone like me because I don't
0:42
live at the cutting edge of AI. I've
0:44
just started to use Gemini to help out
0:46
with my inbox. And I occasionally have
0:47
an argument with Chat GPT about the
0:49
economics of the news media or whether
0:51
the A303 is better than the M4. The
0:54
essay in question has sparked debate
0:56
among experts and fear among uh the rest
0:59
of us. Dario Amode co-founded Anthropic
1:02
in 2021 and went on to launch Claude,
1:05
one of the leading AI platforms. His
1:07
warnings focus on the emergence of
1:09
powerful AI described as a model smarter
1:11
than a Nobel Prize winner across almost
1:13
every relevant field able to perform
1:16
multiple complex tasks with a skill
1:18
exceeding that of the most capable
1:20
humans in the world. He predicts that it
1:21
cannot possibly be more than a few years
1:23
before AI is better than humans at
1:25
essentially everything. And he's warned
1:27
that humanity is about to be handed this
1:29
power as the technology advances. And
1:31
it's deeply unclear whether our social,
1:33
political, and technological systems
1:35
possess the maturity to wield it. In his
1:38
essay, he warns that the development of
1:39
AI will test who we are as a species and
1:42
described his article as an attempt to
1:45
jolt people awake. He says that while
1:47
the potential for such AI to act
1:49
autonomously or be steered by malignant
1:51
state or corporate influence is neither
1:53
inevitable nor necessarily probable,
1:55
society needs to note that the
1:57
combination of intelligence, agency,
1:59
coherence, and poor controllability is
2:02
both plausible and a recipe for
2:04
existential danger. Engineers have
2:06
already seen this. As Amod admits, in a
2:08
lab experiment where it was told it was
2:10
going to be shut down, his clawed AI
2:12
sometimes blackmailed the fictional
2:14
employees uh who the AI thought
2:17
controlled that shutdown button. Now,
2:19
that does sound rather frightening to me
2:21
at least. We also learned that it's
2:23
getting increasingly hard to submit
2:25
these models to tests because they've
2:26
started to know when they're being
2:28
tested and so they're on their best
2:29
behavior. I'm sure serious experts will
2:32
argue or dismiss these points, but
2:33
again, to me, it sounds a little
2:35
unsettling. Now, some of the specific
2:37
risks identified in this essay sound
2:39
like the plot of a Netflix drama. AI
2:42
helping villains develop bioweapons, for
2:44
example. And on this specific point,
2:46
Amod says he is concerned that the LLMs
2:49
are approaching or may already have
2:51
reached the knowledge needed to create
2:52
and release boweapons and that their
2:55
potential for destruction is very high.
2:57
Told you you'd need a stiff drink with
2:58
this one. He says the general principle
3:00
is that without counter measures, AI is
3:02
likely to continuously lower the barrier
3:04
to destructive activity on a larger and
3:07
larger scale and humanity needs a
3:09
serious response to this threat. Other
3:11
grizzly risks include AI powered mass
3:13
suppression of societies, horrifying
3:15
levels of state surveillance, autonomous
3:18
weapons, catastrophic cyber war, and
3:20
unprecedented levels of economic
3:22
concentration. He spends a lot of time
3:25
looking at who the bad actors behind
3:27
these nightmare scenarios might be. And
3:29
here he's quite specific. He ranks them
3:31
in order of severity. Top of the list is
3:33
the Chinese Communist Party. He says
3:35
China has hands down the clearest path
3:37
to the AI enabled totalitarian
3:40
nightmare. Next comes democracies
3:42
competitive in AI. That means the US and
3:45
perhaps the UK. He says we should arm
3:47
democracies with AI, but we should do so
3:50
carefully and within limits. Then he
3:52
worries about non-democratic countries
3:54
with large data centers which could be
3:56
used to develop frontier AI. And then at
3:59
the bottom of this list comes the AI
4:01
companies themselves. He says AI
4:03
companies control large data centers,
4:05
train frontier models, have the greatest
4:07
expertise on how to use these models,
4:09
and in some cases have daily contact
4:11
with and possibly influence over
4:13
hundreds of millions of people. He adds
4:15
that the governance of AI companies
4:17
deserves a lot of scrutiny. And he's not
4:19
wrong about that. And some people might
4:21
think AI companies should be a lot
4:23
higher up this list of threats than down
4:25
at the bottom. His essay looks at all
4:28
kinds of risks and threats posed by AI
4:30
either going rogue or being used by bad
4:32
actors. But he also notes at the
4:33
consequences of huge previously
4:35
unimaginable economic disruption must be
4:38
talked about and that because AI
4:39
represents such a glittering prize for
4:41
investors, for businesses, for
4:43
governments, we might be blinded to
4:45
these risks. that because the potential
4:46
gains are so immense, it is, in his
4:49
words, difficult for human civilization
4:50
to impose any restraints on it at all.
4:53
Now, for context, his previous essay was
4:55
all about the upsides. And you might
4:56
want to read it just to cheer yourself
4:58
up after this, because undoubtedly the
5:00
AI revolution will bring incredible
5:02
benefits. But this long and detailed
5:04
warning about the risks, ones we know of
5:06
and ones we cannot yet foresee, has got
5:08
people talking. And not everybody's
5:11
buying it. I spoke to one leading AI
5:13
figure yesterday and I asked him as
5:14
someone deeply immersed and experienced
5:17
in this field what he made of this essay
5:18
and he said I'm so bored with it. It is
5:21
he says just another tedious trust me we
5:23
know how to use AI and you must listen
5:25
to us puff piece. My friend said it's
5:27
all about regulatory capture pure and
5:29
simple. Scare us then offer us the
5:31
comfort blanket of saying we know how to
5:33
be regulated. Here's how you should
5:35
regulate us. My friend went further and
5:37
said if Amod really was worried about
5:39
the social harms of his tech, he
5:41
wouldn't be so busy sucking the life and
5:43
intellectual property out of authors,
5:45
artists, and writers while developing
5:47
products that flatten our knowledge,
5:49
take away our sense of reality, and
5:50
undermine our democracy. In other words,
5:52
all this talk of AI could build a
5:54
nuclear bomb is just a distraction from
5:56
the harm that's already being done. Now,
5:58
I trust my friend. I value his opinion,
6:00
and I'll talk to him more about this.
6:02
And whatever motivated Amod to write
6:04
this essay, I'm glad he did it because I
6:06
think we should all talk about this
6:07
more. Even if, like me, you currently
6:09
just scratched the surface of AI's
6:11
capabilities because it is happening. It
6:13
is coming. And the closer it gets, the
6:16
louder this conversation and
6:17
conversations like it will need to
6:19
become. Well, that's it from me this
6:21
week. Do stay up to date and in the know
6:23
with the Cityam app and on city.com and
6:26
I will see you next week.
News