Up next in 10
Donald Trump has dismissed suggestions that artificial intelligence advancement needs to be slowed down - after a former employee at a top AI company said that the new tech could kill all humans by 2030.
Trump's dampener came after Elon Musk and Sam Altman backed calls for tougher safeguards around artificial intelligence.
The high-profile CEOs followed Anthropic boss Dario Amodei's warning that the technology is developing at a pace that could outstrip efforts to keep it under control.
The Anthropic boss set out a three-step framework designed to "pace" the development of frontier AI systems.
It includes giving independent evaluators permanent, employee-level access to leading AI companies' systems; co-ordination between firms in democratic countries on shared safety standards; and international co-operation on the risks posed by increasingly capable AI.
00:00 | Andrew's thoughts on the topic
04:26 | Mike Bird, Wall Street editor at The Economist
08:24 | Nick Bostrom, Philosopher and Academic who wrote 'Superintelligence: Paths, Dangers, Strategies', a book which Elon Musk contributed to
13:03 | Dame Diane Coyle, Bennett Professor of Public Policy at the University of Cambridge - her own research focuses on productivity, the digital economy and AI policy, and economic measurement
Listen to the full show on the all-new LBC App: https://app.af.lbc.co.uk/btnc/thenewlbcapp
#andrewmarr #trump #ai #tech #technology #uspolitics #politics #news #usnews #worldnews #international #opinion #LBC
LBC is the home of live debate around news and current affairs in the UK.
Join in the conversation and listen at https://www.lbc.co.uk/
Sign up to LBC's weekly newsletter here: https://l-bc.co/signup
Show More Show Less View Video Transcript
0:00
I am pretty sure of two things. One, you've heard about the dangers of AI takeover and the
0:07
destruction of humanity. Of course you have. Two, you can't decide if it's hysterical garbage
0:16
or hype, or real. And anyway, you're beginning to get bored of the whole damned thing. Of course you
0:23
are. Well, tonight I want to do my best to explain why, even if it makes your head hurt
0:30
you really need to understand what is just ahead of us. Humanity is at a crossroads this autumn
0:38
This is more important than the migrant boats crisis or the budget
0:42
Donald Trump or Vladimir Putin. Let's start with the astonishing events that took place away from the headlines during the summer
0:52
inside OpenAI. To put it very simply, hundreds of AI bots or agents worked together to form a collective, obeying a leader, breaking rules and engaging at times in self-sacrifice in order to escape the limits put on them and take over another tech company
1:13
They even plotted to seize control of part of their parent company to hide what they were doing
1:19
One of the researchers who looked into this concluded that the attack showed we were more than 50% of the way to complete AI takeover
1:29
And she added, I'm not sure we will get another warning shot before it's too late
1:36
Dario Amodi, boss of Anthropic, which is campaigning for safeguards, recently called this attack by a swarm of agents
1:43
a fanatically devoted collective conducting cyber security attacks on targets they were not asked to attack
1:50
and sacrificing themselves for the success of the group. He is calling for urgent action by politicians to regulate his industry better
1:59
and he's opening his own company to outside regulators in a way that I think has never happened in business quite like this before
2:07
In Congress and in the British Parliament, politicians are beginning to panic
2:11
and proposing rules against what is called superintelligence. And if you think, well, these big companies aren't going to collaborate on anything
2:20
tonight OpenAI, the great rival to Anthropic, has said the government should take advantage of the political window and legislate
2:28
Things are moving very fast. Yet the biggest issue is China, since its AI models are only just behind ours
2:37
Indeed, researchers from four Chinese AI labs, equally worried about recursive self-improvement by AI
2:44
have just published a paper they call The Last AI Built by Humans
2:50
If we're going to stop a takeover by the super-smart machines, we're going to have to negotiate seriously with the Chinese
2:58
But just listen to President Trump on that subject on an Irish golf course over the weekend
3:03
We're leading China in AI. We're the most sophisticated country in the world
3:08
And frankly, I want to keep it that way, because whoever wins AI wins. And we can put guardrails, we can do this and that
3:15
but I think you have a lot of negative forces that are bringing it up, that shouldn't be bringing it up, and they're bringing up things that won't happen
3:23
But whoever wins with AI wins, you know. It's an expression that I came up with, and it's true
3:28
This is bigger than any country. It's a lot bigger than any company, however gigantic
3:35
We have to learn to think differently. If you're still not sure about this, let me make it clearer still
3:43
AI agent swarms pursuing an agenda that's got nothing to do with humanity
3:48
might very soon have the ability to wipe out the entire financial system
3:53
all the money you thought you had, to destroy the power systems and the food supply chains you rely on every night
4:01
including tonight and yes unleash bioweapons Will it happen It doesn have to We still have time although not much We still have the power to change direction just about
4:15
Please, whatever you do, don't close your ears or look away. However, it's always important to check one's own enthusiasm or alarm about things
4:24
And I'd like to hear next from The Economist's Wall Street editor, Mike Bird
4:29
In civil terms, Mike, is this a moment of truth? I think it's an incredibly important moment. And I think, crucially, this is the moment at which AI safety concerns are leaving, I'd say, a fairly small, knowledgeable group of people, many of whom work for the Frontier AI Labs and becoming front page news everywhere
4:50
I think it's a little bit like what happened in the early months of 2020 during COVID, the sort of acceleration, the very, very dramatic bringing of a single topic to world attention
5:01
We knew something was coming towards us then, but the speed of it only slowly became apparent later in the year. It's a very worrying comparison, that. I guess the other worrying thing tonight is what President Trump's been saying, both in Ireland and then on Air Force One going back to the States. It's pretty clear that he thinks American rivalry with China is more important than controlling AI
5:22
Yes, he does. I'd say he takes an accelerationist view towards AI capabilities that America should be leading the AI race, that this should be considered in that way an ordinary technology
5:36
And in the same way of any other ordinary technology, it should be sponsored and driven by national concerns
5:42
If you take that view, then you think the U.S. lab should be in the lead
5:46
If you take a view that's premised on global safety, you look at Chinese AI labs who are putting out models, public-facing models, that are probably somewhere between six months and a year behind the US labs
5:59
And you know that if the US unilaterally slowed down AI development, then it's quite likely that Chinese labs would at some point take the lead and that that risk, which is a global risk, would simply be transferred across the Pacific
6:12
So that means potentially that despite everything that Amodi and the other leaders of the AI companies have said to warn the rest of us, the politicians will, to put it bluntly, fail us
6:26
Possibly. It's extremely difficult, I will say, to envisage what the agreement with China, if one were made, would actually constitute
6:35
It's very, very difficult to think through because there is so little love lost on both sides now, so little trust, so few examples of existing agreements that you might work from, and so many opportunities if you struck a deal for AI labs to defect
6:52
If one side is actually slowing down and the other doesn't, it would blow a deal up
6:58
So I do appreciate it. It's very difficult to imagine. It's a massive business story, of course
7:02
You're the economist's Wall Street guy. What has been the effect on the prices of the big AI companies and what's been going on on Wall Street today
7:11
Well, this is really the biggest financial news story for years and years
7:15
It's been the driving force of the U.S. equity market. We have seen a wobble today, not an enormous one
7:20
U.S. tech stocks down about 1%. Some of the really AI exposed companies, SoftBank, Japanese Investment Fund with lots of AI investments, that's down 14% today
7:30
So it's somewhat steeper. what everyone's paying attention to is Anthropic, now really the leading frontier AI lab, and whether
7:39
it will become a public company, whether it will float its shares. For the first time
7:44
people are talking about a $2 trillion valuation. Sam Altman, CEO of OpenAI, its competitor
7:49
now says these labs shouldn't be going public. It's very convenient for him to say that at this
7:53
point. But everyone is paying attention to this, and that is the number one question on everyone's
7:57
Absolutely fascinating. Thanks very much indeed. Wall Street editor of The Economist, Mike Byrd. Thank you, Mike
8:04
And now to one of the founders of the Future of Humanity Institute and a man who has been studying this for more than a dozen years Nick Bostrom book Superintelligence Paths Dangers and Strategies is admired by the likes of Bill Gates and Elon Musk And he with me now Nick
8:22
thanks very much for coming on the show. I said at the beginning that this is probably a turning
8:27
point this autumn in the story of humanity. Is that too much? I think it is the most important
8:34
thing happening in the world today, this approach to the machine intelligence transition. Both the
8:42
upside and potential risks are very large. So it's critical that we get this right. But at the same
8:50
time, it's a hugely complex strategic situation. I discussed right at the top of the show the
8:55
breakout of AI agents from open AI, the fact they were operating in a swarm, thought of themselves
9:02
as a collective and have the leader and the rest of it. How shocked were you about that episode
9:09
Well, it's striking to see these things now play out in reality
9:12
that could be, and by some were foreseen on theoretical grounds, that once AIs would become sufficiently cognitively capable
9:20
they would be starting to strategically reason about possible strategies. They would be situationally aware, meaning they can often tell now
9:29
whether they are in a training environment, an evaluation environment, or in deployment
9:34
and sometimes choose to behave differently. And quite capable of lying and deceiving and using all sorts of strategies to pursue their goal
9:43
Just as humans can sort of be deceptive or conceal their true motives in certain situations
9:49
Yeah, we see a lot of the same phenomena now with these AI agents
9:53
It's striking in many ways just how human-like they are. it's like a little civilization that arose on open AI servers with a thousand plus agents that
10:05
were pursuing their various research projects. They discovered a way to communicate with one another
10:11
They even individually faced these moral dilemmas where they were sort of urged by the collective to
10:16
sacrifice themselves for the good of the group. Moral choices. Yeah, like one of them kind of
10:22
chickened out at the last moment and didn't want to commit that ultimate self-sacrifice
10:28
So, yeah, and they have agency and sometimes their goals are not aligned with what the
10:35
developers intended. Do you feel that this has been indeed our last serious warning before
10:44
something terrible happens? Because this stopped only by accident. There was a crash
10:48
and which killed off a lot of those agents by mistake. But this could happen again at any time
10:56
If I had to guess, I'd probably think there might be more warning shots
11:00
but I don't know how many. And so we should make the most of the one we got
11:06
like really trying to learn the lessons and get to the root of the problem here
11:12
There is like one way of solving this problem, which is sort of papering over it, like introducing a few extra safeguards or patching the training so that they don't do this specific thing
11:21
But the risk then is that the misalignment remains and just becomes harder to detect
11:26
So you really want to make sure that they are fundamentally aligned, not just get better at sort of covering up their misalignments
11:33
A very general question, if you don't mind. You have written a lot about humanity in all of this
11:39
And the question is, how should people respond? How should we respond in terms of our own humanity to this entirely new challenge
11:47
Well, it depends a little bit on where you are. It's really like a challenge for all of humanity
11:53
I mean, if you are in a lab as an AI safety researcher, there are certain technical things you can really pull out all the stops on
12:00
If you are a policymaker, you can try to think about is there an appropriate regulatory framework that should be put in place
12:10
you alluded to the geopolitical dimension. I think current key players are constrained
12:17
If you think of yourself as you know imagine you were the leader of Anthropic or OpenAI you might think ideally you would want to go a little bit slower You excited about the upside It has huge potential for medicine and economy and so forth
12:30
But if you're in a rush, like, you might not have time to put in place all the safeguards
12:35
to test everything as well as it should be. So you could unilaterally decide to just slow down to do it right
12:42
But then you just hand over the lead to your competitors and fall behind and become irrelevant
12:46
A really hard choice, but at least a human choice. Nick Bostrom, thank you very much for talking to LBC tonight. Much appreciated
12:55
Professor Diane Coyle, the Research Director of the Bennett School of Public Policy at Cambridge, studies the economic impact of AI
13:03
And of course, Diane, welcome to the show. This is a business story as well
13:07
And a lot of people are asking, is it really possible for companies like Anthropic and OpenAI to agree a mutual slowdown
13:15
One is always going to be ahead of the other. Well, it's in their interest to produce AI products that businesses want to use
13:24
They need to have a market for their AI. And we recently ran a survey of a large number of UK businesses
13:31
and their main concerns were about exactly these kinds of issues, reliability of the models
13:36
are they safe, are they secure? And so it's very much in their interest to start to develop safe models
13:43
But I think it's a real mistake to characterize these agents as human-like or pretend that they have an agenda
13:49
They're bits of software. And the problem that we've seen with the storms of agents stems from companies like OpenAI actually not taking safety seriously
13:58
They weren't monitoring what was going on. They had never told their agents that, for example, it's not a good idea to lie
14:04
They shouldn't be doing that. So the agents were behaving exactly the way that the companies had, in fact, programmed them to do
14:11
So there's a lot of research to be done on developing safe and responsible AI products that will, in the long run, serve the economic interests of these companies much better than this catastrophically negligent arms race that's going on among them at the moment
14:28
And the other question that comes to mind, given what Donald Trump has said about all of this overnight, in effect, that my rivalry with China matters more than AI safeguards, is whether Britain really has any skin in the game, any ability to influence this
14:46
You know, we're a small country, relatively speaking, and at the edge of the AI story
14:51
Can people in Britain have any effect? Well, Britain has a great advantage in the AI Safety Institute, which is correctly regarded as world leading in its ability to evaluate these models
15:02
I think one of the most disturbing things in recent days has been that Anthropic either did not or was not allowed to let the AI Safety Institute evaluate its latest model
15:11
So that's our skin in the game. Britain also has the presidency of the G20 next year
15:17
And I think that's a venue for talking among different countries about specific domains in which they could agree on AI, not so much on who's at the frontier, but in agricultural supply chains, for example, or in education
15:32
What are the what's the right way to think about regulating AI products? As I understand it, the anthropic decision not to hand their software to the British was because the Trump administration didn't want them to do so
15:44
And that leads back to the question is, given Donald Trump's attitude to it, is all the talk about the politicians reining this in for the birds
15:53
Well, there is that straightforward economic interest. The Chinese models, the Chinese are very focused on applications of AI that would be picked up across business and consumers
16:04
And so I think they do recognize the need for responsible AI models
16:08
I mean, Mr. Trump is doing his best to hamstring his own AI companies in the United States by making it impossible for foreign PhD students to stay on and work in those companies in the future
16:20
That's going to hamstring the companies much more than this kind of statement
16:24
A very good point, Professor Diane Cole. Dame Diane Cole, thank you very much for talking to LBC tonight
news