A 'Godfather of AI' Remains Concerned as Ever About Human Extinction (msn.com)
- Reference: 0179614774
- News link: https://slashdot.org/story/25/10/01/1422204/a-godfather-of-ai-remains-concerned-as-ever-about-human-extinction
- Source link: https://www.msn.com/en-us/money/other/a-godfather-of-ai-remains-concerned-as-ever-about-human-extinction/ar-AA1NF4Np
Bengio founded the nonprofit research organization LawZero earlier this year to explore how to build truly safe AI models. Recent experiments demonstrate AI systems in some circumstances choose actions that cause human death over abandoning their assigned goals. OpenAI recently insisted that current frontier model frameworks will not eliminate hallucinations. Bengio, however, said even a 1% chance of catastrophic events like extinction or the destruction of democracies is unacceptable. He estimates advanced AI capable of posing such risks could arrive in five to ten years but urged treating three years as the relevant timeframe. The race condition between competing AI companies focused on weekly version releases remains the biggest barrier to adequate safety work, he said.
[1] https://tech.slashdot.org/story/23/05/30/1149219/ai-poses-risk-of-extinction-industry-leaders-warn
[2] https://www.msn.com/en-us/money/other/a-godfather-of-ai-remains-concerned-as-ever-about-human-extinction/ar-AA1NF4Np
I...kinda want to watch. (Score:1)
I wish I could be in the room with Sam Altman when the AI releases the fungus that ends the carbon cycle on our planet. When he realizes every human on the plane is going to be dead and it was his fault. I won't be, but my death won't be without a certain bizarre satisfaction.
executing long chains of reasoning (Score:2)
Odd name for really, really ....... long if elseif type of statements. "chains of reasoning"??
Re: (Score:3)
Look up "Chain of Thought" for LLM. That's a terminus technicus for a very specific type of LLM output on which reasoning/thinking models are trained.
Re: (Score:2)
> Odd name for really, really ....... long if elseif type of statements.
No, it is not.
Though "chains of reasoning" is dubious for other reasons.
They're token streams that have an impact on the generation of the model.
In some cases, they're good at reasoning within them. In some cases, they're not.
And most confusingly- the output of the model isn't directly correlated to the quality of the reasoning, only that the token stream was generated.
OpenAI doesnt care (Score:3)
They are in only for the money. The rest of the planet, and the end of civilization? Who cares⦠They will be crawling in money with no one around to watch them.
Propose a mechanism that doesn't require trust (Score:2)
It would, indeed, be *highly* preferable to pause, or at least slow, AI development in order to design and implement safeguards. But there are multiple groups striving to capture the first-mover advantage. Anyone who slows development will be bypassed. And they aren't all under the same legal system, so that approach won't work either.
Consider [1]https://ai-2027.com/ [ai-2027.com] , That's a scenario that currently seems a bit conservative, if you check the postulated timeline against what's been (publicly) happening, t
[1] https://ai-2027.com/
Re: (Score:2)
Even promising to pause AI development while continuing to work on it in secret government labs still has the beneficial effect of slowing progress since you don't also have the private sector and universities throwing all of their resources at it.
The Big Dumb is here (Score:2)
Everything is getting dumber as upper level management continues to cling to their positions of power, completely unaware of their impact, for as long as they can. This is why "LLMs" and diffusion models (which are a red herring) can become useful, when the baseline is someone who never faces consequences.
Law Zero - Asimov Style (Score:2)
The information you provide MUST be provably true and correct
Re: (Score:2)
> The information you provide MUST be provably true and correct
That's not law zero. Law zero was an override that allowed short-term harm to humans for long-term benefit for humanity overall, allowing (can't remember the name) to do something to speed up the atomic decay on Earth to push people to start exploring the stars
Re: (Score:2)
Haha very true! He should have called it LawKelvin
Re: (Score:2)
This is the Zeroth law:
A robot may not harm humanity, or, by inaction, allow humanity to come to harm.
Re: (Score:2)
> This is the Zeroth law:
> A robot may not harm humanity, or, by inaction, allow humanity to come to harm.
Thank you. Been a while since I'd actually read it. I just remembered it caused an internal conflict for the bot that acted on it, due to the other laws sorta clashing with it.
AI isn't needed (Score:2)
We don't need AI to bring about human extinction: we're doing a fine job without it. How many of us are taking anthropogenic climate change seriously?
AI might bring about extinction quicker, but make no mistake: we're on the highway to hell, no stop signs, or speed limit.
Re: (Score:2)
I was thinking the same thing. We don't need AI to consciously act maliciously towards us. The natural path to entropy is underway: all carnival barkers promise the best show, but none of them have it. Who do you listen to? Management anxious not to be left behind are sprinkling AI on everything hoping to cut monthly costs and are killing their golden goose by losing institutional memory and will have no one to train to take over the business because they are not hiring entry level people. Once vibe program
it is impossible that the improbable won't happen (Score:3)
even a 1% chance of catastrophic events like extinction or the destruction of democracies is unacceptable
...it is impossible that the improbable won't happen.
The only solution is to limit individual systems... define a sort of kelly criterion for AI, where a single big failure does not mean extinction. e.g. for military robots, mandate any model/manufacturer/dataset/etc., be limited to say 5% of the entire robot fleet... don't let them share code, or collaborate. We *want* them to have different bugs. That way if there's a glitch/feature/emergence someplace, it's limited.
Same for medicine/treatments synthesized with the help of AI... only let 5% o the population to benefit from any individual "solution". That way if there's an extinction gene-editing virus, only 5% of the population is impacted, etc.
Re: (Score:3)
can I kick that around a bit?
I've been telling chumps for some time now that a mono culture of software and management ideas has lead us to a place where 1 level 10 vulnerability can open the way to hacking vast numbers of systems. So, obviously, no one listens to me, but I think that means your scenario won't happen. It seems to be common (groupthink) knowledge that buying or renting pre built software or software services is the ONLY way to go. Like using libraries and gluing a system together with Ruby o
Re: (Score:2)
GP’s idea is good but has high costs, eg vaccinating population requires 20x the the number of solutions of each solution is limited to 5%. It may be viable for global solutions but not for an individual company creating software. The closest o can think of is duplication or triplication of systems used in for example aerospace such as triple redundancy autopilot /autoland with independent sensors and implementation languages and hardware.
Re: (Score:2)
Great expression in your title. I’ve never heard that before, However, it is untrue with a finite number of “rounds”. Rolling a six is improbable. Not rolling a six in 10 rounds is possible.
We have seen (Score:2)
We have seen all the movies and have seen what the outcome is. He is only trying to drive that point home and no one is listening. So when all the bad things in the movies come true, humans have no one but themselves to blame!
Plots (Score:2)
I saw that movie. The PR lady falls in love with the robot that looks like John Malkovich. I'm not sure what's wrong with that.