OpenAI's Next Big AI Effort GPT-5 is Behind Schedule and Crazy Expensive (msn.com)
- Reference: 0175729251
- News link: https://slashdot.org/story/24/12/22/0333225/openais-next-big-ai-effort-gpt-5-is-behind-schedule-and-crazy-expensive
- Source link: https://www.msn.com/en-us/money/other/the-next-great-leap-in-ai-is-behind-schedule-and-crazy-expensive/ar-AA1wfMCB
[2]Alternate URL here .] But "OpenAI's new artificial-intelligence project is behind schedule and running up huge bills. It isn't clear when — or if — it'll work."
"There may not be enough data in the world to make it smart enough."
> OpenAI's closest partner and largest investor, Microsoft, had expected to see the new model around mid-2024, say people with knowledge of the matter. OpenAI has conducted at least two large training runs, each of which entails months of crunching huge amounts of data, with the goal of making Orion smarter. Each time, new problems arose and the software fell short of the results researchers were hoping for, people close to the project say... [And each one costs around half a billion dollars in computing costs.]
>
> The $157 billion valuation investors [3]gave OpenAI in October is premised in large part on [CEO Sam] Altman's prediction that GPT-5 will represent a " [4]significant leap forward " in all kinds of subjects and tasks.... It's up to company executives to decide whether the model is smart enough to be called GPT-5 based in large part on gut feelings or, as many technologists say, "vibes."
>
> So far, the vibes are off...
OpenAI wants to use its new model to generate high-quality synthetic data for training, according to the article. But OpenAI's researchers also "concluded they needed more diverse, high-quality data," according to the article, since "The public internet didn't have enough, they felt."
> OpenAI's solution was to create data from scratch. It is hiring people to write fresh software code or solve math problems for Orion to learn from. [And also theoretical physics experts] The workers, some of whom are software engineers and mathematicians, also share explanations for their work with Orion... Having people explain their thinking deepens the value of the newly created data. It's more language for the LLM to absorb; it's also a map for how the model might solve similar problems in the future... The process is painfully slow. GPT-4 was trained on an estimated 13 trillion tokens. A thousand people writing 5,000 words a day would take months to produce a billion tokens.
>
> OpenAI's already-difficult task has been complicated by [5]internal turmoil and near-constant attempts by rivals to poach its top researchers, sometimes by offering them millions of dollars... More than two dozen key executives, researchers and longtime employees have left OpenAI this year, including co-founder and Chief Scientist Ilya Sutskever and Chief Technology Officer Mira Murati. This past Thursday, Alec Radford, a widely admired researcher who served as lead author on several of OpenAI's scientific papers, announced his departure after about eight years at the company...
>
> OpenAI isn't the only company worrying that progress has hit a wall. Across the industry, a debate is raging over whether improvement in AIs is starting to plateau. Sutskever, who recently co-founded a new AI firm called Safe Superintelligence or SSI, declared at a recent AI conference that the age of maximum data is over. "Data is not growing because we have but one internet," he told a crowd of researchers, policy experts and scientists. "You can even go as far as to say that data is the fossil fuel of AI."
>
> And that fuel was starting to run out.
[1] https://www.wsj.com/tech/ai/openai-gpt5-orion-delays-639e7693?st=ng5hBi&reflink=desktopwebshare_permalink
[2] https://www.msn.com/en-us/money/other/the-next-great-leap-in-ai-is-behind-schedule-and-crazy-expensive/ar-AA1wfMCB
[3] https://www.wsj.com/tech/ai/openai-nearly-doubles-valuation-to-157-billion-in-funding-round-ee220607?mod=article_inline
[4] https://www.youtube.com/watch?v=Syt39QvVGOk&t=4187s
[5] https://www.wsj.com/tech/ai/open-ai-division-for-profit-da26c24b?mod=article_inline
Solve it like everything else (Score:1)
Ask AI to finish it if humans are too slow for you
Complexity is a bitch (Score:2)
The problem with algorithms is that you have to think of every likely scenario. "AI" provides a shortcut through real world examples training statistical models. Then whatever the given data adds up to as most likely becomes the solution to each new scenario. The problem is, this is just a hack, a brilliant hack, but just a hack. The belief is that, with more complexity in the model, more scenarios can be handled. That works quite well and amazingly, up to a point. Then the shit hits the fan, and the
news companies dumping their gpt5 stuff over the w (Score:2)
OpenAI announces their gpt4 o3 reasoning model that meets some guys definition of agi and now editors are rushing to publish their "gpt5 when?" articles they had queued up in the hopper (and already paid journalists for) to run over the holidays, half baked no less.
looking forward to all the articles about the new o3 capabilities in January once journalists have a chance to wrap their head around how it works/what it does/how it's different
Re: (Score:1)
These are somewhat related. [1]https://www.osnews.com/story/1... [osnews.com] and [2]https://www.wheresyoured.at/th... [wheresyoured.at]
[1] https://www.osnews.com/story/141399/never-forgive-them/
[2] https://www.wheresyoured.at/the-rot-economy/
Gosh (Score:2)
Who could have seen *that* coming. No wonder Sam always looks so scared and confused.
Improvement in LLMs starting to plateau. (Score:2)
LLMs are not the same thing as AI but when this stupid bubble bursts it will drag the whole industry down into another 20 year AI winter. While they have been an interesting curiosity for the last few years LLMs were always going to be a technological dead end. This is because there is literally nowhere for them to go - other than to try to scale up the models. Getting them to do anything useful - like reasoning - has to be a bolt on. In any event, hallucinations mean they can never be used for anything se
Re: Improvement in LLMs starting to plateau. (Score:2)
Nobody, at any level, has an actual clue what they are doing. This. 101%, because my AI counted in its fingers. The backlash has begun.
There may not be enough data in the world (Score:2)
and energy is running out fast too.
Surely this has to be the wrong way to go about doing artificial intelligence: AI companies throws exponentially-increasing amounts of resources at the problem and the amount of smarts they get out of their investment goes up more and more slowly. And privacy, IP laws and and the environment be damned to obtain those ever-dwindling advances.
A human brain burns about 20W. It's slow but it's infinitely better than any AI system on a Watt-per-smarts basis.
I think this was an
That's not how humans learn (Score:2)
I can't help but think that there must be a better way.
On the other hand, if someone wants to pay to maximize the potential of this algorithm, great! It's not my money, and I'll get to see what it can do.
Re: (Score:2)
Why are we even trying to mimic humans? Humans are pretty stupid and can be guided into even greater stupidity and evil.
Re: (Score:2)
Humans are vastly more efficient in terms of how many "tokens" they require to learn the same amount.
Re: (Score:2)
> Humans are vastly more efficient in terms of how many "tokens" they require to learn the same amount.
Humans learn by moving around, observing, and interacting with the world.
The human eye can process 30 frames per second at over 100-megapixel resolution.
By the time a two-year-old learns to talk, they've processed about 200 quadrillion bytes of data.
Re: (Score:2)
> By the time a two-year-old learns to talk, they've processed about 200 quadrillion bytes of data.
Even if that's the case, and even if all that data is actually used for brain remodeling, how much of it isn't redundant? According to the predictive brain theory (which is not proven), data gets dropped at every level of brain processing to the extent that it matches expectations of the deeper neural layers. Only the errors get propagated, which may be substantial for a two year old, but means for example that once objects can be recognized, very little visual data about any known object needs to be proces
Re: (Score:2)
Also, a lot of knowledge and insight does not require all that real world experience at all .
Mathematics is incredibly pure a priori knowledge, so even without any data about the universe an ANN should theoretically be able to reach perfect scores on every challenge. The (lack of) data is clearly not the problem here.
What we need is further architectural and/or training process improvements. "Prospective configuration" (alternative to backprop) might be interesting for the latter.
Re: (Score:2)
Humans don't need to read terabytes to learn how to read.
Re: (Score:2)
> Humans don't need to read terabytes to learn how to read.
An AI can learn to translate text to speech in a few hours of training.
A human takes five years.
Re: That's not how humans learn (Score:2)
Let's just se their output to train our new broken overlords.
Re: That's not how humans learn (Score:2)
Gotta stop posting drunk, sorry. Gorilla thumbs + crapcorrect + no preview on iPhone + adhd = ... yep use that to train ai.