AI Companies Hit Development Hurdles in Race for Advanced Models (yahoo.com)
- Reference: 0175462225
- News link: https://slashdot.org/story/24/11/13/1237220/ai-companies-hit-development-hurdles-in-race-for-advanced-models
- Source link: https://finance.yahoo.com/news/openai-google-anthropic-struggling-build-100020816.html
Google's upcoming Gemini software and Anthropic's Claude 3.5 Opus are facing similar challenges. Google's project is not meeting internal benchmarks, while Anthropic has delayed its model's release, Bloomberg said. Industry insiders cited by the publication pointed to growing scarcity of high-quality training data and mounting operational costs as key obstacles. OpenAI's Orion specifically struggled due to insufficient coding data for training, the report said. OpenAI has moved Orion into post-training refinement but is unlikely to release the system before early 2024. The report adds:
> [...] AI companies continue to pursue a more-is-better playbook. In their quest to build products that approach the level of human intelligence, tech firms are increasing the amount of computing power, data and time they use to train new models -- and driving up costs in the process. Amodei has said companies will spend $100 million to train a bleeding-edge model this year and that amount will hit $100 billion in the coming years.
>
> As costs rise, so do the stakes and expectations for each new model under development. Noah Giansiracusa, an associate professor of mathematics at Bentley University in Waltham, Massachusetts, said AI models will keep improving, but the rate at which that will happen is questionable. "We got very excited for a brief period of very fast progress," he said. "That just wasn't sustainable."
Further reading : [2]OpenAI and Others Seek New Path To Smarter AI as Current Methods Hit Limitations .
[1] https://finance.yahoo.com/news/openai-google-anthropic-struggling-build-100020816.html
[2] https://tech.slashdot.org/story/24/11/11/144206/openai-and-others-seek-new-path-to-smarter-ai-as-current-methods-hit-limitations
What's the holdup? (Score:4, Funny)
> OpenAI has moved Orion into post-training refinement but is unlikely to release the system before early 2024.
I can't wait that long!
AI is not real intelligence (Score:3)
It's a very large, cleverly-designed word prediction system. We're finally hitting some hard limits, and it will help temper our expectations.
Don't get me wrong, I think LLMs are really useful and will do a lot of good, just don't mistake them for actual intelligence.
Re: (Score:2)
A fundamental problem with these models is they only use words and their associations. It doesn't visualize the problem as humans often do. For example, there's Maxwell's equations, but Maxwell also described how he came up with his equations where he imagined physical models to explain the equations. Like rotating vortex tubes. Which if you look at, primatively describes what's taught in school today about atoms and electrons, but the atom hadn't been discovered yet (Maxwell 1850s-70s, atom 1908). Scientis
Re: (Score:2)
To correct myself, not quite accurate to say 1908 as "atom discovery," more of a range where greater understanding of it was made, but it was around that time period.
Re: (Score:2)
> It's a very large, cleverly-designed word prediction system. We're finally hitting some hard limits, and it will help temper our expectations. Don't get me wrong, I think LLMs are really useful and will do a lot of good, just don't mistake them for actual intelligence.
Ridiculous. What we need is a book of infinite wisdom to train the models on. Then the more power the greater the wisdom. Like forever. I don’t understand why people think computer science is hard.
Why is it more data is needed? (Score:2)
I don't understand these repeated calls for more data. For almost everything you would want a LLM to solve now, the answer very likely already is in the training data for the current models. For e.g. coding challenges I doubt there's any facts, information etc. missing in the data, so it is weird it somehow comes down to a question of 'volume'. Isn't it a question of how well the model works with the data it has got? While I think LLM's are more powerful than many skeptics give it credit for (just statistic
Re: (Score:2)
> While I think LLM's are more powerful than many skeptics give it credit for (just statistical machines that find similar things in training data etc.) I do think that if size of training data is a limit as of now, it suggests LLM's seemingly do not have sufficient cognitive depth.
LLM are fancy autocomplete so the novel aspects of its output is more of an existing gap filler where existing patterns neatly outline something a bit like negative space in art. But it seems to not be able to hold concepts together in a long chain of reasoning, logic, or coding because it truly understands nothing even if it’s capable of novel output. LLM are likely just a small component of the AI investors and the public are looking for and we simply need a few hundred or thousand such diverse sy
Want to make a quick buck? (Score:2)
You might have missed the train already (compared to 2023), but if you're okay doing gruntwork for 20 USD/hr and know how to code, there are clients spending a lot on fabricating training data for models like Orion. One I saw recently is dataannotation.tech. I don't see these models going anywhere fast, but may as well take advantage of the optimistic vendors while you can.
Could it be (Score:5, Insightful)
That AI is overhyped and the bubble is well overdue for the burst it richly deserves?
Re: (Score:3)
With most tech the first 90% is easy to quickly achieve. And then the last 10% takes decades to sort out. I don't see why AI should be any different.
Re: (Score:2)
> With most tech the first 90% is easy to quickly achieve. And then the last 10% takes decades to sort out. I don't see why AI should be any different.
Call me back when it's more than a glorified albeit extremely convincing chatbot.
Re: (Score:2)
Could be. We hope.
I would suggest that there are psychological factors influencing what's going on. The people calling the shots on financing are... shall we say, deluded, uninformed, and investing with emotions, basically FOMO. Believe it or not, there's too much loose change in the world. Hedge funds for instance, by virtue of massive amounts of capital, and a bit of luck, can amass shitloads of money, and it has to go somewhere. There's too much money chasing profits. So the lineup of loaded dummies is l
Re: (Score:2)
> Their customers have been trapped in the idioms of the tools, and cannot conceptualize solutions outside that trap.
You're right on that one.
Here's a little anecdote that happened a few months ago:
One of the new recruits in our company, fresh out of college, was assigned to work on a project that requires a Python class I wrote. One day he popped into my office and asked me how to perform some I/O function. I told him the name of the method in the class.
15 minutes later, he came back and told me the method doesn't exist. Whaa...? So I opened the file and sure enough, there was the method.
You know what he told me? "But it