AI future: Nvidia boffin hopes 'everything that moves will eventually be autonomous'
(2024/05/30)
- Reference: 1717061589
- News link: https://www.theregister.co.uk/2024/05/30/nvidia_ai_conference/
- Source link:
Video The GenAI Summit 2024 opened at the Palace of Fine Arts in San Francisco, California, on Wednesday, and the people, who came to hear about artificial intelligence, had made a mess of things.
Around 0900, as things were getting underway, a large crowd of attendees waited outside the venue in disorganized lines, held up as staff scrambled to find badges. This reporter was admitted after mentioning his media affiliation, without any verification challenge or scrutiny of identification. Everyone just wanted to get on with the show. Inside, admission to the VIP-only AGI keynote in the theater was similarly lax.
Jim Fan, senior research scientist at Nvidia and lead of its AI Agents Initiative, opened [1]the show by revisiting the history of artificial intelligence, starting with Claude Shannon's chess machine [2]Endgame . Various milestones were mentioned along the road to the "agentic era."
[3]
To begin at the end of Fan's presentation, the agentic era is where AI tech is headed, toward the development of software agents that orchestrate how foundational models interact with other models and systems.
[4]
[5]
The "agentic" era, Fan contends, is the next technological step after the "generative," "neural," and "classical" eras of AI.
"I believe in a future where everything that moves will eventually be autonomous," said Fan, without reflecting upon the potential implications.
[6]
Fan is trying to realize that vision through his work at Nvidia's [7]GEAR Lab , where GEAR stands for Generalist Embodied Agent Research.
[8]
Slide from Jim Fan's GenAI Summit 2024 presentation - Click to enlarge
A generalist agent, Fan explained, needs to be able to survive, navigate, and explore an open-ended world. It needs to have vast knowledge of that world. And it should be able to do pretty much any task.
"First, the environment needs to be open-ended enough because the agent's capability will ultimately be upper-bounded by the environment complexity," said Fan. "And the planet Earth we live on is a perfect example, because Earth is so complex that it allows an algorithm called natural evolution over billions of years to create all the humans in this room."
Massive amounts of data are also required, said Fan, "because it's not possible to explore from scratch. You need some common sense to bootstrap the learning."
Also, he said, you need a foundation model powerful enough to learn from all these sources. "And this train of thought lands us in Minecraft," said Fan.
[9]
Through Minecraft and related projects like [10]MineDojo , which consists of a simulator, database, and agent, [11]Voyager , a lifelong learning agent for Minecraft, [12]Eureka , an agent for training robots, [13]MetaMorph , and [14]Isaac Sim , Fan believes technologists will be able to train foundational agents to the point that they can perform a vast array of useful tasks.
Minecraft can be used as a simulator to teach agents how to perform specific tasks. And with Isaac Sim, that training can be done incredibly quickly.
"Isaac Sim's greatest strength is to run physics simulation at a thousand times or more faster than real-time," said Fan.
[15]MIT professor hoses down predictions AI will put a rocket under the economy
[16]China shows off machine-gun-toting robot dog and its AI-powered puppy
[17]Colorado governor signs 'best in the world' right-to-repair law
[18]Ex-OpenAI board member accuses Sam Altman of 'outright lying'
In other words, the path to get from chatbots to robots that can do useful tasks in the real world gets much shorter with simulation tools that can cram years worth of training into days. In fact, for one demonstration, teaching a robot hand to spin a pen in its fingers, the software would outperform most human pen-spinners if the hardware were up to the task.
"There's actually no real five finger hardware hack in the world that can have so much force and agility to spin a pen," said Fan. "So we're still waiting for hardware providers to catch up with Eureka."
But for some applications, like teaching a robot dog to walk and maintain its balance atop a deformable yoga ball, foundation agents [19]look promising .
[20]Youtube Video
"I believe training foundation agents will be very similar to ChatGPT," said Fan. "All language tasks can be expressed as text in and text out. And ChatGPT simply trains it by scaling it up across lots and lots of text. And very similar here, the foundation agent takes as a prompt an embodiment specification and a language instruction and then it outputs actions."
"The foundation agent is the next chapter for our GEAR Lab."
The robots are coming. ®
Get our [21]Tech Resources
[1] https://genaisummit.ai/
[2] https://news.mit.edu/2007/shannon-0530
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZlhOQ-wvrdy96bmmZqmh7AAAARE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZlhOQ-wvrdy96bmmZqmh7AAAARE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZlhOQ-wvrdy96bmmZqmh7AAAARE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZlhOQ-wvrdy96bmmZqmh7AAAARE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://research.nvidia.com/labs/gear/
[8] https://regmedia.co.uk/2024/05/30/foundation_agent_slide.jpg
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZlhOQ-wvrdy96bmmZqmh7AAAARE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] https://minedojo.org/
[11] https://voyager.minedojo.org/
[12] https://blogs.nvidia.com/blog/eureka-robotics-research/
[13] https://metamorph-iclr.github.io/site/
[14] https://developer.nvidia.com/isaac/sim
[15] https://www.theregister.com/2024/05/29/ai_gdp_inequality/
[16] https://www.theregister.com/2024/05/29/china_miiltary_robot_dogs/
[17] https://www.theregister.com/2024/05/29/colorado_governor_signs_best_in/
[18] https://www.theregister.com/2024/05/29/ex_openai_board_member_accuses_altman_lying/
[19] https://www.youtube.com/watch?v=kRj3crlFdOU
[20] https://www.youtube.com/watch?v=kRj3crlFdOU
[21] https://whitepapers.theregister.com/
Around 0900, as things were getting underway, a large crowd of attendees waited outside the venue in disorganized lines, held up as staff scrambled to find badges. This reporter was admitted after mentioning his media affiliation, without any verification challenge or scrutiny of identification. Everyone just wanted to get on with the show. Inside, admission to the VIP-only AGI keynote in the theater was similarly lax.
Jim Fan, senior research scientist at Nvidia and lead of its AI Agents Initiative, opened [1]the show by revisiting the history of artificial intelligence, starting with Claude Shannon's chess machine [2]Endgame . Various milestones were mentioned along the road to the "agentic era."
[3]
To begin at the end of Fan's presentation, the agentic era is where AI tech is headed, toward the development of software agents that orchestrate how foundational models interact with other models and systems.
[4]
[5]
The "agentic" era, Fan contends, is the next technological step after the "generative," "neural," and "classical" eras of AI.
"I believe in a future where everything that moves will eventually be autonomous," said Fan, without reflecting upon the potential implications.
[6]
Fan is trying to realize that vision through his work at Nvidia's [7]GEAR Lab , where GEAR stands for Generalist Embodied Agent Research.
[8]
Slide from Jim Fan's GenAI Summit 2024 presentation - Click to enlarge
A generalist agent, Fan explained, needs to be able to survive, navigate, and explore an open-ended world. It needs to have vast knowledge of that world. And it should be able to do pretty much any task.
"First, the environment needs to be open-ended enough because the agent's capability will ultimately be upper-bounded by the environment complexity," said Fan. "And the planet Earth we live on is a perfect example, because Earth is so complex that it allows an algorithm called natural evolution over billions of years to create all the humans in this room."
Massive amounts of data are also required, said Fan, "because it's not possible to explore from scratch. You need some common sense to bootstrap the learning."
Also, he said, you need a foundation model powerful enough to learn from all these sources. "And this train of thought lands us in Minecraft," said Fan.
[9]
Through Minecraft and related projects like [10]MineDojo , which consists of a simulator, database, and agent, [11]Voyager , a lifelong learning agent for Minecraft, [12]Eureka , an agent for training robots, [13]MetaMorph , and [14]Isaac Sim , Fan believes technologists will be able to train foundational agents to the point that they can perform a vast array of useful tasks.
Minecraft can be used as a simulator to teach agents how to perform specific tasks. And with Isaac Sim, that training can be done incredibly quickly.
"Isaac Sim's greatest strength is to run physics simulation at a thousand times or more faster than real-time," said Fan.
[15]MIT professor hoses down predictions AI will put a rocket under the economy
[16]China shows off machine-gun-toting robot dog and its AI-powered puppy
[17]Colorado governor signs 'best in the world' right-to-repair law
[18]Ex-OpenAI board member accuses Sam Altman of 'outright lying'
In other words, the path to get from chatbots to robots that can do useful tasks in the real world gets much shorter with simulation tools that can cram years worth of training into days. In fact, for one demonstration, teaching a robot hand to spin a pen in its fingers, the software would outperform most human pen-spinners if the hardware were up to the task.
"There's actually no real five finger hardware hack in the world that can have so much force and agility to spin a pen," said Fan. "So we're still waiting for hardware providers to catch up with Eureka."
But for some applications, like teaching a robot dog to walk and maintain its balance atop a deformable yoga ball, foundation agents [19]look promising .
[20]Youtube Video
"I believe training foundation agents will be very similar to ChatGPT," said Fan. "All language tasks can be expressed as text in and text out. And ChatGPT simply trains it by scaling it up across lots and lots of text. And very similar here, the foundation agent takes as a prompt an embodiment specification and a language instruction and then it outputs actions."
"The foundation agent is the next chapter for our GEAR Lab."
The robots are coming. ®
Get our [21]Tech Resources
[1] https://genaisummit.ai/
[2] https://news.mit.edu/2007/shannon-0530
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZlhOQ-wvrdy96bmmZqmh7AAAARE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZlhOQ-wvrdy96bmmZqmh7AAAARE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZlhOQ-wvrdy96bmmZqmh7AAAARE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZlhOQ-wvrdy96bmmZqmh7AAAARE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://research.nvidia.com/labs/gear/
[8] https://regmedia.co.uk/2024/05/30/foundation_agent_slide.jpg
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZlhOQ-wvrdy96bmmZqmh7AAAARE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] https://minedojo.org/
[11] https://voyager.minedojo.org/
[12] https://blogs.nvidia.com/blog/eureka-robotics-research/
[13] https://metamorph-iclr.github.io/site/
[14] https://developer.nvidia.com/isaac/sim
[15] https://www.theregister.com/2024/05/29/ai_gdp_inequality/
[16] https://www.theregister.com/2024/05/29/china_miiltary_robot_dogs/
[17] https://www.theregister.com/2024/05/29/colorado_governor_signs_best_in/
[18] https://www.theregister.com/2024/05/29/ex_openai_board_member_accuses_altman_lying/
[19] https://www.youtube.com/watch?v=kRj3crlFdOU
[20] https://www.youtube.com/watch?v=kRj3crlFdOU
[21] https://whitepapers.theregister.com/
Bollocks
"First, the environment needs to be open-ended enough because the agent's capability will ultimately be upper-bounded by the environment complexity," said Fan. "And the planet Earth we live on is a perfect example, because Earth is so complex that it allows an algorithm called natural evolution over billions of years to create all the humans in this room."
There are two core problems with this statement, [1] the Earth is a closed system, albeit with energy fed from outside, not open-ended at all; [2] natural evolution is not an algorithm -- it's an outcome of multiple independent processes.
It would be really great if public pronouncements (particularly those made by leading technocrats) were more often based on real knowledge in the areas they discuss. Then the myth that everything can be computerised (just like the 19th century myth that everything could be driven by, first, steam, and then electricity) would at last be recognised as just that -- a myth. There's an old Dilbert cartoon in which D states that there's an engineering solution to every problem. As an engineer, I can categorically state that he was wrong -- a huge array of human/social problems can not be overcome by engineering, they can only be buried from view by it while persisting (or indeed escalating).