Machine learning research in acoustics could open up multimodal metaverse
(2022/11/02)
- Reference: 1667385130
- News link: https://www.theregister.co.uk/2022/11/02/ml_research_acoustics/
- Source link:
Researchers at MIT and the IBM Watson AI Lab have created a machine learning model to predict what a listener would hear in a variety of locations within a 3D space.
The researchers first used the ML model to understand how any sound in a room will propagate through the space, building up a picture of a 3D room in the same way people use sound to understand their environment.
In a paper co-authored by Yilun Du, an MIT grad student in the Department of Electrical Engineering and Computer Science (EECS), the researchers show how techniques similar to visual 3D modeling can be applied to acoustics.
[1]
But they also struggled with elements where sound and light diverge. For example, changing the location of the listener in a room can create a very different impression of the sound due to obstacles, the shape of the room, and nature of the sound, making the outcome difficult to predict.
[2]
[3]
To overcome this problem, the researchers built into their model features of acoustics. Firstly, that the source of the sound and the listener can swap places without change in what the listener hears, all other things being equal. Sound is also specifically dependent on local features such as obstacles in the way of the listener or sound.
"Most researchers have only focused on modeling vision so far. But as humans, we have multimodal perception. Not only is vision important, sound is also important. I think this work opens up an exciting research direction on better-utilizing sound to model the world," [4]Du said .
[5]Self-driving truck startup TuSimple ousts CEO over ties to Chinese rival
[6]Bumble open sources AI code to automatically blur NSFW photos
[7]Enterprises are rolling out more AI – to 'middling results'
[8]Privacy watchdog urges companies drop emotional analysis AI software
Using the approach, the resulting neural acoustic field (NAF) model was able to randomly sample points on that grid to learn the features at specific locations. For example, proximity to a doorway strongly affects what that listener hears relative to other geometric features further away on the other side of the room.
The model was then able to predict what the listener might hear from a specific acoustic stimulus based on their relative locations in the room.
[9]
"By modeling acoustic propagation in a scene as a linear time-invariant system, NAFs learn to continuously map all emitter and listener location pairs to a neural impulse response function that can then be applied to arbitrary sounds," the [10]paper said [PDF]. "We demonstrate that the continuous nature of NAFs enables us to render spatial acoustics for a listener at an arbitrary location, and can predict sound propagation at novel locations."
Chuang Gan, a principal research staff member at the MIT-IBM Watson AI Lab who also worked on the project, said: "This new technique might open up new opportunities to create a multimodal immersive experience in the metaverse application."
We understand not all Reg readers will be excited about the above use case. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y2JN0MPOCvirhYSCvWGEKQAAANM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y2JN0MPOCvirhYSCvWGEKQAAANM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y2JN0MPOCvirhYSCvWGEKQAAANM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://news.mit.edu/2022/sound-model-ai-1101
[5] https://www.theregister.com/2022/11/01/tusimple_selfdriving_ceo_fired/
[6] https://www.theregister.com/2022/10/31/in_brief_ai/
[7] https://www.theregister.com/2022/10/31/deloitte_ai_enterprise/
[8] https://www.theregister.com/2022/10/29/data_privacy_ai_emotions/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y2JN0MPOCvirhYSCvWGEKQAAANM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://arxiv.org/pdf/2204.00628.pdf
[11] https://whitepapers.theregister.com/
The researchers first used the ML model to understand how any sound in a room will propagate through the space, building up a picture of a 3D room in the same way people use sound to understand their environment.
In a paper co-authored by Yilun Du, an MIT grad student in the Department of Electrical Engineering and Computer Science (EECS), the researchers show how techniques similar to visual 3D modeling can be applied to acoustics.
[1]
But they also struggled with elements where sound and light diverge. For example, changing the location of the listener in a room can create a very different impression of the sound due to obstacles, the shape of the room, and nature of the sound, making the outcome difficult to predict.
[2]
[3]
To overcome this problem, the researchers built into their model features of acoustics. Firstly, that the source of the sound and the listener can swap places without change in what the listener hears, all other things being equal. Sound is also specifically dependent on local features such as obstacles in the way of the listener or sound.
"Most researchers have only focused on modeling vision so far. But as humans, we have multimodal perception. Not only is vision important, sound is also important. I think this work opens up an exciting research direction on better-utilizing sound to model the world," [4]Du said .
[5]Self-driving truck startup TuSimple ousts CEO over ties to Chinese rival
[6]Bumble open sources AI code to automatically blur NSFW photos
[7]Enterprises are rolling out more AI – to 'middling results'
[8]Privacy watchdog urges companies drop emotional analysis AI software
Using the approach, the resulting neural acoustic field (NAF) model was able to randomly sample points on that grid to learn the features at specific locations. For example, proximity to a doorway strongly affects what that listener hears relative to other geometric features further away on the other side of the room.
The model was then able to predict what the listener might hear from a specific acoustic stimulus based on their relative locations in the room.
[9]
"By modeling acoustic propagation in a scene as a linear time-invariant system, NAFs learn to continuously map all emitter and listener location pairs to a neural impulse response function that can then be applied to arbitrary sounds," the [10]paper said [PDF]. "We demonstrate that the continuous nature of NAFs enables us to render spatial acoustics for a listener at an arbitrary location, and can predict sound propagation at novel locations."
Chuang Gan, a principal research staff member at the MIT-IBM Watson AI Lab who also worked on the project, said: "This new technique might open up new opportunities to create a multimodal immersive experience in the metaverse application."
We understand not all Reg readers will be excited about the above use case. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y2JN0MPOCvirhYSCvWGEKQAAANM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y2JN0MPOCvirhYSCvWGEKQAAANM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y2JN0MPOCvirhYSCvWGEKQAAANM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://news.mit.edu/2022/sound-model-ai-1101
[5] https://www.theregister.com/2022/11/01/tusimple_selfdriving_ceo_fired/
[6] https://www.theregister.com/2022/10/31/in_brief_ai/
[7] https://www.theregister.com/2022/10/31/deloitte_ai_enterprise/
[8] https://www.theregister.com/2022/10/29/data_privacy_ai_emotions/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y2JN0MPOCvirhYSCvWGEKQAAANM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://arxiv.org/pdf/2204.00628.pdf
[11] https://whitepapers.theregister.com/