Tech demo takes brain scan and creates a picture of what you're looking at
(2023/03/07)
- Reference: 1678201215
- News link: https://www.theregister.co.uk/2023/03/07/tech_demo_makes_pics_from_brainscan/
- Source link:
There's a lot of noise to signal in the machine-learning model world, but this demo is genuinely impressive – or scary, if you are given to recreationally climbing into MRI scanners.
The new research is presented in a paper titled "High-resolution image reconstruction with latent diffusion models from human brain activity" co-authored by Professor [1]Shinji Nishimoto and Assistant Professor [2]Yu Takagi of the Graduate School of Frontier Biosciences ( [3]FBS ) at [4]Osaka University . What the boffins have done is found a way to pass fMRI brain scans into the [5]open source Stable Diffusion latent-variable model created by [6]billion dollar startup unicorn Stability AI .
The results are startling, to say the least. Presented with the output of an fMRI brain scan – which to our eyes, looks very close to random noise – the researchers' limited-diffusion model can, in their words:
reconstruct high-resolution images with high fidelity in straightforward fashion, without the need for any additional training and fine-tuning of complex deep-learning models.
The preprint paper showcases five recovered images: a teddy bear, complete with its bow-tie; an avenue of trees; a jet airliner landing (or possibly taking off); a snowboarder on the slopes; and a tapering clocktower. The level of match is variable, and a sixth image, of a steam locomotive, is less clear, but it's remarkably good, even if the researchers had picked the best of their results, as we suspect they would naturally be inclined to do.
The boffins say that they the source code of their model is "coming soon". Their input data was four of the eight volunteers whose scans are in the University of Minnesota's public Natural Scenes Dataset or [7]NSD , and the sample images given in the paper are from one person.
[8]
Stable Diffusion itself has become famous for taking textual descriptions and generating sometimes very realistic images from just a handful of words – and if those are carefully enough chosen, [9]the text can evoke the original images used to train the model .
[10]
[11]
So while this is not precisely a computer reading someone's mind, it delivers markedly better results than, for example, some earlier efforts in this direction [12]which we reported on in 2021 . If we follow the paper correctly, they're using Stable Diffusion to improve the recovered images by incorporating elements from its training database. For comparison, some 12 years ago, a comparable [13]paper [PDF] using Bayesian statistics and modelling did produce some recognizable images, but of significantly lower quality.
[14]This app could block text-to-image AI models from ripping off artists
[15]It's official! Space travel increases the brain size of astronauts, even when they're back on Mother Earth
[16]Time for another cuppa then? Tea-drinkers have better brains, say boffins with even better brains
[17]Don't trust deep-learning algos to touch up medical scans: Boffins warn 'highly unstable' tech leads to bad diagnoses
As we have reported in the past, [18]the claims of fMRI research have long been controversial , but this is the sort of area where machine-learning and neural-network algorithms can be at their most useful: finding very faint signals, correlating and matching them with their huge libraries of images, to produce easily-recognizable results.
Functional MRI is a subset of magnetic resonance imaging, or nuclear magnetic resonance imaging as it used to be called before the boffins realized that the "n word" scared people off. The scanners involved are extremely large (and this vulture can attest, having been in more than one, extremely loud ) machines. Nobody is going to point a parabolic dish at your head from across the street and read what you're thinking about. But if you first sign a bunch of waivers, then spend an hour lying with your head clamped still while a huge donut-shaped magnet is rotated around it, yes, this kind of technique might be able to tell what picture you're looking at.
[19]
MRI - Click to enlarge
The two profs have a [20]page about their work, and you can read the [21]abstract or the whole 11-page [22]paper [PDF] on the bioRchiv preprint server. They will be presenting their findings at this year's [23]CVPR in Vancouver in June. ®
Get our [24]Tech Resources
[1] https://www.fbs.osaka-u.ac.jp/en/research_group/detail/25
[2] https://yu-takagi.github.io/
[3] https://www.fbs.osaka-u.ac.jp/en/
[4] https://www.osaka-u.ac.jp/en
[5] https://github.com/CompVis/stable-diffusion
[6] https://www.theregister.com/2022/10/23/two_generative_ai_startups_javis/
[7] https://naturalscenesdataset.org/
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/science&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZAdtuiqr-mopC3XBZDaH5wAAAIo&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[9] https://www.theregister.com/2023/02/06/uh_oh_attackers_can_extract/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/science&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZAdtuiqr-mopC3XBZDaH5wAAAIo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/science&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZAdtuiqr-mopC3XBZDaH5wAAAIo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[12] https://www.theregister.com/2021/02/12/ai_dr_who/
[13] https://www.biorxiv.org/content/biorxiv/early/2022/11/21/2022.11.18.517004.full.pdf
[14] https://www.theregister.com/2023/02/16/computer_scientists_develop_new_technique/
[15] https://www.theregister.com/2020/04/15/astronaut_brain_matter/
[16] https://www.theregister.com/2019/09/14/teadrinkers_have_better_brains_say_boffins_with_even_better_brains/
[17] https://www.theregister.com/2020/05/13/ai_medical_scans/
[18] https://www.theregister.com/2016/07/03/mri_software_bugs_could_upend_years_of_research/
[19] https://regmedia.co.uk/2023/03/07/shutterstock_mri.jpg
[20] https://sites.google.com/view/stablediffusion-with-brain/
[21] https://www.biorxiv.org/content/10.1101/2022.11.18.517004v2
[22] https://www.biorxiv.org/content/biorxiv/early/2022/11/21/2022.11.18.517004.full.pdf
[23] https://cvpr2023.thecvf.com/
[24] https://whitepapers.theregister.com/
The new research is presented in a paper titled "High-resolution image reconstruction with latent diffusion models from human brain activity" co-authored by Professor [1]Shinji Nishimoto and Assistant Professor [2]Yu Takagi of the Graduate School of Frontier Biosciences ( [3]FBS ) at [4]Osaka University . What the boffins have done is found a way to pass fMRI brain scans into the [5]open source Stable Diffusion latent-variable model created by [6]billion dollar startup unicorn Stability AI .
The results are startling, to say the least. Presented with the output of an fMRI brain scan – which to our eyes, looks very close to random noise – the researchers' limited-diffusion model can, in their words:
reconstruct high-resolution images with high fidelity in straightforward fashion, without the need for any additional training and fine-tuning of complex deep-learning models.
The preprint paper showcases five recovered images: a teddy bear, complete with its bow-tie; an avenue of trees; a jet airliner landing (or possibly taking off); a snowboarder on the slopes; and a tapering clocktower. The level of match is variable, and a sixth image, of a steam locomotive, is less clear, but it's remarkably good, even if the researchers had picked the best of their results, as we suspect they would naturally be inclined to do.
The boffins say that they the source code of their model is "coming soon". Their input data was four of the eight volunteers whose scans are in the University of Minnesota's public Natural Scenes Dataset or [7]NSD , and the sample images given in the paper are from one person.
[8]
Stable Diffusion itself has become famous for taking textual descriptions and generating sometimes very realistic images from just a handful of words – and if those are carefully enough chosen, [9]the text can evoke the original images used to train the model .
[10]
[11]
So while this is not precisely a computer reading someone's mind, it delivers markedly better results than, for example, some earlier efforts in this direction [12]which we reported on in 2021 . If we follow the paper correctly, they're using Stable Diffusion to improve the recovered images by incorporating elements from its training database. For comparison, some 12 years ago, a comparable [13]paper [PDF] using Bayesian statistics and modelling did produce some recognizable images, but of significantly lower quality.
[14]This app could block text-to-image AI models from ripping off artists
[15]It's official! Space travel increases the brain size of astronauts, even when they're back on Mother Earth
[16]Time for another cuppa then? Tea-drinkers have better brains, say boffins with even better brains
[17]Don't trust deep-learning algos to touch up medical scans: Boffins warn 'highly unstable' tech leads to bad diagnoses
As we have reported in the past, [18]the claims of fMRI research have long been controversial , but this is the sort of area where machine-learning and neural-network algorithms can be at their most useful: finding very faint signals, correlating and matching them with their huge libraries of images, to produce easily-recognizable results.
Functional MRI is a subset of magnetic resonance imaging, or nuclear magnetic resonance imaging as it used to be called before the boffins realized that the "n word" scared people off. The scanners involved are extremely large (and this vulture can attest, having been in more than one, extremely loud ) machines. Nobody is going to point a parabolic dish at your head from across the street and read what you're thinking about. But if you first sign a bunch of waivers, then spend an hour lying with your head clamped still while a huge donut-shaped magnet is rotated around it, yes, this kind of technique might be able to tell what picture you're looking at.
[19]
MRI - Click to enlarge
The two profs have a [20]page about their work, and you can read the [21]abstract or the whole 11-page [22]paper [PDF] on the bioRchiv preprint server. They will be presenting their findings at this year's [23]CVPR in Vancouver in June. ®
Get our [24]Tech Resources
[1] https://www.fbs.osaka-u.ac.jp/en/research_group/detail/25
[2] https://yu-takagi.github.io/
[3] https://www.fbs.osaka-u.ac.jp/en/
[4] https://www.osaka-u.ac.jp/en
[5] https://github.com/CompVis/stable-diffusion
[6] https://www.theregister.com/2022/10/23/two_generative_ai_startups_javis/
[7] https://naturalscenesdataset.org/
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/science&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZAdtuiqr-mopC3XBZDaH5wAAAIo&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[9] https://www.theregister.com/2023/02/06/uh_oh_attackers_can_extract/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/science&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZAdtuiqr-mopC3XBZDaH5wAAAIo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/science&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZAdtuiqr-mopC3XBZDaH5wAAAIo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[12] https://www.theregister.com/2021/02/12/ai_dr_who/
[13] https://www.biorxiv.org/content/biorxiv/early/2022/11/21/2022.11.18.517004.full.pdf
[14] https://www.theregister.com/2023/02/16/computer_scientists_develop_new_technique/
[15] https://www.theregister.com/2020/04/15/astronaut_brain_matter/
[16] https://www.theregister.com/2019/09/14/teadrinkers_have_better_brains_say_boffins_with_even_better_brains/
[17] https://www.theregister.com/2020/05/13/ai_medical_scans/
[18] https://www.theregister.com/2016/07/03/mri_software_bugs_could_upend_years_of_research/
[19] https://regmedia.co.uk/2023/03/07/shutterstock_mri.jpg
[20] https://sites.google.com/view/stablediffusion-with-brain/
[21] https://www.biorxiv.org/content/10.1101/2022.11.18.517004v2
[22] https://www.biorxiv.org/content/biorxiv/early/2022/11/21/2022.11.18.517004.full.pdf
[23] https://cvpr2023.thecvf.com/
[24] https://whitepapers.theregister.com/
nuclear magnetic resonance imaging
Anonymous Coward
Sometime in the 90's, whilst being wheeled about a hospital with a broken leg, we passed a sign for the "Department of Nuclear Medicine". Although "Nuclear Medicine" might have sounded pretty awesome, on the whole I was probably better off in Orthopaedics.
Yeah, this won't be abused.
NoneSuch
US Border check points to scan your brain for "criminal activity" when travelling. The CCP will love this for scanning dissidents in mainland China. Not to mention HR using this as a pre-scan for job interviews.
Re: Yeah, this won't be abused.
alain williams
What was more scary was a paper that I read that showed different brain activity when the person was shown something that they had seen before. So: worry when the border guards ask you if you "have seen this person before".
Sorry, I cannot find the link - maybe I need my brain scanning :-)
Re: Yeah, this won't be abused.
chivo243
I'm wondering if the George Costanza factor can be invoked... "It's not a lie if you believe it". Think happy thoughts!
After this, what?
Primus Secundus Tertius
So far, it is pretty pictures rather than sounds or smells.
If the picture is a page of text, can it produce a Word file of the result?
No Not The Mind Probe!
The Oncoming Scorn
When one is used on the Doctor it ends up being destroyed when General Williams refuses to accept it as truth. The Doctor talks of a similar Noodle Incident where he was captured by the Medusoids and broke their own mind probe. He was freed when the Medusoids ran out of mind probes.
Finally
lukewarmdog
A machine that can tell me what video I really want to watch on PornHub tonight!
(other websites may be available..)
Interesting how those in the know (the paper authors) in their paper don't just hide the faces of the people in photos presented to those being scanned by fMRI, but their whole bodies from other data models that no doubt in future get input from their paper.