Through the Looking Glass – holographic display hardware is great, but it's not enough
(2021/09/29)
- Reference: 1632902473
- News link: https://www.theregister.co.uk/2021/09/29/through_the_looking_glass_the/
- Source link:
Review Four years ago in a feature for The Register , [1]I wrote about the latest technologies for three-dimensional photography and videography. At the time, the tech required an array of tens to hundreds of cameras, all pointed inward at a subject, gathering reams of two-dimensional data immediately uploaded to the cloud for hours of post-processing, image recognition, feature extraction, and assembly into three- or four-dimensional media.
Today, I can fire up an app on my whizzy new iPhone 13 Pro, point its onboard LiDAR sensor at a subject, and record – in four dimensions – and in real time. That's enormous progress – a real revolution in sensors that gives our devices the capacity to capture depth. But, as I noted in the closing paragraphs of that feature, capturing depth does not mean that you can display it. All of our screens live in [2]Flatland – everything projected onto a surface of zero depth. Even the lively four-dimensional worlds of computer gaming still squash themselves against the screen. There may be depth within those virtual worlds, but it's not presented that way to our eyes.
Three-dimensional displays have been a bit of a holy grail of computing. Virtual- and augmented-reality systems use stereo pairs, projecting a slightly different image into each eye, but they're still too big and clumsy to be widely adopted, even for professional uses. Far better to use something that looks like a screen – but with depth. 3DTV [3]had a go at that a decade ago, but fell into the chicken-and-egg abyss of little content meaning slow adoption, meaning even less content, meaning … extinction.
There has to be a better way
On a tour of a friend's lab at PARC a few years ago (yes, although a shadow of its former self, PARC is still going), I caught a glimpse of something that looked a bit like the marriage of a screen and a small fishtank. In it swam a three-dimensional simulation of a fish – presented in 3D! That was my first encounter with one of the brand-new 3D displays from Brooklyn-based Looking Glass Factory.
The core of the technology behind Looking Glass Factory's displays dates back a century, when innovations in [4]lenticular printing placed a series of lens ridges above a printed surface, refracting different portions of the print to the viewer as they changed their viewing position relative to the lenses. Lenticular printing provides an illusion of depth – leading to a century of artwork in questionable taste.
[5]
Revived for the 21st century, lenticular displays place an LCD display behind the lenses, then mathematically distort the image on the display, using depth information, so that the right bits of it are mapped to the right angles as the display refracts through the lenses. (As I understand it, this mapping is part of Looking Glass Factory's heavily patented 'secret sauce'.) The result is something that looks very much like a photograph – until you realise that you can 'look around' within it:
In January, Looking Glass Factory launched a [6]Kickstarter campaign for its newest and most affordable device, called "Portrait" – a 20cm-diagonal lenticular display – garnering over US$2.5 million in pre-orders.
I borrowed one from a friend and have played with it for a month.
[7]
[8]
While it's pleasing to see so much progress being made on depth displays, it's disappointing to have such a promising device so frequently hamstrung by weak software support.
Performance issues
Although it does have an embedded Raspberry Pi 4 and can run standalone – working its way through a display list of pre-loaded animations – Portrait really shines when connected to a computer, via HDMI and USB ports. Although recognised as an auxiliary display, the lenticular elements make it seem blurry and not very useful as a display extender. The USB connection enables the computer to engage in a data conversation with the Portrait, mediated by a "Holoplay Service" that must be running in the background.
Because of the calculations required to distort images so they will correctly display on the Portrait, it really helps if the computer driving the display has a beefy GPU. Where my GeForce GTX 2070 super-powered desktop purred right along, my 2015 laptop struggled – and gave me the impression that the software itself had locked up. (It hadn't, but it was running at 1/100th the speed of my desktop machine.)
Once connected, it's a matter of using the Holoplay Studio app to get media onto the device. It's easy enough to create still depth images – both iOS and Android smartphones offer "portrait" mode in their onboard camera apps, and photos snapped in that mode contain visual data, plus a "depth map" providing three-dimensional metadata. Load one of those photos into the app, and viola!
[9]
(Just remember to be careful while uploading that photo from your smartphone, as many tools will strip out this 'extraneous' metadata. If that happens, you'll be left with a flat photo that Holoplay Studio will refuse to load.)
[10]Horizon Workrooms promises a virtual future of teal despair
[11]Trust Facebook to find a way to make video conferencing more miserable and tedious
[12]Ordnance Survey to take a poke at Pokémon-style gaming with outdoorsy AR adventure
[13]Singaporean boffins tame urban jungle with VR simulation of vertical greenery
Video is somewhat more complicated. Unless you have a depth-map-equipped smartphone (on iOS, that's anything with either FaceID or a LiDAR sensor), shooting four-dimensional video will require a device like a Kinect, Azure Kinect, Intel RealSense, or the like. Once captured, bringing it into Holoplay Studio should be as easy as drag-and-drop. But these four-dimensional video images are incredibly compute-intensive, so best save this work for a machine with serious GPU acceleration.
So close. yet so far away
Given my own background in [14]web-based real-time 3D , I really wanted to be able to create a web-based 3D animation that I could view in … well … 3D. Here I hit a series of roadblocks: code that had once worked, but has since been abandoned, or closed-beta repos that hadn't been touched in years, offering code samples that promised so much, yet did so little.
Such real-time web-based four-dimensional visualization looks like it should possible with Looking Glass Portrait – given the software interfaces the company does provide for Blender, Unity and Unreal – but doesn’t appear to be a priority for the firm. This means that the web – with its endless applications and creativity – is still a world apart.
That's quite an own goal. Google's recent [15]Project Starline – using its own version of the display technologies within the Looking Glass Portrait – shows us how much we long for communication that goes beyond the flat screen. Two years confined as heads-in-boxes on endless Zoom calls makes us ache for something more substantial – something with a bit more depth. It's easy to imagine a combination of smartphone plus Portrait plus web revolutionising video calls, but without a lot of developer software support it's very difficult to make that happen.
Looking Glass Portrait brings the experience of depth to the desktop. Right now, that's little more than a curiosity – barely useful, and rarely used. As we've learned over and over in the history of computing, new and non-standard hardware needs to be supported with excellent developer resources, so that some enterprising individual can build a killer app.
[16]
The hardware is ready. Although it should be the easy bit, the software is woefully lacking – and until that gets fixed I can't see how the Portrait ever achieves its intriguing potential. ®
Get our [17]Tech Resources
[1] https://www.theregister.com/2017/05/24/star_wars_new_hope_40_holograms/
[2] https://en.wikipedia.org/wiki/Flatland
[3] https://www.theregister.com/2011/09/28/brits_not_keen_on_3d_but_watch_it_anway/
[4] https://en.wikipedia.org/wiki/Lenticular_printing#History
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YVQ5X8SVdU7HZL0iAUAxsAAAAJU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[6] https://www.kickstarter.com/projects/lookingglass/looking-glass-portrait
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YVQ5X8SVdU7HZL0iAUAxsAAAAJU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YVQ5X8SVdU7HZL0iAUAxsAAAAJU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YVQ5X8SVdU7HZL0iAUAxsAAAAJU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://www.theregister.com/2021/08/23/horizon_workrooms_promises_a_virtual/
[11] https://www.theregister.com/2021/08/19/facebooks_oculus_horizon/
[12] https://www.theregister.com/2021/07/19/ordnance_survey_ar_game/
[13] https://www.theregister.com/2021/07/12/singapore_vr_greenery_study/
[14] https://www.theregister.com/2021/08/11/column_cyberbanana_windows/
[15] https://www.theregister.com/2021/05/18/google_io_2021/
[16] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YVQ5X8SVdU7HZL0iAUAxsAAAAJU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[17] https://whitepapers.theregister.com/
Today, I can fire up an app on my whizzy new iPhone 13 Pro, point its onboard LiDAR sensor at a subject, and record – in four dimensions – and in real time. That's enormous progress – a real revolution in sensors that gives our devices the capacity to capture depth. But, as I noted in the closing paragraphs of that feature, capturing depth does not mean that you can display it. All of our screens live in [2]Flatland – everything projected onto a surface of zero depth. Even the lively four-dimensional worlds of computer gaming still squash themselves against the screen. There may be depth within those virtual worlds, but it's not presented that way to our eyes.
Three-dimensional displays have been a bit of a holy grail of computing. Virtual- and augmented-reality systems use stereo pairs, projecting a slightly different image into each eye, but they're still too big and clumsy to be widely adopted, even for professional uses. Far better to use something that looks like a screen – but with depth. 3DTV [3]had a go at that a decade ago, but fell into the chicken-and-egg abyss of little content meaning slow adoption, meaning even less content, meaning … extinction.
There has to be a better way
On a tour of a friend's lab at PARC a few years ago (yes, although a shadow of its former self, PARC is still going), I caught a glimpse of something that looked a bit like the marriage of a screen and a small fishtank. In it swam a three-dimensional simulation of a fish – presented in 3D! That was my first encounter with one of the brand-new 3D displays from Brooklyn-based Looking Glass Factory.
The core of the technology behind Looking Glass Factory's displays dates back a century, when innovations in [4]lenticular printing placed a series of lens ridges above a printed surface, refracting different portions of the print to the viewer as they changed their viewing position relative to the lenses. Lenticular printing provides an illusion of depth – leading to a century of artwork in questionable taste.
[5]
Revived for the 21st century, lenticular displays place an LCD display behind the lenses, then mathematically distort the image on the display, using depth information, so that the right bits of it are mapped to the right angles as the display refracts through the lenses. (As I understand it, this mapping is part of Looking Glass Factory's heavily patented 'secret sauce'.) The result is something that looks very much like a photograph – until you realise that you can 'look around' within it:
In January, Looking Glass Factory launched a [6]Kickstarter campaign for its newest and most affordable device, called "Portrait" – a 20cm-diagonal lenticular display – garnering over US$2.5 million in pre-orders.
I borrowed one from a friend and have played with it for a month.
[7]
[8]
While it's pleasing to see so much progress being made on depth displays, it's disappointing to have such a promising device so frequently hamstrung by weak software support.
Performance issues
Although it does have an embedded Raspberry Pi 4 and can run standalone – working its way through a display list of pre-loaded animations – Portrait really shines when connected to a computer, via HDMI and USB ports. Although recognised as an auxiliary display, the lenticular elements make it seem blurry and not very useful as a display extender. The USB connection enables the computer to engage in a data conversation with the Portrait, mediated by a "Holoplay Service" that must be running in the background.
Because of the calculations required to distort images so they will correctly display on the Portrait, it really helps if the computer driving the display has a beefy GPU. Where my GeForce GTX 2070 super-powered desktop purred right along, my 2015 laptop struggled – and gave me the impression that the software itself had locked up. (It hadn't, but it was running at 1/100th the speed of my desktop machine.)
Once connected, it's a matter of using the Holoplay Studio app to get media onto the device. It's easy enough to create still depth images – both iOS and Android smartphones offer "portrait" mode in their onboard camera apps, and photos snapped in that mode contain visual data, plus a "depth map" providing three-dimensional metadata. Load one of those photos into the app, and viola!
[9]
(Just remember to be careful while uploading that photo from your smartphone, as many tools will strip out this 'extraneous' metadata. If that happens, you'll be left with a flat photo that Holoplay Studio will refuse to load.)
[10]Horizon Workrooms promises a virtual future of teal despair
[11]Trust Facebook to find a way to make video conferencing more miserable and tedious
[12]Ordnance Survey to take a poke at Pokémon-style gaming with outdoorsy AR adventure
[13]Singaporean boffins tame urban jungle with VR simulation of vertical greenery
Video is somewhat more complicated. Unless you have a depth-map-equipped smartphone (on iOS, that's anything with either FaceID or a LiDAR sensor), shooting four-dimensional video will require a device like a Kinect, Azure Kinect, Intel RealSense, or the like. Once captured, bringing it into Holoplay Studio should be as easy as drag-and-drop. But these four-dimensional video images are incredibly compute-intensive, so best save this work for a machine with serious GPU acceleration.
So close. yet so far away
Given my own background in [14]web-based real-time 3D , I really wanted to be able to create a web-based 3D animation that I could view in … well … 3D. Here I hit a series of roadblocks: code that had once worked, but has since been abandoned, or closed-beta repos that hadn't been touched in years, offering code samples that promised so much, yet did so little.
Such real-time web-based four-dimensional visualization looks like it should possible with Looking Glass Portrait – given the software interfaces the company does provide for Blender, Unity and Unreal – but doesn’t appear to be a priority for the firm. This means that the web – with its endless applications and creativity – is still a world apart.
That's quite an own goal. Google's recent [15]Project Starline – using its own version of the display technologies within the Looking Glass Portrait – shows us how much we long for communication that goes beyond the flat screen. Two years confined as heads-in-boxes on endless Zoom calls makes us ache for something more substantial – something with a bit more depth. It's easy to imagine a combination of smartphone plus Portrait plus web revolutionising video calls, but without a lot of developer software support it's very difficult to make that happen.
Looking Glass Portrait brings the experience of depth to the desktop. Right now, that's little more than a curiosity – barely useful, and rarely used. As we've learned over and over in the history of computing, new and non-standard hardware needs to be supported with excellent developer resources, so that some enterprising individual can build a killer app.
[16]
The hardware is ready. Although it should be the easy bit, the software is woefully lacking – and until that gets fixed I can't see how the Portrait ever achieves its intriguing potential. ®
Get our [17]Tech Resources
[1] https://www.theregister.com/2017/05/24/star_wars_new_hope_40_holograms/
[2] https://en.wikipedia.org/wiki/Flatland
[3] https://www.theregister.com/2011/09/28/brits_not_keen_on_3d_but_watch_it_anway/
[4] https://en.wikipedia.org/wiki/Lenticular_printing#History
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YVQ5X8SVdU7HZL0iAUAxsAAAAJU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[6] https://www.kickstarter.com/projects/lookingglass/looking-glass-portrait
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YVQ5X8SVdU7HZL0iAUAxsAAAAJU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YVQ5X8SVdU7HZL0iAUAxsAAAAJU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YVQ5X8SVdU7HZL0iAUAxsAAAAJU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://www.theregister.com/2021/08/23/horizon_workrooms_promises_a_virtual/
[11] https://www.theregister.com/2021/08/19/facebooks_oculus_horizon/
[12] https://www.theregister.com/2021/07/19/ordnance_survey_ar_game/
[13] https://www.theregister.com/2021/07/12/singapore_vr_greenery_study/
[14] https://www.theregister.com/2021/08/11/column_cyberbanana_windows/
[15] https://www.theregister.com/2021/05/18/google_io_2021/
[16] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YVQ5X8SVdU7HZL0iAUAxsAAAAJU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[17] https://whitepapers.theregister.com/
steelpillow
Indeed. Nobody has yet produced a fully effective static 3D display - not only the old artworks were questionable, the quality of the 3D rendering was as well. Personally I'll want to see this new system finding a sustainable market for simple printed 3D images before I'll get hot under the collar about e3D, let alone the groaningly cliche'd fourth dimension.
I suppose a decent lensing system would use little blobs rather than ridges and multiple spotted rather than striped images. Oh, and how many pixels to display all the different views per blob? I can see depth pixel resolution being a thing. But wait, what?! You mean we have to wear a special locator on our forehead so we look like a Brahmin, and the 2-px depth display is adjusted on-the-fly for just the one viewer; a film audience would each need their own screen?!
Not quite a tankful of fish, either way.
Natalie Gritpants Jr
Lenticular printing/displaying is really 2.5D. It's fine as you move left to right, but you can't see a different view by moving up and own. It fails miserably if you're lying sideways, which some of my family do to watch TV.
Nice reference
DarkwavePunk
to Flatland. A charming book although the prose does somewhat date itself. I'd also recommend Flatterland which I believe was written early 2000s (I'd check but it's probably filed in my bookcase between "I know it's here somewhere" and "Ooh! Forgot I owned that").
As for 3D displays - I just don't see it being possible without becoming some uncanny valley/nausea inducing effect.
Progress, of sorts
Pascal Monett
We have achieved videoconferencing for the masses. We can even use our smartphones for that now, something that the phone industry promised decades ago but failed to deliver.
The result ? Everybody hates it.
Now, we are witnessing a new gimmick : pics which have some depth to them. To make it work, you have to have bespoke hardware. That is very much going to limit the market. Yet another thing with batteries to follow.
It's an interesting idea, but I want it on my 26" widescreen.
Re: Progress, of sorts
Whiskers
... or 50" TV ...
The problem with video calls is not that they are 2D. A 3D head in a box is still a head in a box.