Proof-of-concept open-source app can cut'n'paste from reality straight into Photoshop using a neural network
- Reference: 1588671728
- News link: https://www.theregister.co.uk/2020/05/05/cut_paste_ar_photoshop/
- Source link:
For those of you thinking about toying with machine-learning in a practical sense, how about this interesting open-source proof-of-concept application as an inspiration or a starting guide?
A programmer has built an Android smartphone app in JavaScript and TypeScript, and corresponding back-end server in Python, that allows you to snap a photo of a real object and then transfer that image over the air almost instantly into Adobe Photoshop, with the background automatically removed by a neural network. Support for other imaging editing programs in the works, wee're told.
The end result is the ability to cut'n'paste stuff from what you can see through your phone straight into Photoshop. If that doesn’t sound exciting then, well, perhaps you have to see it to really appreciate it:
4/10 - Cut & paste your surroundings to Photoshop
Code: [1]https://t.co/cVddH3u3ik
Book: [2]@HOLOmagazine
Garment: SS17 by [3]@thekarentopacio
Type: Sainte Colombe by [4]@MinetYoann [5]@ProductionType
Technical Insights: ↓ [6]#ML [7]#AR [8]#AI [9]#AIUX [10]#Adobe [11]#Photoshop [12]pic.twitter.com/LkTBe0t0rF — Cyril Diagne (@cyrildiagne) [13]May 3, 2020
To make it work, Cyril Diagne, a self-proclaimed digital interaction artist, implemented [14]BASNet , a neural network that extracts the boundary of an object in an image. This is the magic that makes the automatic cut-out work. This approach is supposed to produce smoother results than traditional image-processing algorithms that detect the outlines of objects.
BASNet, for what it's worth, works by transforming an image into a saliency map, which is a fancy term for a heat map that blocks out the background of a photo to highlight the pictured object itself. That is then passed onto a refinement module that sharpens the boundaries to reduce any effects of noise.
The end result of this project is an [15]Android app that passes, via a REST API, camera photos to a [16]back-end server that does the actual AI processing. The server also connects to your installation of Photoshop via a remote-control connection to paste the snapped object into the open document. It's a neat way to bring stuff straight into Photoshop to place and mess around with.
The code for the copy-and-paste augmented reality prototype can be found [17]here . The Register has asked Diagne for comment. ®
Sponsored: [18]Legacy Modernization: Finding Your Way With Low-Code
[1] https://t.co/cVddH3u3ik
[2] https://twitter.com/HOLOmagazine?ref_src=twsrc%5Etfw
[3] https://twitter.com/thekarentopacio?ref_src=twsrc%5Etfw
[4] https://twitter.com/MinetYoann?ref_src=twsrc%5Etfw
[5] https://twitter.com/ProductionType?ref_src=twsrc%5Etfw
[6] https://twitter.com/hashtag/ML?src=hash&ref_src=twsrc%5Etfw
[7] https://twitter.com/hashtag/AR?src=hash&ref_src=twsrc%5Etfw
[8] https://twitter.com/hashtag/AI?src=hash&ref_src=twsrc%5Etfw
[9] https://twitter.com/hashtag/AIUX?src=hash&ref_src=twsrc%5Etfw
[10] https://twitter.com/hashtag/Adobe?src=hash&ref_src=twsrc%5Etfw
[11] https://twitter.com/hashtag/Photoshop?src=hash&ref_src=twsrc%5Etfw
[12] https://t.co/LkTBe0t0rF
[13] https://twitter.com/cyrildiagne/status/1256916982764646402?ref_src=twsrc%5Etfw
[14] http://openaccess.thecvf.com/content_CVPR_2019/papers/Qin_BASNet_Boundary-Aware_Salient_Object_Detection_CVPR_2019_paper.pdf
[15] https://github.com/cyrildiagne/ar-cutpaste/tree/master/app
[16] https://github.com/cyrildiagne/ar-cutpaste/tree/master/server
[17] https://github.com/cyrildiagne/ar-cutpaste
[18] https://go.theregister.co.uk/tl/1936/-8553/legacy-modernization-finding-your-way-with-low-code?td=wptl1936
Re: OK, I'll bite.
It's open source, and the code is linked to in the article. You know you could always raise a pull request to allow it to support your image editor of choice if it doesn't already.
Re: OK, I'll bite.
a) I would imagine that the code will rip out the object central to the focal point of the camera image
b) at some point the image will become a bitmap object in the code, so this should not be hard. I think that because Adobe supports remote commands it makes it easier to go via a server to your desktop. The AI bit is working on a server, too, but I wonder if it's simple enough to work on a phone?
Alternately, this would be a neat raddition to the "magic wand" select tools in most image editors.
Re: OK, I'll bite.
You can see it working on the demo, when he copies the book. That image also has a potted plant a little off-centre that's disregarded.
Well deserved pint!
Impressive
That's actually incredible and well done them for producing it.
If you are going to slate it please give an example of something you've produced which is better. I certainly haven't.
Unintended consequences
This is an excellent piece of work for sure but, I do see some "dark" applications for it. For instance, how about taking an image of your bosses signature and posting it into an important document? The process is so fast that you could do this by a quick visit to his desk while he is taking a comfort break. Yes, there are other ways of doing this but nothing like as fast as this is.
How about putting people into security images just before presenting evidence at a court case?
I think that this is just the start of some very scary technology......
OK, I'll bite.
If I take a photo of my physical desktop, which of the many objects on it are pasted into Photoshop? And if I take a picture outside my office window?
More importantly, for those of us who wouldn't run Adobe code if they paid us to run it, can I use the image editor of my choice?