Microsoft seeks patent for tech to put words into your mouth
- Reference: 1707391747
- News link: https://www.theregister.co.uk/2024/02/08/microsoft_patent_speach/
- Source link:
The patent [1][PDF] describes a system made up of an audio processing module, supported by visual and text code, and married with voice tracking software. The audio from a file is extracted and removed and a new voice is generated, matching the timing of the original speech, and even translating audio and text to another language if desired.
Here's a schematic drawing portraying the system's components from the patent filing:
[2]
From Microsoft's application ... The proposed processing pipeline
AI can be used to clone people's voices, and automatically translate them into different languages. The system detailed in the patent brings that functionality to personal devices, websites, DVD players, and more.
[3]The FCC wants to criminalize AI robocall spam
[4]Uncle Sam will pay for your big ideas to end AI voice-cloning fraud
[5]Is it 2000 or 2023? Get ready for AI-anchored news. Again
In one "implementation scenario", Microsoft describes how its automatic dubbing can be used to customize a character's voice in a film or game. It envisions a user replacing their own voice with an automatically generated synthetic version across multiple formats.
It's easy to imagine how such a system can be used in the future. The voices of actors, podcasters, or politicians could be preserved and translated across multiple languages to reach a wider audience using AI. Maybe fake AI-generated voices can be used instead, converting text to speech in apps and bringing virtual personalities to life in apps or devices. Users could choose how they want to listen to an audiobook or chatbot, for example.
[6]
Microsoft’s filing doesn’t state the intended purpose of its invention but suggests it might reduce dubbing costs for entertainment and media companies. They could use their software to translate a voice actor's speech into different languages, instead of hiring human linguists, although the actor's unions [7]might have something to say about that – especially as in some countries voice actors specialize in dubbing particular Hollywood stars into local languages, and achieve a decent measure of fame for doing so.
[8]
The Register is also reminded of the TomTom satnav unit that in 2009 offered the voice of Homer Simpson, actor Dan Castellaneta, as a [9]paid upgrade for those seeking directions to the nearest Krusty Burger. Microsoft’s patent could make Castellaneta’s voice a tick-box option in any language.
The Register has asked Microsoft for comment but has not received a response at the time of publication. D’oh! ®
Get our [10]Tech Resources
[1] https://image-ppubs.uspto.gov/dirsearch-public/print/downloadPdf/11887578
[2] https://regmedia.co.uk/2024/02/07/ms_audio_dubbing_patent.jpg
[3] https://www.theregister.com/2024/02/02/fcc_ai_robocall/
[4] https://www.theregister.com/2024/01/05/ftc_voice_cloning_solution/
[5] https://www.theregister.com/2023/12/15/channel_1_ai_news/
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZcUIuC7vvyePyvsHIMWCVAAAAdU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[7] https://www.theregister.com/2023/11/13/actors_strike_ai/
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZcUIuC7vvyePyvsHIMWCVAAAAdU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://www.theregister.com/2009/06/16/tomtom_homer/
[10] https://whitepapers.theregister.com/
Re: Prior Art
I suspect prior art may already exist. Text-to-speech systems have existed for decades and voice synthesis has been around for a few years. Is this another case of an American company trying to patent a pipeline? The only slightly novel thing I can think of is maybe getting Einstein to speak Klingon correctly, though I wonder if the all the voice shifts are possible? I know that my voice changes depending on which language I'm speaking.
Imagine what havoc this technology would unleash
if it were used in elections.
Welcome to the Age of Plausible Deniability
The tape recorder and video camera gave us the "Age of Accountability" where incontrovertible evidence could be produced to expose liars, cheats and criminals
Now, generative AI gives us the Age of Plausible Deniability, where anyone can simply claim that whatever evidence it is must be "fake news", especially if they are powerful enough to be widely believed
Re: Welcome to the Age of Plausible Deniability
It's worse: anything not witnessed in person, face to face, with someone whom you personally know and can identify, cannot be trusted to have said/done what it claimed.
Re: Imagine what havoc this technology would unleash
Already has. They tracked the GQP russian cult scammer that imitated President Biden's voice in Texas.
Re: Imagine what havoc this technology would unleash
Or in a telephone fraud coming to your neighbourhood soon…
Theoretically speaking?
This patent application is very odd to a UK observer. Nowhere is any finite mechanism described that fulfils any of the proposed functionality, so it's an entirely conceptual document. If this high level kind of application can succeed in the US it means that entire disciplines could be locked down by opportunists without the need to actually implement any demonstrable invention. I do hope that's not the case, as it would stifle innovation, the very thing patents were created to encourage.
Re: Theoretically speaking?
"it means that entire disciplines could be locked down by opportunists without the need to actually implement any demonstrable invention"
Yup - that's exactly how the US patent system works. Patents are only tested for originality or practicability when they are challenged by either a competitor, ie a holder of a similar patent, or when the holder uses said patent to crush (read: overwhelm and buyout) an "imposter" who actually gets something relatively similar to work.
From a UK perspective it is indeed a very, very odd system: only of any benefit to lawyers (guess who wrote the rules...)
Re: Theoretically speaking?
Look, that old world we knew...It's gone. It doesn't exist. All that stuff we learned our whole lives. "Just because you can doesn't mean you should", wrong... they'll do it if it can be done. "Anything worth doing should be done right", wrong, it's all just instantly reproducible, objectified digital content now. "Two wrongs don't make a right", nope... it's eye for an eye now. Why you ask?
Because, you buy it. Someone will. These are companies. Their job is to sell you ish. These are not good people doing good things.
They will sell you ish in a bag if you will buy it for a penny, and someone will. Someone will spend a penny on dog ish in a bag if it's marketed right.
It's all cheaply produced, easily reproducible, instantaneous dog ish in a bag. And, people buy it.
Re: Theoretically speaking?
It's a method patent (often called process patents in other countries, such as the UK), rather than a machine patent, as you described. They crop up in chemistry-related topics a fair amount, to describe a particular production process.
https://en.wikipedia.org/wiki/Method_(patent)
Edited to make clear: It's an appication at this stage, not a granted patent!
Looking forward to...
...this technology being deployed remotely on your BlueTooth equipped toothbrushes (yes they are out there).
Stop putting words in my mouth, just shut the f*** up and clean my teeth!
But, does it lip sync?
I'd have thought that either replacing a video's original speech track with one in another language and/or replacing it with a different speech in the same language would both show a rather obvious lip sync mismatch, but there's no indication of whether the AI model is capable of recreating lip movements to match the revised audio.
It would be nice to know whether if it has that capability.
Not really that paranoid... and have no money anyway but...
Think of all the places where telephone calls are recorded for training and quality purposes. Not stored securely and able to be listened to/downloaded by anyone.
My bank uses "My voice is my password"
Prior Art
Neal Stephenson's "REAMDE" featured a game which did exactly that. That was published in 2011.
Now in 2024, after the boom of generative AI, and audio deepfakes becoming commonplace, this could also be considered "bleedin' obvious" and thus not patentable