Google isn't even trying to not be creepy: 'Continuous Match Mode' in Assistant will listen to everything until it's disabled
- Reference: 1592572507
- News link: https://www.theregister.co.uk/2020/06/19/google_assistant_new_dev_tools_features/
- Source link:
The Chocolate Factory has big plans for its voice assistant. "We consider voice to be the biggest paradigm shift around us," said director of product Baris Gultekin, speaking at the Voice Global summit, where the new features were [1]introduced .
The goal is "ambient computing", where you can interact with the big G anywhere at any time, so pervasively that you do not notice it. Voice interaction is a key part of this since it extends the ability to perform searches or run applications to scenarios where tapping a keyboard or touching a display are not possible.
Google Assistant exists in many guises such as on smartphones and watches, TVs, PCs, and also on dedicated hardware, such as the voice-only Google Home and Google Home Mini, or with "smart display" screens on the Google Nest Hub or devices from Lenovo and Harman. While assistant devices have been popular, Android phones (which nag you to set up the Assistant) must form the largest subset of users. Over all the device types, the company claims over 500 million active users.
As with any evolving platform, developer support is critical. At the Voice Global event, the company introduced new tooling, the biggest piece being the Actions Builder, a web-based tool that lets you add and assemble the pieces of an "action", essentially an application for Assistant. Actions Builder is an easy-to-use tool that integrates with Natural Language Understanding for parsing user input into what makes sense for an action, and lets you create the flow of intents and responses that form a complete action.
Payam Shodjai, senior director of product for Google Assistant, explained at the summit that Builder is a response to complaints that the previous approach, where developers switched back and forth between an Actions Console and a [2]DialogFlow editor, was too disjointed.
Actions Builder will "replace DialogFlow as the preferred way to develop actions on the assistant," said Shodjai.
[3]
Google's new Action Builder at work
Trying out the new Action Builder, we discovered that running an action under development is impossible if you have the Web and App Activity permission, which lets Google keep a record of your actions, disabled. A dialog appears prompting you to enable it. It is a reminder of how Google Assistant is entwined with the notion that you give Google your data in return for personalised experiences.
Some developers prefer to work with text-based code rather than a visual builder, and Google is responding to this need with an updated Actions SDK which lets you work in your preferred IDE or programmer's editor. This also makes it easier to use source control and continuous integration tools with action projects.
There are also new features in the Assistant platform. Home Storage is a way of storing values across multiple sessions in a single household, as defined by Google's [4]Home Graph , part of its smart home API. This is significant because it enables multi-user applications. The examples demonstrated were games such as word puzzles.
DOn't say we didn't warn you
The [5]documents reveal a strong warning, though, in red type: "Important: Do not use home storage to store any personal data. Home storage data applies across multiple individuals in the same household who use the same action."
It will be down to developers to heed such warnings.
The Media API for Google Assistant has been updated and now handles play and resume across devices, and the general ability to start playback from a specific point rather than from the beginning.
Devices with Smart Displays are gaining the ability to display web content – provided it is delivered using the [6]AMP component framework . Initial support will be for news sites, coming later this summer, but with more categories to follow.
The most intriguing new feature is called Continuous Match Mode. This touches on an area of debate, which is the extent an Assistant device listens in on conversations without specifically being asked to do so. Currently you have to use a wake word to initialise an Assistant action, though devices must be listening at all times to some extent in order to recognise the wake word – "Hey Google" or "OK Google" – which can be followed by a custom invocation for a third-party action.
Once in a conversation – the equivalent to running an application – the system follows a flow based on parsing the user's words into intents that are meaningful in the the context of the conversation. The action continues to completion of the flow, or quits if the action cannot parse the user's intents, or after a system command such as "cancel". The developer can override the first exit command with one last plea to continue, but not the second.
"Sometimes you want to build experiences that enable the mic to remain open, to enable users to speak more naturally with your action, without waiting for a change in mic states," said Shodjai at the summit and in the [7]developer post .
"Today we are announcing an early access program for Continuous Match Mode, which allows the assistant to respond immediately to user's speech enabling more natural and fluid experiences. This is done transparently, so that before the mic opens the assistant will announce, 'the mic will stay open temporarily', so users know they can now speak freely without waiting for additional prompts."
The mode is not yet publicly documented. The demonstrated example was for a game with jolly cartoon pictures; but there may be privacy implications since in effect this setting lets the action continue to listen to everything while the mode is active.
Shodjai did not explain how users will end a Continuous Match Mode session but presumably this will be either after a developer-defined exit intent, or via a system intent as with existing actions. Until that happens, the action will be able to keep running.
Just as with personalisation via tracking and data collection, privacy and pervasive computing do not sit comfortably together, and with the new Continuous Match Mode a little more privacy slips away. ®
Sponsored: [8]Continuous Lifecycle Online, 15 July 2020. Register Now!
[1] https://www.voicesummit.ai/global
[2] https://dialogflow.com/
[3] https://regmedia.co.uk/2020/06/18/spyonme.jpg
[4] https://developers.google.com/assistant/smarthome/concepts/homegraph
[5] https://developers.google.com/assistant/conversational/storage-home
[6] https://amp.dev/
[7] https://developers.googleblog.com/2020/06/voice-global-2020-new-improvements-to-core-platform.html
[8] https://go.theregister.com/tl/1968/shttps://www.bigmarker.com/series/continuous-lifecycle-online/series_summit
Also, isn’t there a GDPR issue to something like this being scrapped? As I understand it data can be collected, processed and stored only for a specific purpose. If something like this is scrapped after a short time, there wasn’t really a purpose for that data to be collected in the first place, was there?
At the very least, Google’s T&Cs are going to have to start saying “we’re experimenting” as the primary purpose for snaffling data, not “we’re providing a service”...
It's Google though... so GDPR is done on a kinda flexi basis - more of a collect now, worry later - and the worry is in the most minimal way
GDPR
So if I go to somebody's house and Continuous Match Mode is enabled, who has to get my consent? Bearing in mind the voice surveillance isn't necessary for my visit, so by GDPR law I have to be proactively offered an opt-in or -out...
Re: GDPR
"So if I go to somebody's house and Continuous Match Mode is enabled, who has to get my consent?"
Simple: google will simply track where you are and who is near you and your relationship with those people and their preferences, and from that detect which people within voice range have and have not given consent, identifying them by their voices. It will automatically discard data from anyone who has explicitly rejected CMM, keeping data from all other users stored in perpetuity on the assumption that they will later accept the CMM terms and conditions. Incidentally, this will allow your always-on display to show appropriate advertising not just to you, but to the people around you, improving your life in every possible way.
Re: GDPR
so by GDPR law I have to be proactively offered an opt-in or -out
FTFY
Catching up, maybe???
Is this Google getting left behind by BETTER SNOOPING TECHNOLOGY from Amazon or Microsoft?
*
I think we should be told!
Re: Catching up, maybe???
There's always one one every Google article going "Look!! M$!! Aren't they the REAL bastards here?"
Why is that?
Re: Catching up, maybe???
Skype for business is always listening to the microphone. When disabled in the privacy settings calls are not heard.
Oh mighty one...
If they want to be omnipresent and all powerful, why don't they just change their name to God?
Re: Oh mighty one...
Goddle?
"It will be down to developers to heed such warnings."
Oh good...
It seems that
advertising business is slowly declining for Google now that it has to share the turf with Facebook, Apple, Microsoft and others (cloud is also nothing they can brag about) but they still need to report growth in order to please the Wall Street moneymen. While searching for a new business model, they have intensified slurping in the hope they will find new way of monetizing the big data, something useful that would put them back on an ascending trajectory.
The thing is...
...I would quite like being able to ask questions to an "assistant" naturally as part of any given conversation or situation and have an appropriate response in a timely manner.
But only (and this is a huge BUT) if all audio processing is performed locally, and any subsequent searches were performed anonymously.
Unfortunately this is Google, and their entire business model is based around non-anonymous online systems, so thanks, but no thanks.
Does not compute.
>>>Shodjai did not explain how users will end a Continuous Match Mode.<<<
Google ending a data collection session ... It's nice to end the week with a laugh.
Re: Does not compute.
"...Shodjai did not explain how users will end a Continuous Match Mode session"
I think a sledge hammer would do the trick!
Shodjai did not explain how users will end a Continuous Match Mode session but presumably this will be ...
... after the heat death if the universe?
To be fair it ends when the extra processing drains the battery to empty. A new session starts when you recharge the phone
"Google Assistant exists in many guises such as on smartphones and watches, TVs, PCs, and also on dedicated hardware, such as the voice-only Google Home and Google Home Mini, or with "smart display" screens on the Google Nest Hub or devices from Lenovo and Harman."
This is the lie (or rather one of many lies) Google would like perpetuated. Google Assistant is not in these devices, only a bit of software that switches the mic on and off is in these. Google Assistant is in a dark data center somewhere, one you cannot access, influence nor control. Racks of blinkenlights acknowledge the receipt of another packet of your private existence for processing, monetisation and long, long term storage.
Microphone never turns off. Microphone is always on, as it's at least listening for an activation phrase.
Power without abuse never loses it's charm.
Microphone never turns off. Microphone is always on, as it's at least listening for an activation phrase.
Sooo.. Assuming you're not on a metered Internet connection. Stick a mic in a sound damped box, add speaker, add loop saying 'OK Google, FOAD'. Close box.
Software types might be able to do this without hardware, although it sounds like Google's going to make it difficult to turn off snooping. Bigger challenge if people have mic enabled iThings that they are either unaware of, or can't disconnect/turn off mics.
Important: Do not use home storage to store any personal data.
Oh, good. A statement that you should not use "home storage" for many of the sorts of things you might actually want to put on something called "home storage". Just as well everybody carefully reads all the warnings, otherwise all kinds of privacy-violating "accidents" might occur.
It’s alright, they’ll have scrapped it inside 2 years anyway..: